Design a A/B Testing for Oracle

Question

Design a event-driven a/b testing system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements:
1. **Experiment Creation**: Users (data scientists/marketers) should be able to create A/B tests with specified variants, targeting parameters, and duration.
2. **Event Tracking**: The system must track user interactions with variants in real-time.
3. **Results Analysis**: Users must be able to view aggregated results, conversion rates, and statistical significance.
4. **API Access**: Provide RESTful APIs for creating tests, tracking events, and retrieving results.

### Non-Functional Requirements:
1. **Scalability**: The system should handle millions of requests per day, scaling horizontally.
2. **Availability**: The system must maintain high availability with an uptime of 99.9%.
3. **Performance**: Latency for tracking events should be under 200ms.
4. **Data Consistency**: Event data should ensure eventual consistency to handle real-time analytics.

Assuming Oracle has approximately 10 million active users:
- **Daily Events**: If each user interacts with the system 5 times a day, we will have around 50 million events daily.
- **Peak Load**: During promotional campaigns, this could spike to 100 million events per day or about 1,157 events per second.
- **Storage Needs**: Assuming each event requires 1KB of storage, we would need around 50GB of storage per day, or approximately 18TB per year. 
- **Data Retention**: If we retain data for 1 year, we would need roughly 18TB * 1 year = 18TB of storage.

### Components:
1. **API Gateway**: An NGINX or AWS API Gateway to handle incoming requests and route them.
2. **Event Processing Service**: A microservice (e.g., written in Node.js) that handles event tracking and processing.
3. **Experiment Management Service**: A service that manages A/B test lifecycle and metadata.
4. **Database**: Use a distributed NoSQL database like Cassandra for storing event data, allowing for high availability and scalability.
5. **Analytics Engine**: A service (e.g., Apache Spark) for processing and analyzing results, providing insights through a dashboard.
6. **Caching Layer**: Redis for caching frequently accessed experiment data and results to reduce database load.

### Data Flow:
- Users interact with the API Gateway, which routes requests to the appropriate services.

### Schema Design:
1. **Events Table** (Cassandra):
   - `event_id` (UUID, Primary Key)
   - `user_id` (UUID)
   - `experiment_id` (UUID)
   - `variant` (String)
   - `timestamp` (DateTime)

2. **Experiments Table** (Cassandra):
   - `experiment_id` (UUID, Primary Key)
   - `name` (String)
   - `start_time` (DateTime)
   - `end_time` (DateTime)
   - `variants` (List<String>)

3. **Results Table** (PostgreSQL for relational data):
   - `experiment_id` (UUID, Foreign Key)
   - `variant` (String)
   - `conversion_count` (Integer)
   - `views_count` (Integer)
   - `created_at` (DateTime)

### Access Patterns:
- Frequent writes to the Events table, and reads from Experiments and Results tables for analytics.

### Trade-offs Considered:
1. **Consistency vs. Availability**: By opting for an eventually consistent model with Cassandra, we ensure high availability but may face slight delays in data accuracy. This is acceptable for A/B testing where real-time analysis is not critical.
2. **Caching vs. Freshness**: Utilizing Redis for caching results improves performance, but we must manage cache invalidation to ensure freshness of data. A TTL (time-to-live) strategy can be implemented for cache entries.
3. **Microservices vs. Complexity**: While microservices promote scalability and independent deployment, they introduce complexity in deployment and monitoring. We will use Kubernetes for orchestration to mitigate this.
4. **Event Processing Latency**: To manage the latency of event processing, we may consider batching events before writing to the database, thus reducing the number of write operations but increasing the time taken to reflect results.

Design a A/B Testing for Oracle

Oracle

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements:

Capacity Estimation

Submit Your Answer

Oracle Machine Learning Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search