Design a distributed A/B Testing System

Question

Design a distributed a/b testing system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

- **Functional Requirements:**  
  1. **Experiment Creation**: Users should be able to create A/B tests with multiple variants.  
  2. **Traffic Allocation**: The system must allocate traffic to different variants according to predefined percentages.  
  3. **Data Collection**: Capture user interactions and conversion metrics for each variant.  
  4. **Results Dashboard**: Provide analytics on the performance of variants over time, including statistical significance.  
  5. **API Integration**: Expose REST APIs for clients to manage tests and retrieve results.

- **Non-Functional Requirements:**  
  1. **Scalability**: Must handle millions of requests per day with minimal latency.  
  2. **High Availability**: System should be operational 99.9% of the time.  
  3. **Consistency**: Ensure that users see the same variant during the test, regardless of their session.  
  4. **Performance**: Response time for variant allocation should be under 50ms.  
  5. **Security**: Authenticate API requests and secure user data.

Assuming 10 million users per day, and each user interacts with the A/B testing system once, we can estimate the following:  
- **Requests Per Second**: 10 million users / (24 hours * 3600 seconds) = ~115.74 requests per second.  
- **Data Storage**: If each user interaction generates 1 KB of data (user ID, variant ID, timestamp), we need to store 10 million KB, or approximately 9.54 GB per day.  
- **Scaling**: To handle peak loads, we should design for 3x capacity, or about 347 requests per second and around 28.62 GB per day.

The distributed A/B Testing System will consist of the following components:  
1. **Client Application**: Frontend application that communicates with the A/B testing API.  
2. **Load Balancer**: Distributes incoming requests to multiple API servers for scaling.  
3. **API Gateway**: Routes requests to the appropriate microservices.  
4. **Variant Allocation Service**: Handles the logic for allocating users to different test variants.  
5. **Database (MongoDB)**: Stores user interaction data, test configurations, and results.  
6. **Analytics Service**: Aggregates data and performs statistical analysis on test results.  
7. **Monitoring and Logging**: Tools like Prometheus and ELK Stack for real-time monitoring and logging.

**Technology Choices**:  
- **Load Balancer**: NGINX or AWS Elastic Load Balancer.  
- **API Gateway**: Kong or AWS API Gateway.  
- **Database**: MongoDB for its flexibility and scalability.  
- **Analytics**: Apache Spark for processing large datasets efficiently.

The schema design for MongoDB will include:  
1. **Users Collection**:  
   - `userId`: String (unique identifier)  
   - `sessionId`: String  
   - `assignedVariant`: String  
   - `timestamp`: Date

2. **Tests Collection**:  
   - `testId`: String (unique identifier)  
   - `variants`: Array of Objects (each containing `variantId`, `allocationPercentage`)  
   - `startDate`: Date  
   - `endDate`: Date

3. **Results Collection**:  
   - `testId`: String  
   - `variantId`: String  
   - `conversionCount`: Integer  
   - `totalCount`: Integer  
   - `timestamp`: Date

**Access Patterns**:  
- Queries will frequently fetch user data based on `userId` and `testId`, thus indexing on these fields will improve performance.  
- Aggregations for analytics will be optimized for read-heavy operations.

1. **Consistency vs. Availability**:  
   - Choosing eventual consistency for analytics data may improve performance but could delay insights. Strong consistency on user variant assignments ensures users see the same variant during their session.

2. **Database Choice**:  
   - MongoDB provides flexibility with its document model, which is suitable for storing user interactions. However, it may require additional work for complex aggregations compared to traditional SQL databases.

3. **Load Balancing**:  
   - Implementing a round-robin or least-connections strategy can optimize resource use, but may lead to uneven load distribution during peak traffic times.

4. **Caching**:  
   - Using Redis for caching variant assignments can reduce database load and improve performance. However, it adds complexity in terms of cache invalidation.

5. **Monitoring**:  
   - Setting up comprehensive monitoring with alerts can detect issues early but may introduce overhead in terms of additional infrastructure and management.

Design a distributed A/B Testing System

MongoDB

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Capacity Estimation

Submit Your Answer

MongoDB Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search