Design a A/B Testing Service

Question

Design a low-latency a/b testing system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements:
1. **Experiment Creation**: Users must be able to create A/B tests by specifying variants, metrics, and targeting.
2. **Randomization**: Users should be randomly assigned to control and variant groups.
3. **Metrics Tracking**: The system should track user interactions and metrics for each variant, such as click-through rates and conversion rates.
4. **Results Dashboard**: Provide a dashboard for users to view test results, including statistical significance.

### Non-Functional Requirements:
1. **Low Latency**: The system should respond to requests in under 100ms to ensure real-time user experience.
2. **Scalability**: The service should handle millions of requests per day, scaling horizontally as needed.
3. **Fault Tolerance**: The system must gracefully handle failures and ensure data integrity during outages.
4. **Security**: Implement security measures to protect test data and user information, including authentication and authorization.

Assuming Redfin has around 10 million active users and an average of 10 A/B tests running concurrently, we can estimate:
- **QPS**: If each user interacts with the A/B testing service 5 times a day, the total QPS would be:
  - Total interactions = 10 million users * 5 interactions = 50 million interactions per day.
  - QPS = 50 million / (24 * 60 * 60) ≈ 578 QPS.

- **Storage**: Each A/B test could require 1 MB of metadata, and with 10 million users participating across 10 tests, that's:
  - Total storage = 10 million users * 10 tests * 1 MB = 10 TB of metadata.

- **Bandwidth**: Assuming each request/response payload is about 1 KB, the bandwidth would be:
  - Total daily traffic = 50 million interactions * 1 KB = 50 GB/day.

### Component Diagram:
- **User Interface**: Web and mobile applications where users can create and view A/B tests.
- **API Gateway**: Handles incoming requests, performs load balancing, and directs traffic to appropriate services.
- **A/B Test Service**: Manages test creation, randomization logic, and result computation.
- **Metrics Collection Service**: Asynchronously collects metrics from user interactions and stores them.
- **Database**: A NoSQL database (e.g., MongoDB) for storing A/B test configurations and user interaction data.
- **Cache**: Redis for caching recent A/B test data to reduce database load and improve latency.
- **Message Queue**: Kafka for decoupling the metrics processing from the A/B test service to handle spikes in traffic.
- **Monitoring and Alerting**: Tools like Prometheus and Grafana for monitoring system health and performance.

### Schema Design:
1. **A/B Test Collection**:  
   - `test_id`: String (Unique identifier)  
   - `variants`: Array of strings (Variant identifiers)  
   - `metrics`: Document (Data structure for metrics like clicks, conversions)  
   - `status`: String (Active, Completed, etc.)  
   - `created_at`: Timestamp

2. **User Interaction Collection**:  
   - `interaction_id`: String (Unique identifier)  
   - `user_id`: String  
   - `test_id`: String  
   - `variant`: String  
   - `timestamp`: Timestamp  
   - `event_type`: String (Click, Conversion, etc.)

### Access Patterns:
- Frequent reads for active tests and metrics.  
- Write-heavy during peak interaction times.

1. **Consistency vs. Availability**: 
   - By using a NoSQL database, we favor availability and partition tolerance over strict consistency. This is acceptable as slight staleness in metrics is tolerable for A/B tests.

2. **Synchronous vs. Asynchronous Processing**: 
   - The choice to use Kafka for metrics collection allows for decoupled and asynchronous processing, improving system responsiveness but introducing complexity in handling eventual consistency.

3. **Caching Strategy**: 
   - Using Redis for caching recent A/B test data reduces load on the database but might serve stale data. We have to implement cache invalidation strategies to ensure users see current tests.

4. **Database Choice**: 
   - Opting for a NoSQL solution provides flexibility in schema design and scalability but requires careful consideration of data relationships and transactions.

Design a A/B Testing Service

Redfin

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements:

Capacity Estimation

Submit Your Answer

Redfin Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search