Design a A/B Testing Service

Question

Design a real-time a/b testing system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements
1. **Experiment Creation**: Users can create A/B tests with multiple variants and targeting rules.
2. **Real-time Metrics**: Display metrics like conversion rates, click-through rates, and user engagement in real-time.
3. **User Segmentation**: Ability to segment users based on characteristics (e.g., geography, device).
4. **Reporting**: Generate historical reports for A/B tests, showing performance over time.
5. **Rollout Management**: Support gradual rollout of variants based on performance.

### Non-Functional Requirements
1. **Scalability**: Handle millions of concurrent users and requests.
2. **Low Latency**: Response time for serving experiments should be under 100ms.
3. **Data Consistency**: Ensure users consistently see the same variant during their session.
4. **High Availability**: System should be available 99.99% of the time.
5. **Security**: Implement OAuth for user authentication and secure data transmission.

### Back-of-Envelope Calculations
Assuming Splunk has an active user base of 10 million, with an estimated 10% engaging in A/B testing at any point:
- **Daily Active Users (DAU)**: 1 million
- **Requests per User**: Each user might generate around 5 requests (creating tests, checking metrics, etc.)
- **Total Requests per Day**: 1 million users * 5 requests = 5 million requests/day
- **Requests per Second (RPS)**: For peak traffic, assume 20% of DAU is active in a given minute:
  - 200,000 users * 5 requests = 1 million requests/minute
  - ~17,000 RPS

### Storage Requirements
- **Metrics Storage**: Assuming each metric for an A/B test is about 1 KB, with 10 tests running simultaneously:
  - 10 tests * 1 KB * 10 million users = 10 GB per day. Considering a 30-day retention, around 300 GB/month.

### Component Diagram
1. **Frontend**: React.js for user interfaces, allowing real-time updates.
2. **API Gateway**: AWS API Gateway to manage API calls and routing.
3. **Microservices**:
   - **Experiment Service**: Manages creation and lifecycle of A/B tests.
   - **Metrics Service**: Collects and aggregates metrics for A/B tests.
4. **Database**: 
   - **Primary**: Amazon DynamoDB for real-time, scalable data storage (user interactions).
   - **Analytics**: Amazon Redshift for historical data analysis and reporting.
5. **Caching**: Redis for in-memory caching of frequently accessed A/B test configurations.
6. **Load Balancer**: AWS Elastic Load Balancer to distribute incoming requests.
7. **Monitoring**: Prometheus and Grafana for performance monitoring and alerting.

### Schema Design
1. **Experiment Table**:  
   - `experiment_id` (PK)  
   - `name`  
   - `variants` (JSON)  
   - `start_time`  
   - `end_time`  
   - `targeting_rules` (JSON)  
2. **User Assignment Table**:  
   - `user_id` (PK)  
   - `experiment_id` (FK)  
   - `variant_id`  
   - `timestamp`  
3. **Metrics Table**:  
   - `experiment_id` (FK)  
   - `timestamp`  
   - `conversion_count`  
   - `click_count`  
   - `user_count`

### Access Patterns
- Frequent reads for variant assignments and metrics should be optimized with the caching layer.

### Architectural Decisions
1. **Consistency vs. Availability**: Opting for eventual consistency in metrics reporting allows for higher availability and scalability, which is critical for real-time analytics.
2. **Caching Strategies**: Using Redis for caching experiment configurations provides low-latency access; however, it introduces a cache invalidation challenge.
3. **Data Storage Choices**: DynamoDB's NoSQL design offers flexibility and scalability but may complicate complex querying compared to SQL-based systems. The choice of Redshift balances this by allowing historical data analysis.
4. **Monitoring & Alerting**: Building a robust monitoring solution using Prometheus ensures operational excellence but requires additional effort in setting up metrics collection and alert configurations.
5. **Phased Rollout**: Start with an MVP handling basic experiments and metrics, then incrementally add features like user segmentation and complex targeting rules based on user feedback and performance metrics.

Design a A/B Testing Service

Splunk

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Back-of-Envelope Calculations

Submit Your Answer

Splunk Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search