Design a large-scale A/B Testing Platform

Question

Design a distributed a/b testing system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements
1. **Experiment Creation**: Users should be able to create A/B tests specifying variations and user segments.
2. **Traffic Allocation**: The system must route traffic to different variations based on predefined percentages.
3. **Data Collection**: Collect user interaction data in real-time for statistical analysis.
4. **Result Analysis**: Provide an interface for users to visualize results and metrics (e.g., conversion rates).

### Non-Functional Requirements
1. **Scalability**: The platform should handle millions of concurrent requests with horizontal scaling.
2. **Latency**: Response times for experiment queries must be under 100ms for a seamless user experience.
3. **Availability**: The system should maintain 99.9% uptime, allowing for background processing of data.
4. **Fault Tolerance**: Ensure that the system can recover from failures without data loss.

Assuming Bloomberg handles 10 million users daily, and each user might trigger 5 requests to the A/B testing platform:
- **Total Requests per Day**: 10 million users * 5 requests = 50 million requests/day
- **Requests per Second (RPS)**: 50 million requests / 86400 seconds = ~578 requests/second.
- For peak times, we estimate a 3x increase in load, leading to ~1734 RPS. Thus, the system should be designed to handle at least 2000 RPS to accommodate peak loads and future growth.

The architecture consists of the following components:
1. **Frontend**: A web interface built with React for creating and managing A/B tests.
2. **API Gateway**: NGINX to route requests to appropriate microservices.
3. **Microservices**: 
   - **Experiment Service**: Handles experiment creation and management (Node.js).
   - **Traffic Routing Service**: Implements traffic allocation logic (Go).
   - **Data Processing Service**: Collects and processes user interactions (Python).
4. **Database**: PostgreSQL for structured data storage, with Redis for caching frequently accessed data.
5. **Message Queue**: RabbitMQ for decoupling data processing from request handling, ensuring fault tolerance and async processing.

### Schema Design
1. **Experiments Table**:
   - `experiment_id`: UUID (Primary Key)
   - `name`: String
   - `variations`: JSON (stores variations)
   - `created_at`: Timestamp
   - `status`: Enum (active, paused, completed)

2. **UserInteractions Table**:
   - `interaction_id`: UUID (Primary Key)
   - `experiment_id`: UUID (Foreign Key)
   - `user_id`: String
   - `variation`: String
   - `timestamp`: Timestamp

### Access Patterns
- Frequent reads for experiment status and variations, necessitating caching strategies (Redis).
- Write-heavy interactions for real-time data collection.

1. **Consistency vs. Availability**: Opting for eventual consistency in data collection allows the system to remain available during high load, sacrificing immediate consistency for better performance.
2. **Caching Strategy**: Using Redis introduces complexity in cache invalidation, but greatly improves read performance. Trade-offs include stale data if not managed properly.
3. **Microservices vs. Monolith**: While microservices provide scalability and flexibility, they also increase operational overhead. A monolithic approach could simplify deployments but limit scalability.
4. **Synchronous vs. Asynchronous Processing**: Using RabbitMQ for async processing improves user experience but could delay the availability of real-time analytics, requiring careful monitoring of processing times.

Design a large-scale A/B Testing Platform

Bloomberg

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Submit Your Answer

Bloomberg Product Manager Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search