Design A/B Testing Infrastructure for global users

Question

Design a low-latency a/b testing system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements
1. **User Segmentation**: Ability to segment users based on various criteria (geography, device type, etc.).
2. **Experiment Creation**: Allow product managers to create A/B tests with control and variant groups.
3. **Real-time Metrics**: Provide real-time analytics on user behavior, conversion rates, and other KPIs.
4. **Rollout Controls**: Enable gradual rollout of experiments to manage exposure.
5. **API Integration**: Offer a robust API for client applications to trigger experiments and report outcomes.

### Non-Functional Requirements
1. **Low Latency**: Response time must be under 100ms for 95th percentile.
2. **Scalability**: Must support millions of requests per day, with horizontal scaling capabilities.
3. **High Availability**: System should be operational 99.99% of the time.
4. **Data Durability**: Ensure that experiment data is persistent and recoverable in case of failures.

Assuming PlanetScale has 10 million daily active users:
- **Peak Traffic Estimation**: Let's estimate peak traffic at 20% of users concurrently active, leading to 2 million simultaneous requests.
- **QPS Calculation**: If we assume an average user generates 2 requests per minute, the QPS would be:
  - 2 million users * 2 requests/minute / 60 seconds = ~66,667 QPS.
- **Data Storage**: Each A/B test could store metadata and user interactions:
  - If each test has 1,000 users and we run 1,000 tests, that’s 1 billion records. Assuming 100 bytes per record, total storage = 100GB.
- **Bandwidth**: If each experiment results in 1KB of data sent back, then for 1 million users, we expect 1TB of data transfer daily.

### Component Diagram
- **Load Balancer (Nginx)**: Distributes incoming traffic to application servers.
- **Application Servers (Node.js)**: Handle A/B test logic, user segmentation, and metrics collection.
- **Caching Layer (Redis)**: Cache user segments and experiment metadata to reduce database load.
- **Database (PlanetScale MySQL)**: Store user data, experiment details, and metrics records.
- **Analytics Service (Apache Kafka + Spark)**: Process real-time user interaction data and generate reports.
- **Monitoring (Prometheus + Grafana)**: Monitor system performance and alert on anomalies.

### Technology Choices
- **Node.js** for its event-driven architecture for handling high concurrency.
- **Redis** for fast, in-memory caching to reduce latency.
- **PlanetScale MySQL** for its scalability and built-in support for horizontal partitioning.

### Schema Design
1. **Users Table**: `user_id`, `user_segment`, `created_at`.
2. **Experiments Table**: `experiment_id`, `name`, `variant_a_id`, `variant_b_id`, `status`, `start_time`, `end_time`.
3. **User_Experiments Table**: `user_id`, `experiment_id`, `variant_id`, `timestamp`.
4. **Metrics Table**: `experiment_id`, `conversion_count`, `interaction_count`, `timestamp`.

### Access Patterns
- Frequent reads for user segments and experiment metadata; hence the use of Redis.
- Analytics performed in batch processes, requiring efficient writes to the Metrics table.

### Architectural Decisions
1. **Consistency vs Availability**: Chose eventual consistency for user experiment assignments to allow for low-latency responses. This means users might see different experiences based on timing but ensures high availability.
2. **Caching**: While caching experiment data in Redis speeds up access, it introduces complexity and potential stale data. We mitigate this by setting appropriate TTLs and invalidating the cache on updates.
3. **Partitioning Strategy**: Database sharding based on user geography to minimize latency for global users. This trade-off allows us to scale horizontally but adds complexity in cross-region queries.
4. **Monitoring**: Choosing Prometheus for monitoring gives us time-series data but requires additional setup for alerting. We need to ensure alerts are fine-tuned to avoid alert fatigue.

Design A/B Testing Infrastructure for global users

PlanetScale

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Submit Your Answer

PlanetScale Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search