Build a scalable A/B Testing Pipeline

Question

Design a scalable a/b testing system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

- **Functional Requirements**:
  - Ability to create and manage A/B tests with multiple variations.
  - Real-time analytics dashboard for monitoring test performance.
  - User segmentation support for targeted experiments.
  - Automated statistical analysis to determine winner of A/B tests.

- **Non-Functional Requirements**:
  - **Scalability**: Must support millions of requests per second with low latency.
  - **Availability**: 99.99% uptime, with a focus on fault tolerance across regions.
  - **Consistency**: Eventual consistency for analytics data while ensuring strong consistency for user allocation.
  - **Performance**: Response time under 100ms for user requests and analytics queries.

Assuming Goldman Sachs manages approximately 10 million active users:
- **Daily Requests**: Each user generates about 5 requests for A/B tests daily.
- Total Requests per Day = 10M users * 5 requests = 50M requests/day.
- **Per Second Calculation**: 50M requests/day / 86400 seconds/day = ~578 requests/second.
- To accommodate spikes, we can estimate a 5x increase during peak times, leading to a requirement of around 2900 requests/second.
- Assuming a 50% read to write ratio, we might need to handle ~1450 reads and ~725 writes per second.

The architecture consists of:
1. **Load Balancer** (AWS Elastic Load Balancer) to distribute incoming traffic among multiple application servers.
2. **Application Layer**:
   - **API Gateway** (AWS API Gateway) for managing requests and rate limiting.
   - **Microservices** for handling A/B test creation, user allocation, and analytics processing (built with Node.js).
3. **Database Layer**:
   - **Primary Database** (Amazon Aurora) for strong consistency in user allocation and metadata storage.
   - **Analytics Database** (Amazon Redshift) for read-heavy analytics queries with eventual consistency.
4. **Caching Layer**: Redis for caching frequently accessed data and results.
5. **Queue System**: AWS SQS for handling asynchronous processing of analytics data.

The schema design will include:
- **Users Table**:
  - user_id (PK)
  - allocated_test_id (FK)
  - variation_id
  - created_at

- **Tests Table**:
  - test_id (PK)
  - name
  - status (active, completed)
  - created_at

- **Variations Table**:
  - variation_id (PK)
  - test_id (FK)
  - weight (percentage allocation)

- **Analytics Table**:
  - analytics_id (PK)
  - test_id (FK)
  - variation_id (FK)
  - conversion_count
  - total_impressions
  - created_at

This design supports efficient lookup for user allocation and allows for aggregation of analytics data.

1. **Consistency vs. Availability**:
   - We prioritize availability for user requests (eventual consistency in analytics) to ensure users always see a test variation. This means using a multi-region setup where data replication might be eventually consistent.

2. **Read vs. Write Optimization**:
   - The decision to use Amazon Aurora for writes ensures strong consistency, but we rely on Redshift for read-heavy analytics, which sacrifices some real-time data visibility.

3. **Caching Strategy**:
   - Using Redis for caching improves read performance but introduces complexity in cache invalidation strategies, especially during A/B test updates.

4. **Operational Concerns**:
   - Implementing chaos engineering to test failure modes, ensuring that the system can gracefully degrade without impacting the user experience. Monitoring and alerting (using AWS CloudWatch) will maintain operational excellence.

Build a scalable A/B Testing Pipeline

Goldman Sachs

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Capacity Estimation

Submit Your Answer

Goldman Sachs Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search