Design a geo-distributed A/B Testing System

Question

Design a geo-distributed a/b testing system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

- **Functional Requirements:**  
  1. **User Segmentation:** Ability to segment users geographically for A/B testing.  
  2. **Experiment Creation:** Users can create, modify, and delete A/B tests through an API.  
  3. **Real-time Results:** Provide real-time analytics on experiment performance across different geographies.  
  4. **Traffic Routing:** Ability to route user requests to different variants based on experiment configuration.  
  5. **Monitoring & Alerts:** System should monitor experiment performance and alert on anomalies.

- **Non-Functional Requirements:**  
  1. **Scalability:** Must handle millions of requests per second with low latency.  
  2. **Availability:** 99.99% uptime across geo-distributed regions.  
  3. **Consistency:** Eventual consistency is acceptable for reporting results, but strong consistency is required for traffic routing.  
  4. **Latency:** Response time for user requests should be under 100ms.  
  5. **Security:** Comply with data protection regulations (e.g., GDPR) for user data.

- **User Base:** Assume 10 million active users.  
- **Requests per Second (RPS):** If each user generates roughly 10 requests per day, we expect about 1,157 RPS (10 million users / 86400 seconds).  
- **A/B Tests:** Assume 100 concurrent A/B tests, with each test serving 5 variants.  
- **Traffic Distribution:** Assuming uniform distribution, each variant would receive about 231 RPS (1,157 RPS / 5).  
- **Storage Needs:** If each experiment logs data (e.g., user IDs, variant IDs, timestamps) at an average of 100 bytes per request, total daily logs would be approximately 86.4 MB (1,157 RPS * 86400 seconds * 100 bytes).  
- **Total Storage:** For a month, this would be around 2.6 GB, not accounting for additional metadata and user segmentation data.

The architecture consists of the following components:  
- **API Gateway:** Handles incoming requests and routes them to appropriate services. Technology: AWS API Gateway.  
- **Load Balancer:** Distributes traffic across multiple instances of the A/B testing service. Technology: AWS Elastic Load Balancer.  
- **Experiment Service:** Manages A/B test creation, modification, and deletion. Technology: Node.js for API, deployed in AWS Lambda for scalability.  
- **Routing Service:** Determines which variant to serve based on user segmentation. Technology: Redis for caching variant assignments with TTL to refresh assignments periodically.  
- **Analytics Service:** Collects and processes experiment results. Technology: Apache Kafka for message queuing, Kafka Streams for real-time processing.  
- **Database:** Stores user data and experiment metadata. Technology: PostgreSQL for relational data with read replicas for scalability.  
- **Monitoring & Alerting:** Use Prometheus and Grafana for metrics collection and visualization, and AWS CloudWatch for alerting.

The data schema includes:  
- **Users Table:**  
  - `user_id` (PK, UUID)  
  - `geo_location` (STRING)  
  - `created_at` (TIMESTAMP)

- **Experiments Table:**  
  - `experiment_id` (PK, UUID)  
  - `name` (STRING)  
  - `created_at` (TIMESTAMP)  
  - `status` (ENUM: ACTIVE, INACTIVE)  
  - `variants` (JSONB)

- **Experiment Results Table:**  
  - `experiment_id` (FK, UUID)  
  - `user_id` (FK, UUID)  
  - `variant_id` (STRING)  
  - `timestamp` (TIMESTAMP)  
  - `conversion` (BOOLEAN)

- **Access Patterns:**  
  - Users will query their assigned variant and log results.  
  - Frequent reads from the `Experiments` and `Experiment Results` tables for real-time analytics.

- **Consistency vs Availability:**  
  - Chose eventual consistency for the analytics service to allow for high availability and scalability.  
  - Strong consistency for the Routing Service ensures users are consistently assigned to the correct variant.

- **Database vs Caching:**  
  - Implemented Redis caching for variant assignment to reduce database load and improve response times at the cost of potential staleness in variant assignments.

- **Scalability vs Complexity:**  
  - Using AWS Lambda for the Experiment Service improves scalability but adds complexity in terms of deployment and monitoring.

- **Real-time vs Batch Processing:**  
  - Real-time processing with Kafka for analytics allows immediate insights, while batch processing could simplify architecture but delay reporting.

Design a geo-distributed A/B Testing System

Plaid

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Capacity Estimation

Submit Your Answer

Plaid Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search