Design A/B Testing Infrastructure for microservices

Question

Design a geo-distributed a/b testing system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements
1. **User Segmentation**: Ability to define user segments for A/B testing based on demographics, behavior, etc.
2. **Experiment Creation**: Interface for product teams to create and manage experiments, including defining control and variant groups.
3. **Traffic Distribution**: Real-time traffic routing to the appropriate experiment variants based on predefined rules.
4. **Analytics Dashboard**: Provide insights and analytics on A/B test performance, including conversion rates, user engagement, etc.
5. **Feedback Loop**: Ability to pause or terminate experiments based on statistical significance thresholds.

### Non-Functional Requirements
1. **Scalability**: Must handle millions of requests per second with low latency.
2. **Geo-Distribution**: Ensure low-latency access across different regions while maintaining consistency.
3. **High Availability**: 99.99% uptime, with minimal downtime during deployments.
4. **Security**: Ensure user data is protected, with encryption in transit and at rest.
5. **Monitoring and Alerting**: Real-time monitoring of experiment performance, with alerts for anomalies.

### Back-of-Envelope Calculations
To estimate capacity, assume we have:
- **100 million users** actively participating in A/B tests.
- On average, each user interacts with the system **10 times per day**.
- This leads to a total of **1 billion requests per day**, or approximately **11,574 requests per second (QPS)**.

For storage, assuming each experiment record (including metadata and results) is roughly **1 KB**, with **1 million experiments** running concurrently, we can estimate:
- **Storage requirement** = 1 million experiments * 1 KB = **1 GB**. This is just for active experiments; historical data will require additional space.

Considering logs and metrics, we could anticipate an additional **10 GB** of storage per day, leading to a total of **3.65 TB/year**.

### Component Diagram and Technology Choices
The architecture consists of the following components:
1. **API Gateway**: Using **AWS API Gateway** to handle incoming requests, route them to the appropriate microservices, and manage throttling.
2. **Experiment Service**: A microservice built with **Node.js** to handle experiment creation, management, and traffic distribution. This service would utilize **Redis** for caching active experiments for quick access.
3. **User Service**: A microservice that manages user profiles and segmentation, built with **Spring Boot** and connected to a **PostgreSQL** database for structured user data storage.
4. **Analytics Service**: A data processing service using **Apache Kafka** for streaming data from experiments to a processing pipeline managed by **Apache Spark** for real-time analytics.
5. **Monitoring and Logging**: Implementing **Prometheus** for monitoring and **ELK Stack** (Elasticsearch, Logstash, Kibana) for logging and visualizing system performance.

### Schema Design
- **Experiments Table**
  - `experiment_id` (Primary Key, UUID)
  - `name` (String)
  - `description` (Text)
  - `start_time` (Timestamp)
  - `end_time` (Timestamp)
  - `variant_a_id` (Foreign Key)
  - `variant_b_id` (Foreign Key)
  - `status` (Enum: RUNNING, PAUSED, COMPLETED)

- **Variants Table**
  - `variant_id` (Primary Key, UUID)
  - `experiment_id` (Foreign Key)
  - `variant_name` (String)
  - `weight` (Integer)

- **User Experiments Table**
  - `user_id` (Primary Key, UUID)
  - `experiment_id` (Foreign Key)
  - `variant_id` (Foreign Key)
  - `timestamp` (Timestamp)

This model supports quick lookups for user assignments to experiments and variants, facilitating fast traffic routing.

### Architectural Decisions
1. **Consistency vs. Availability**: Opting for eventual consistency in user assignment to experiments to enhance availability and reduce latency. This means users might see slightly different experiment variants in rare cases, but overall system responsiveness is prioritized.
2. **Caching**: Using Redis for caching active experiments to reduce database load. The trade-off here is the potential for stale data if caches are not invalidated promptly.
3. **Geo-Distribution**: Replicating data across regions for low-latency access versus managing data synchronization complexities. Utilizing a multi-region setup in AWS (with **DynamoDB Global Tables**) can help maintain data locality for users.
4. **Monitoring**: Implementing real-time monitoring comes with overhead, but it is essential for quick failure detection and performance tuning. Balancing the granularity of metrics collected with the performance impact on the system is critical.

Design A/B Testing Infrastructure for microservices

Plaid

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Back-of-Envelope Calculations

Submit Your Answer

Plaid Product Manager Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search