Build a event-driven Analytics Pipeline

Question

Design a event-driven analytics system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements
1. **Event Ingestion**: The system must be capable of ingesting events from various sources like user actions, system logs, and external APIs.
2. **Real-time Analytics**: The system should provide real-time analytics capabilities, processing events within seconds and delivering insights instantly.
3. **User Interface**: A dashboard for users to visualize analytics data in real-time, including graphs and charts.
4. **Data Storage**: Event data should be stored for historical analysis, with a retention policy that allows users to query data from the past 30 days.
5. **API for Data Access**: Provide a RESTful API for external systems to access aggregated analytics data.

### Non-Functional Requirements
1. **Scalability**: The system should handle millions of events per minute.
2. **Availability**: Aim for 99.99% uptime to ensure continuous access to analytics.
3. **Latency**: The system must process events with an average latency of under 5 seconds.
4. **Fault Tolerance**: The system should continue operating smoothly even in case of component failures.

### Back-of-Envelope Calculations
- **Estimated Events Per Minute**: If Morgan Stanley anticipates 10 million events per day, that translates to approximately 6,944 events per minute.
- **Peak Load**: During peak times, assume a spike to 5x this load, resulting in 34,720 events per minute.
- **Data Size**: If each event averages 1KB (including metadata), then:
  - Daily Data Volume: 10 million events * 1KB = 10GB/day
  - Peak Load Data Volume: 34,720 events * 1KB = ~34.7MB/minute
- **Storage Needs**: For 30 days of historical data, the total storage requirement would be 300GB (10GB/day * 30 days).

### Component Diagram
1. **Event Producers**: Sources generating events (web applications, mobile apps, etc.).
2. **Message Queue**: Use **Apache Kafka** for reliable event ingestion, capable of handling high throughput.
3. **Stream Processing**: Utilize **Apache Flink** for real-time event processing and analytics.
4. **Data Store**: Use **Amazon S3** for raw event storage and **Amazon Redshift** for structured querying and analytics.
5. **Cache Layer**: Implement **Redis** for caching frequently accessed analytics results.
6. **Dashboard and API Layer**: Build a web interface using **React** and expose analytics data through a REST API built with **Node.js**.
7. **Monitoring**: Use **Prometheus** and **Grafana** for system performance monitoring.

### Schema Design
1. **Event Table** (in Redshift):
   - `event_id` (UUID, Primary Key)
   - `timestamp` (TIMESTAMP)
   - `user_id` (VARCHAR)
   - `event_type` (VARCHAR)
   - `metadata` (JSONB)

2. **Aggregated Analytics Table**:
   - `aggregate_id` (UUID, Primary Key)
   - `event_type` (VARCHAR)
   - `counts` (INTEGER)
   - `timestamp` (TIMESTAMP)

### Access Patterns
- Frequent reads on the aggregated analytics table for dashboard queries.
- Raw event data accessed less frequently for deep-dive analysis.

### Architectural Decisions
1. **Consistency vs. Availability**: By using Kafka, we favor availability and partition tolerance (AP model of CAP theorem) over immediate consistency. Events can be processed out of order, but eventually become consistent.
2. **Caching Strategy**: Choosing Redis for caching provides fast access to frequently queried data but introduces complexity in cache invalidation. We opt for a time-to-live (TTL) strategy for cache entries to ensure freshness.
3. **Data Storage**: Using Amazon S3 for raw data offers durability and cost-effectiveness, but querying raw data is slower. The trade-off is between faster analytics using Redshift vs. slower access to raw data in S3.
4. **Failure Handling**: Implementing a circuit breaker pattern in the API layer allows the system to gracefully handle downstream failures without crashing the entire service.

Build a event-driven Analytics Pipeline

Morgan Stanley

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Back-of-Envelope Calculations

Submit Your Answer

Morgan Stanley Product Manager Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search