Design a event-driven Feed Generation System

Question

Design a event-driven feed generation system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements:
1. **Event Ingestion**: The system must be able to ingest millions of events per day from various sources (e.g., trading platforms, market data feeds).
2. **Event Processing**: Events must be processed in real-time to generate feeds that reflect the latest market data.
3. **Feed Generation**: The system should generate different types of feeds (e.g., market updates, trade confirmations) based on processed events.
4. **Subscription Management**: Users should be able to subscribe to specific feeds based on their requirements.
5. **Error Handling**: The system should gracefully handle errors and provide notifications to users.

### Non-Functional Requirements:
1. **Scalability**: The system should scale horizontally to handle increasing loads without significant performance degradation.
2. **Availability**: The system should achieve 99.99% uptime to ensure continuous access to feeds.
3. **Latency**: The system should deliver feeds with a latency of less than 100 milliseconds.
4. **Data Consistency**: Event processing should ensure eventual consistency across different feeds.
5. **Monitoring and Logging**: The system should provide comprehensive monitoring, logging, and alerting capabilities to track system health and performance.

To estimate capacity, we can assume:
- **Event Rate**: 5 million events per day, which translates to approximately 58 events per second.
- **Feed Requests**: Each event may trigger 10 feed requests on average.
- **Total Requests**: Therefore, total feed requests = 5 million events * 10 = 50 million feed requests per day, or about 580 requests per second.

**Storage Needs**:
- Each feed entry is approximately 1KB. 
- Total storage for a day = 50 million * 1KB = ~47.7GB.
- For retention of 30 days, we need about 1.43TB of storage. This can be distributed across multiple nodes for redundancy and high availability.

### Component Diagram:
1. **Event Producer**: Various sources (trading platforms, APIs) that send events.
2. **Message Queue (Kafka)**: Acts as a buffer and provides durability for incoming events.
3. **Event Processor (Apache Flink)**: Real-time processing of events to generate feeds.
4. **Feed Storage (Cassandra)**: Stores generated feeds for quick access and retrieval.
5. **Feed Service (REST API)**: Exposes endpoints for clients to retrieve feeds and manage subscriptions.
6. **Monitoring (Prometheus + Grafana)**: For logging, metrics collection, and visualization.

### Technology Choices:
- **Kafka** for high-throughput message queuing.
- **Apache Flink** for real-time stream processing due to its low latency.
- **Cassandra** for scalable and distributed storage, suitable for time-series data.

### Schema Design:
1. **Event Table**:
   - `event_id`: UUID (Primary Key)
   - `event_type`: String
   - `timestamp`: DateTime
   - `payload`: JSON (to store event data)

2. **Feed Table**:
   - `feed_id`: UUID (Primary Key)
   - `feed_type`: String
   - `generated_at`: DateTime
   - `data`: JSON (contains the feed information)

3. **Subscription Table**:
   - `user_id`: UUID (Primary Key)
   - `feed_id`: UUID (Foreign Key)
   - `subscription_time`: DateTime

### Access Patterns:
- Events are ingested primarily for processing.
- Feeds are queried based on user subscriptions, which can be indexed for efficient retrieval.

### Architectural Decisions:
1. **Consistency vs. Availability**: Opting for eventual consistency with Kafka and Cassandra sacrifices immediate consistency for greater availability and scalability. This is acceptable in the context of market data feeds where slight delays are tolerable.
2. **Real-time Processing vs. Batch Processing**: Choosing Apache Flink for real-time processing ensures lower latency feeds at the cost of complexity in handling state and fault tolerance. Batch processing could introduce higher latency, which is not acceptable for this application.
3. **Message Queue Durability**: While Kafka provides durability, it can introduce latency. We need to balance between durability settings and performance, possibly configuring retention policies to optimize storage while retaining necessary data.
4. **Horizontal Scaling**: Both Kafka and Cassandra allow horizontal scaling, but require careful partitioning strategies to avoid hotspots, particularly in the feed generation process.

Design a event-driven Feed Generation System

JPMorgan

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements:

Capacity Estimation

Submit Your Answer

JPMorgan Machine Learning Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search