Architect a real-time Monitoring Engine

Question

Design a real-time monitoring system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements
1. **Real-Time Data Ingestion:** The system must ingest data from various sources (e.g., trading platforms, market feeds) with minimal latency.
2. **Monitoring and Alerting:** Provide real-time alerts based on predefined thresholds for various metrics (e.g., price volatility, trade execution times).
3. **Dashboard for Visualization:** A user interface to visualize metrics and alerts in real-time.
4. **Model Retraining:** Automatically retrain models based on new data and performance metrics.

### Non-Functional Requirements
1. **Scalability:** The system should handle millions of requests per second as trading volumes increase.
2. **Low Latency:** Targeting a sub-second response time for data processing and alerting.
3. **High Availability:** The system must be operational 99.99% of the time to ensure continuous monitoring.
4. **Data Consistency:** Ensure the accuracy of data processed for real-time alerts, prioritizing consistency over availability during critical operations.

Assuming HRT processes an average of 10 million trades per day with each trade generating an average of 10 data points:
1. **Total Data Points:** 10 million trades * 10 data points = 100 million data points per day.
2. **Per Second Ingestion Rate:** 100 million data points / (24 * 60 * 60) ≈ 1.16k data points per second.
3. **Real-Time Alerts:** If each trade generates an alert on average every 10 seconds, we would expect 1 million alerts per day, or about 11.6 alerts per second.
4. **Overall Request Handling:** Accounting for both data ingestion and alerting, the system needs to handle around 1.2k data points + 11.6 alerts ≈ 1.3k requests per second.

### Component Diagram
1. **Data Ingestion Layer:** Use **Apache Kafka** for real-time data streaming from multiple sources.
2. **Processing Layer:** Implement **Apache Flink** for stream processing to handle real-time computations on incoming data.
3. **Feature Store:** Utilize **Feast** for managing features used in machine learning models, allowing for both batch and real-time access.
4. **Model Serving:** Employ **TensorFlow Serving** for real-time model inference with low latency.
5. **Database:** Use **Amazon DynamoDB** for storing processed metrics and alerts, ensuring high availability and scalability.
6. **Dashboard:** Build a web-based dashboard using **React** to visualize metrics and alerts for users in real-time.

### Schema Design
1. **Trade Data Table:**
   - **trade_id (PK)**: UUID
   - **timestamp**: DateTime
   - **price**: Float
   - **volume**: Float
   - **exchange**: String

2. **Alerts Table:**
   - **alert_id (PK)**: UUID
   - **trade_id (FK)**: References Trade Data
   - **alert_type**: String (e.g., high volatility)
   - **alert_timestamp**: DateTime
   - **status**: String (e.g., triggered, acknowledged)

3. **Feature Store Schema:**
   - **feature_id (PK)**: UUID
   - **trade_id (FK)**: References Trade Data
   - **feature_name**: String
   - **feature_value**: Float
   - **feature_timestamp**: DateTime

This schema supports efficient retrieval of trade data for monitoring and alert generation.

1. **Consistency vs. Availability:** In a real-time trading environment, prioritizing consistency is critical. While using **eventual consistency** for alerting might increase availability, it can lead to false alerts. We’ll enforce strong consistency for critical alerts but may allow eventual consistency for less critical data.
2. **Complexity vs. Latency:** Using complex models can improve accuracy but may introduce latency. We will prefer simpler models (e.g., linear regression) for real-time predictions while reserving more complex models for offline analysis and retraining.
3. **Feature Engineering Trade-offs:** Extensive feature engineering may improve model performance but can slow down processing. We will implement a balance by focusing on essential features that provide high value while ensuring fast processing times.

Architect a real-time Monitoring Engine

HRT

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Submit Your Answer

HRT Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search