Architect a low-latency Logging Engine

Question

Design a low-latency logging system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements
1. **Real-time Logging**: Capture and store logs from millions of requests per second, ensuring minimal latency (under 100ms).
2. **Structured Data**: Logs should include structured information (timestamp, user ID, action type, response time, error codes).
3. **Search and Filter**: Ability to query logs based on various parameters (e.g., user ID, date range, action type).
4. **Analytics Dashboard**: Provide a UI for monitoring logs, trends, and anomalies in real-time.
5. **Integration with ML Models**: Logs should feed into machine learning models for real-time feedback and retraining.

### Non-Functional Requirements
1. **High Availability**: The logging system must be available 99.99% of the time.
2. **Scalability**: Should handle spikes in traffic, especially during promotional events or holidays.
3. **Data Retention**: Logs should be retained for at least 30 days for compliance and auditing purposes.
4. **Latency**: End-to-end logging should not exceed 100ms under peak loads.
5. **Data Quality**: Ensure high integrity and accuracy of log data with validation checks.

Assuming Instacart processes approximately 10 million requests per day, we estimate:
- **Requests per second**: 10 million requests / 86,400 seconds = ~116 requests/second.
- **Log Size**: If each log entry averages 1 KB (considering structured data), the daily log size would be:
  - 10 million requests * 1 KB = ~10 GB/day.
- **Storage Needs**: Over 30 days, we would need:
  - 10 GB/day * 30 days = ~300 GB of storage.
- **Throughput Requirement**: To handle bursts, we should consider a system that can process up to 1,000 requests/second during peak times, leading to a requirement for horizontally scalable storage solutions.

### Component Diagram
- **Log Producers**: Microservices generating logs (e.g., checkout, search, and inventory services).
- **Load Balancer**: Distributes incoming log requests across multiple instances of the logging service.
- **Logging Service**: A stateless service (using Node.js) that receives logs and writes them to a queue.
- **Message Queue**: Use Kafka for buffering logs to ensure durability and handle spikes in load.
- **Storage Layer**: Write logs to a NoSQL database (e.g., Amazon DynamoDB) for fast read and write capabilities.
- **Analytics and Monitoring**: Integrate tools like Grafana or Kibana for real-time dashboarding and querying.
- **ML Pipeline**: Logs fed into a feature store (e.g., Feast) for model training and real-time inference.

### Technology Choices
- **Load Balancer**: NGINX
- **Message Queue**: Apache Kafka
- **Database**: Amazon DynamoDB for low-latency access
- **Monitoring**: Grafana for visualization and alerting

### Schema Design
- **Log Table Structure**:
  - `log_id` (UUID, Primary Key)
  - `timestamp` (DATETIME)
  - `user_id` (STRING)
  - `action_type` (STRING)
  - `response_time` (INTEGER)
  - `error_code` (STRING, nullable)

### Access Patterns
- The schema allows for efficient querying by user ID and date range, supporting analytics and monitoring features. Indices will be created on `user_id` and `timestamp` to facilitate quick access to relevant logs.

1. **Consistency vs. Availability**: Choosing eventually consistent storage (DynamoDB) allows for higher availability and lower latency, but may lead to temporary discrepancies in log data.
2. **Real-time Processing vs. Batch Processing**: Real-time processing via Kafka allows immediate log availability but increases complexity. A hybrid approach can be considered for less critical logs.
3. **Data Quality vs. Performance**: Adding validation checks can introduce latency. A balance must be struck to ensure high data quality while maintaining the low-latency requirement.
4. **Cost vs. Scalability**: Using a managed service like DynamoDB provides scalability but may incur higher costs. Self-hosting a Cassandra cluster could reduce costs but requires more maintenance.

Architect a low-latency Logging Engine

Instacart

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Submit Your Answer

Instacart Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search