Design Order Processing Infrastructure for real-time analytics

Question

Design a event-driven order processing system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements:
1. **Event-Driven Processing:** The system should process order events in real-time, handling order creation, updates, and cancellations.
2. **Real-Time Analytics:** Provide analytics features such as order tracking, inventory management, and customer behavior analysis.
3. **Scalability:** Handle millions of requests per second with low latency.
4. **API Access:** Expose RESTful APIs for external systems to push and pull order data.

### Non-Functional Requirements:
1. **Availability:** Aim for 99.99% uptime to ensure continuous order processing.
2. **Consistency:** Implement eventual consistency for analytics features while ensuring strong consistency for order processing.
3. **Latency:** Maintain an average response time of under 100 ms for API calls.
4. **Security:** Ensure data is encrypted in transit and at rest, complying with PCI DSS standards.

### Back-of-Envelope Calculations:
1. **Expected Transactions:** Assume 1 million orders processed per day, translating to approximately 11.57 orders per second (QPS).
2. **Data Size:** Each order event may average around 2 KB (including metadata). For 1 million orders, the daily data size would be about 2 GB.
3. **Storage Requirements:** Assuming a retention period of 1 year, we will need approximately 730 GB for orders alone. Analytics data may require 2-3 times this, estimating total storage needs at around 2-3 TB annually.
4. **Bandwidth:** With an average payload size of 2 KB for order events, bandwidth required would be approximately 23.14 MB/hour for order processing, factoring in peak load times.

### Component Diagram:
- **Event Producers:** Microservices that generate order events (e.g., Order Service, Payment Service).
- **Message Broker:** Use Apache Kafka for real-time event streaming, ensuring durability and fault tolerance.
- **Order Processing Service:** A stateless service that consumes events from Kafka, processes them, and updates the database.
- **Database:** Use PostgreSQL for strong consistency in order data, with read replicas for analytics.
- **Analytics Engine:** A separate service that consumes events for real-time analytics using Apache Flink.
- **Cache Layer:** Use Redis for caching frequently accessed order data and analytics results.
- **API Gateway:** AWS API Gateway to handle API requests and rate limiting, ensuring scalability.

### Schema Design:
1. **Orders Table:**
   - `order_id` (PK, UUID)
   - `customer_id` (FK, UUID)
   - `status` (ENUM - Pending, Completed, Cancelled)
   - `created_at` (TIMESTAMP)
   - `updated_at` (TIMESTAMP)
   - `total_amount` (DECIMAL)

2. **Order Items Table:**
   - `order_item_id` (PK, UUID)
   - `order_id` (FK, UUID)
   - `product_id` (FK, UUID)
   - `quantity` (INTEGER)
   - `price` (DECIMAL)

3. **Analytics Table:**
   - `event_id` (PK, UUID)
   - `order_id` (FK, UUID)
   - `event_type` (ENUM - OrderCreated, OrderUpdated, OrderCancelled)
   - `timestamp` (TIMESTAMP)

### Access Patterns:
- Orders are frequently accessed for updates, requiring efficient indexing. Analytics queries will be optimized for read operations.

### Architectural Decisions:
1. **Consistency vs Availability:** Opting for eventual consistency for analytics allows faster processing, while maintaining strong consistency in the order processing service ensures accurate order states.
2. **Database Choice:** Using PostgreSQL for transactional integrity but introducing a separate analytics database to optimize read-heavy operations.
3. **Caching Strategy:** Implementing Redis for caching critical order data improves read performance but introduces complexity in cache invalidation strategies.
4. **Eventual Consistency:** Decisions around the delay in analytics updates are necessary; while this allows for real-time processing, it could lead to stale data in analytics dashboards. Balancing the frequency of analytics updates against performance is crucial.

### Monitoring and Alerting:
1. **Monitoring:** Use Prometheus and Grafana for monitoring system health (latency, error rates).
2. **Alerting:** Set up alerts for high error rates or latency spikes, ensuring immediate attention to potential system failures.

Design Order Processing Infrastructure for real-time analytics

Mastercard

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements:

Capacity Estimation

Back-of-Envelope Calculations:

Submit Your Answer

Mastercard Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search