Architect a multi-tenant Order Processing Engine

Question

Design a multi-tenant order processing system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

- **Functional Requirements:**  
  1. **Multi-Tenancy:** Support multiple clients, ensuring data isolation and security.  
  2. **Order Processing:** Handle create, read, update, and delete (CRUD) operations for orders.  
  3. **Real-time Processing:** Process orders in real-time with low latency.  
  4. **Notification System:** Notify clients of order status changes via WebSockets or push notifications.  
  5. **Audit Logging:** Maintain logs of all order-related activities for compliance and debugging.  
  
- **Non-Functional Requirements:**  
  1. **Scalability:** The system should scale horizontally to handle millions of requests per second.  
  2. **Availability:** 99.99% uptime with fault tolerance mechanisms in place.  
  3. **Security:** Implement authentication and authorization for each tenant, adhering to industry standards.  
  4. **Performance:** Response time for order processing should be under 100 milliseconds.  
  5. **Consistency:** Ensure eventual consistency for order data while prioritizing availability.

- **Assumptions:**  
  1. **Client Base:** Assume 100 tenants with an average of 10,000 orders per day per tenant.  
  2. **Peak Load:** During peak trading hours, assume 50% of daily orders are placed in a 1-hour window.  
  3. **Total Requests:** 100 tenants x 10,000 orders = 1,000,000 orders per day; during peak: 500,000 orders/hour (or ~138 orders/second).  
  4. **Message Queues:** If each order generates a confirmation or notification, we may see an additional load of ~138 messages/second.  
  5. **Total Throughput:** The system needs to handle at least 276 requests per second (orders + notifications) during peak times.

- **Component Diagram:**  
  ![High-Level Architecture](https://www.example.com/architecture-diagram)  
  - **API Gateway:** Handles incoming requests and routes them to appropriate services.  
  - **Order Service:** Main service for processing order requests; implemented in Node.js for non-blocking I/O.  
  - **Message Queue (e.g., RabbitMQ):** For asynchronous processing of orders and notifications.  
  - **Database (e.g., Amazon RDS with PostgreSQL):** For storing orders, utilizing a multi-tenant schema for logical separation.  
  - **Cache Layer (e.g., Redis):** For caching frequent queries and reducing database load.  
  - **Notification Service:** Publishes order status updates to tenants through a WebSocket connection.

- **Schema Design:**  
  1. **Order Table:**  
     - `id`: UUID (Primary Key)  
     - `tenant_id`: UUID (Foreign Key)  
     - `status`: ENUM('Pending', 'Completed', 'Cancelled')  
     - `created_at`: TIMESTAMP  
     - `updated_at`: TIMESTAMP  
  2. **Tenant Table:**  
     - `tenant_id`: UUID (Primary Key)  
     - `name`: VARCHAR  
     - `api_key`: VARCHAR (for authentication)  
  3. **Audit Log Table:**  
     - `log_id`: UUID (Primary Key)  
     - `order_id`: UUID (Foreign Key)  
     - `event`: VARCHAR  
     - `timestamp`: TIMESTAMP  
  
- **Access Patterns:**  
  - Frequent reads for order status (cached in Redis), infrequent updates to the order table.

- **Trade-offs in Consistency, Availability, and Performance:**  
  1. **Eventual Consistency:**  
     - Favoring availability by allowing temporary inconsistencies during peak loads, ensuring all orders eventually converge to a consistent state.  
  2. **Caching:**  
     - Using Redis for caching leads to improved performance but risks stale data.  
  3. **Asynchronous Processing:**  
     - By using RabbitMQ for notifications, we can handle bursts of traffic but delay the user’s experience for order confirmations.  
  4. **Database Scaling:**  
     - Choosing a relational database like PostgreSQL provides strong consistency but may become a bottleneck; sharding may be necessary for horizontal scaling.  
  5. **Security vs. Performance:**  
     - Implementing stringent security measures (like OAuth) may add latency but is crucial for protecting sensitive financial data.

Architect a multi-tenant Order Processing Engine

Goldman Sachs

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Capacity Estimation

Submit Your Answer

Goldman Sachs Product Manager Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search