Build a fault-tolerant Payment Pipeline

Question

Design a fault-tolerant payment system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements
1. **Payment Processing**: Support various payment methods (credit/debit cards, digital wallets, etc.).
2. **Transaction Management**: Ability to initiate, confirm, and refund transactions.
3. **Fraud Detection**: Implement mechanisms to detect and prevent fraudulent transactions.
4. **Notifications**: Provide real-time notifications for transaction status (success, failure, pending).

### Non-Functional Requirements
1. **Fault Tolerance**: The system should remain operational despite failures in components.
2. **Scalability**: Handle millions of concurrent payment requests, especially during peak hours.
3. **Latency**: Average transaction processing time should be under 200ms.
4. **Data Consistency**: Ensure eventual consistency across distributed components.
5. **Compliance**: Adhere to PCI DSS regulations for secure handling of payment information.

Assuming Doordash processes approximately 10 million orders daily:
- **Requests per second (RPS)**: 10 million orders / 24 hours = ~115,740 RPS.
- **Peak Load**: Consider a peak where 20% of daily transactions occur in one hour, leading to ~640,000 RPS.
- **Throughput Requirements**: Each payment transaction might require 5 API calls (initiate, confirm, fraud check, etc.), leading to a capacity requirement of ~3.2 million requests per second at peak.
- **Database Load**: If each transaction requires a write operation, we need a database capable of handling ~3 million writes per second.

### Component Diagram
- **API Gateway**: NGINX for load balancing incoming requests.
- **Microservices**:
  - **Payment Service**: Handles payment processing logic.
  - **Fraud Detection Service**: Analyzes transactions for fraud.
  - **Notification Service**: Sends transaction status updates.
- **Database**: Sharded PostgreSQL for transaction storage, with each shard handling a subset of transactions based on user geography.
- **Caching Layer**: Redis for caching frequently accessed data like user payment methods.
- **Queue**: RabbitMQ for asynchronous processing of transactions to decouple services.
- **Monitoring**: Prometheus and Grafana for real-time monitoring and alerting.

### Schema Design
1. **Users Table**:  
   - `user_id (PK)`  
   - `email`  
   - `payment_methods (JSONB)`  
   - `created_at`

2. **Transactions Table**:  
   - `transaction_id (PK)`  
   - `user_id (FK)`  
   - `amount`  
   - `status (ENUM: pending, success, failed)`  
   - `created_at`  
   - `updated_at`

3. **Fraud Alerts Table**:  
   - `alert_id (PK)`  
   - `transaction_id (FK)`  
   - `reason`  
   - `created_at`

### Access Patterns
- Frequent reads for user payment methods and transaction statuses will benefit from caching to reduce database load.

1. **Consistency vs. Availability**: Choosing eventual consistency for payments to enhance availability during peak loads. This means users may see a pending status for a short time post-transaction.
2. **Latency vs. Complexity**: Implementing a microservices architecture increases latency due to network calls but improves modularity and fault isolation.
3. **Simplicity vs. Scalability**: Initial MVP could utilize a monolithic architecture for rapid deployment, transitioning to microservices as scaling demands increase.
4. **Monitoring vs. Performance**: Extensive monitoring can introduce performance overhead; we will selectively monitor critical paths while aiming to keep the system responsive.

Build a fault-tolerant Payment Pipeline

Doordash

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Submit Your Answer

Doordash Product Manager Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search