Build a high-throughput Payment Pipeline

Question

Design a high-throughput payment system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements:
1. **Transaction Processing**: The system must process payment transactions quickly and accurately, supporting various payment methods (credit cards, PayPal, etc.).
2. **Fraud Detection**: Integrate fraud detection mechanisms to flag suspicious transactions.
3. **Transaction History**: Provide APIs for users to retrieve their transaction history.
4. **User Notifications**: Send notifications for transaction confirmations and failures.
5. **Rate Limiting**: Implement rate limiting to prevent abuse of the payment endpoints.

### Non-Functional Requirements:
1. **Scalability**: The system should handle millions of transactions per day with the capability to scale horizontally.
2. **Availability**: Aim for 99.99% uptime by using distributed systems and failover strategies.
3. **Consistency**: Ensure eventual consistency for transaction data across distributed nodes.
4. **Latency**: Maintain a response time of <100ms for payment processing.

### Back-of-Envelope Calculations:
1. **Transaction Volume**: Assume Adobe processes 10 million transactions per day.
2. **Transactions per Second (TPS)**: 
   - 10,000,000 transactions / 86400 seconds = ~115.74 TPS.
3. **Peak Load**: Considering peak times, we can estimate a peak of 5x the average, leading to:
   - 115.74 TPS * 5 = ~578.7 TPS.
4. **Data Storage Needs**: If each transaction record is approximately 1KB, then: 
   - 10 million transactions * 1KB = 10GB of data per day, leading to 3.65TB per year.
5. **Database Reads/Writes**: Assume a read/write ratio of 70/30; this implies:
   - 578 TPS * 0.30 = ~173 writes/sec and 578 TPS * 0.70 = ~405 reads/sec.

### Component Diagram:
1. **API Gateway**: Handles incoming requests, provides rate limiting, and directs traffic to appropriate services.
2. **Payment Service**: Core service responsible for processing payments.
3. **Fraud Detection Service**: Asynchronously checks transactions against fraud rules.
4. **Database**: Use PostgreSQL for transactional integrity and reporting, with read replicas for scaling reads.
5. **Message Queue**: Implement Apache Kafka to handle asynchronous processing and communication between services.
6. **Cache Layer**: Use Redis for caching frequently accessed data like user transaction history.
7. **Notification Service**: Sends out confirmation and alert notifications via email/SMS.

### Technology Choices:
- **API Gateway**: AWS API Gateway
- **Payment Service**: Node.js with Express
- **Fraud Detection Service**: Python with machine learning models
- **Database**: PostgreSQL with AWS RDS
- **Message Queue**: Apache Kafka
- **Cache Layer**: Redis

### Schema Design:
1. **Transaction Table**:
   - `transaction_id (UUID) PRIMARY KEY`
   - `user_id (UUID) NOT NULL`
   - `amount (DECIMAL) NOT NULL`
   - `status (ENUM('PENDING', 'COMPLETED', 'FAILED')) NOT NULL`
   - `created_at (TIMESTAMP) DEFAULT CURRENT_TIMESTAMP`
   - `updated_at (TIMESTAMP) DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP`

2. **User Table**:
   - `user_id (UUID) PRIMARY KEY`
   - `name (VARCHAR)`
   - `email (VARCHAR)`
   - `created_at (TIMESTAMP) DEFAULT CURRENT_TIMESTAMP`

3. **Fraud Detection Logs Table**:
   - `log_id (UUID) PRIMARY KEY`
   - `transaction_id (UUID) NOT NULL`
   - `reason (TEXT)`
   - `created_at (TIMESTAMP) DEFAULT CURRENT_TIMESTAMP`

### Access Patterns:
- Frequent reads from the Transactions table for user history.
- Writes to the Transactions table on payment processing.

### Architectural Decisions:
1. **Consistency vs. Availability**: Opting for eventual consistency allows for higher availability, especially during peak loads, while ensuring that all transactions are eventually captured.
2. **Synchronous vs. Asynchronous Processing**: Using Kafka for asynchronous processing improves throughput and decouples services, though it introduces complexity in handling failures.
3. **Caching Strategy**: Implementing Redis for caching reduces database load for frequent reads, but adds complexity in invalidating cache entries.
4. **Database Choice**: Choosing PostgreSQL ensures strong consistency for transactions but may limit horizontal scaling compared to NoSQL options. The use of read replicas mitigates some of these challenges.
5. **Rate Limiting**: Implementing rate limiting at the API Gateway level helps protect backend services but may create user frustration during high traffic.

Build a high-throughput Payment Pipeline

Adobe

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements:

Capacity Estimation

Back-of-Envelope Calculations:

Submit Your Answer

Adobe Product Manager Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search