System requirements

Functional:

List functional requirements for the system (Ask the chat bot for hints if stuck.)...

Users can choose the product and its number in the vending machine.

Users can pay for the product with a bank card, cash, Paypal, the cryptocurrency and get the choosen product.

If users choose a product number that isn't available, the vending machine shows an error.

If users choose the product and there is a blackout, the vending machine shows an error.

If users choose the product and there is an internet connection issue, the vending machine shows an error.

Non-Functional:

• Performance: The service should serve the users with minimal latency.
• Reliability: The service should be highly available and fault-tolerant.
• Scalability: Should scale dynamically based on user load.
• Security: Ensure secure processing of the user's payment transactions.
• Maintainability: Code and system architecture should be easy to maintain and extend.

Capacity estimation

Estimate the scale of the system you are going to design...

Assumptions

1. Total Number of the vending machines: Assume the system needs to support 10 million vending machines.
2. Product Selection Frequency: On average, an active user selects a product in the vending machine once every 5 minutes.

Calculations

• Write Operations: Each user selects and buys a product in a vending machine every 5 minutes. For 10 million vending machines, that's (10,000,000 users×1 update)/5 minutes=2,000,000 buys/minutes

Resource Estimation

• Data Storage: Each product might need about 100 bytes (product ID, status, timestamp, name, price, count) and each vending machine needs 100 bytes contains(ID, status, timestamp, location) ,one vending machine may sell 100 products and it requires 10 Kbytes,

10 million vending machines need about 100 GBytes.

• Network Bandwidth: With frequent buying products and high concurrency, substantial bandwidth will be needed, especially for service synchronization across different regions if the service is globally distributed.
• Memory and CPU: To ensure real-time performance, a significant portion of operations should be handled in memory.
• High CPU resources will be required to process the continuous flow of reads and writes.

Considerations

• Database Choices: Given the midle write and read requirements, and the keeping the strong consistency, it needs to use SQL database like PostgresSQL, which can handle high throughput and horizontal scaling.
• Caching: Implementing caching mechanisms to reduce database load, using solutions like Redis to cache frequent queries or active user statuses.
• Load Balancers: To manage traffic and ensure even distribution of requests across servers.

API design

Define what APIs are expected from the system...

API Endpoints

1. Pay for the products selected by the user

2. Get the product list from the vending machine

Admin API:

1. Add a new vending machine:

1. Get the list of the vending machines:

Database design

Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...

Below is a detailed schema using PostgresSQL.

Main Tables in PostgressSQL

VendingMachineProduct Table

• productId (Composite Partition Key): String that uniquely identifies a product.
• vendingMachineId (Composite Partition Key): String that uniquely identifies a vending machine
• status: String indicating the current status (e.g., "available", "none").
• title: String
• lastUpdated: Timestamp of the last product update.
• expDate: Timestamp expiration date
• price: Double price for one unit of a product
• amount: Int amount of the product
• This table serves as a list of the products loaded into the vending machines.

VendingMachine Table

• vendingMachineId (Partition Key): String
• name: String name of the vending machine.
• createdAt: Timestamp when it was created.
• longitude: String
• latitude: String

This table has information about the vending machines that are installed.

Order Table

• orderId (Partition Key): String
• vendingMachineId: String
• createdAt: Timestamp when it was created.
• productId: String
• totalPrice: Double
• status: String ("processed","error","pending")
• updatedAt: timestamp

This table has information about the vending machine orders.

OrderDetail Table

• orderId (Partition Key): String
• productId (Partition Key): String
• price: Double
• amount: Double

This table has information about the specific vending machine orders.

High-level design

Key Components

API Gateway

• Distributes incoming traffic and requests evenly across web servers.
• Enhances the system's reliability and availability.
• Secure the HTTP traffic by using HTTPS

Payment service

• Process the payments with a bank card or cash.
• Integrates with third-party payment services.
• Persists the payment tracking id that third-party services return after payment processing.

Order service

• Store the vending machine information about the products the users selected.
• Check if the product amount is available by calling the Product List service.
• Sends payment requests to the Payment service and manages the payment process.
• Perisists the product information that the users can select in a vending machine, before that, the Order service checks the product amount.

Product List service

• Supports the list of products that the users can select and pay.
• Controls the number of products and vending machines.
• Persists the information about products and the vending machines to Database (PostgresSQL)
• Caches the product information about the most frequently requested.

Request flows

We'll examine the scenario:

Selecting and paying for the product from the vending machine.

1. Selecting and paying for the product from the vending machine.

This flow describes what happens when a user selects the product.

Sequence of Events:

1. Client Request: A client application selects a number of products in the vending machine.
2. API Gateway: Distributes the request to one of the available Order service.
3. The Order service: Receives the request and starts the business transaction, and persists the information about it to the Database.
4. The client application can select the list of products.
5. When the client application finishes the business transaction, the Order service calls the Product List service to check product number availability. If all necessary products are presented, the Order service waits from the client application the payment request. Otherwise, the Order service sends the error message to the client application.
6. If the Payment service receives the payment request from Order service,it starts the payment process by contacting the third-party service like Stripe, before that, the Payment service persists the payment tracking Id, the current timestamp, and status.
7. The Order service waits for the Payment service sends back the payment status.If a payment is successful, the Order service commit the transaction and returns success to the client application, otherwise it sends the error message.
8. The product List service cache: caches the the most frequently requested products.

Detailed component design

1. The Order service

Purpose: handles all incoming API requests for selecting and paying for the products in the vending machines.

Key Responsibilities:

• Drives the business transactions: commits or rolls back the transaction depending on the product availability and the result of the payment.

Scalability:

• Load Balancing: Utilizes a load balancer to distribute incoming traffic evenly across multiple instances, ensuring no single point of overload.
• Statelessness: Maintains no local state, allowing any request to be served by any instance, facilitating easy scaling.

Technologies:

• Language/Framework: Could use Node.js for its non-blocking I/O model, well-suited for handling a large number of simultaneous connections.

2. The Payment service

Purpose: a facade for the third-party payment services.

Key Responsibilities:

• Integration with the third-party payment services: send the payment request and persist a payment tracking id and its status.

Scalability:

• Horizontal Scaling: Supports scaling out by adding more nodes.

Technologies:

• Java: Ideal for this role due to its many Java frameworks for integration.

3. The Product List service and its cache.

Purpose: Manages the product list and the vending machine list.

Key Responsibilities:

• Product control: Processes product amounts and prices for the list of vending machines.
• Caches the product information.

Scalability:

• Horizontal Scaling: Supports scaling out by adding more nodes

Technologies:

• Java: Ideal for this role due to Java allowing to building of performant applications.

Cache is used to offload the The Product List service.

Scalability:

Horizontal Scaling: cache is scaled out by using productId as the sharding key.

Technologies:

Redis: Ideal for this role due to its performance and features like publish/subscribe, which are useful for invalidating cache entries when data changes.

Trade offs/Tech choices

1. NoSQL vs. SQL Databases

Trade-off:

• Scalability and Performance: Chose NoSQL AWS DynamoDB to store product information to leverage its fast writes and reads, which are essential for real-time applications.
• Complexity and Feature Richness: Sacrificed some of the advanced querying capabilities and transactional integrity offered by SQL databases.

Technology Choice: WS DynamoDB for real-time operations due to its superior performance in handling large volumes of data with high write and read throughput. PostgreSQL was chosen for detailed logs and analytics where complex queries are more common.

2. Stateless Order services, Payment service, and Product List service

Trade-off:

• Scalability: By making services stateless, they can easily handle increases in traffic by simply adding more instances.
• Resource Utilization: Stateless services might lead to more frequent cache misses and database hits, which could increase latency and load.

Technology Choice: Using stateless architecture facilitated by a load balancer (like NGINX or AWS ELB) to ensure requests can be served by any server, enhancing reliability and ease of scaling.

3. Cache Usage (Redis)

Trade-off:

• Performance vs. Cost: Implementing a caching layer improves read performance drastically but adds additional complexity and cost in terms of maintaining another layer in the infrastructure.
• Data Consistency: Caching can lead to situations where stale data is served unless proper invalidation strategies are employed.

Technology Choice: Redis, known for its quick data access speeds and robustness, fitting the requirement for a high-performance, in-memory data store that supports rapid read and write operations.

4. Asynchronous Processing with Message Queues

Trade-off:

• Complexity vs. Responsiveness: Introducing message queues (Kafka) increases system complexity but allows for decoupling components, leading to a more resilient and scalable architecture.
• Latency vs. Throughput: Asynchronous processing might introduce slight delays in processing but significantly increases throughput and system resilience.

Technology Choice: Kafka offers reliable messaging with strong delivery guarantees and is widely used in systems requiring high levels of decoupling and scalability.

Failure scenarios/bottlenecks

Failure Scenarios

Database Failure

• Scenario: A failure in the PostgreSQL cluster could prevent the vending machines work.
• Mitigation: Use PostgreSQL built-in replication features to replicate data across multiple nodes in different data centers. Employ a failover strategy that automatically switches to a healthy replica in case of a node failure.

Cache Failure

• Scenario: If Redis experiences an outage, the system would face high latency as all requests would need to go to the database.
• Mitigation: Implement redundant caching layers and use a master-slave Redis configuration. On cache failure, switch to a slave until the master is restored. Additionally, ensure that cache misses are gracefully handled by fetching data from the database.

Order service, Payment service and Product List service Downtime

• Scenario: Downtime or high latency in those services due to traffic spikes or DDoS attacks.
• Mitigation: Use auto-scaling groups for those services behind a load balancer to handle sudden increases in load. Implement rate limiting and DDoS protection strategies to mitigate abusive traffic patterns.

Network Issues

• Scenario: Network latency or partitioning can delay or prevent data synchronization across the distributed components.
• Mitigation: Design the network for redundancy, with multiple connectivity paths to prevent single points of failure. Use health checks and automatic rerouting to handle network partitioning.

Bottlenecks

Cache Hotspots

• Bottleneck: Over-reliance on certain cache nodes due to uneven data distribution can lead to hotspots.
• Mitigation: Use consistent hashing for cache distribution to evenly distribute load across cache nodes. Regularly analyze access patterns and adjust the caching strategy as needed.

Database Write Throughput

• Bottleneck: High write throughput demands on the database during peak times could lead to performance degradation.
• Mitigation: Use database sharding to distribute writes across multiple database instances. Consider write-behind caching to batch and asynchronously write updates to the database.

Future improvements

1. Adaptive Caching Strategies

• Improvement: Implement machine learning algorithms to predict user activity patterns and adjust the caching strategy dynamically. This can help optimize cache hit rates and reduce latency.
• Benefit: By adapting to user behavior, the system can ensure that resources are allocated more efficiently, improving overall performance.

2. Service Decomposition

• Improvement: Break down the service into more granular microservices. For example, separate services for handling user activity logging, and notifications.
• Benefit: This approach enhances scalability by allowing individual components to scale independently based on demand. It also improves maintainability and makes it easier to deploy updates without affecting the entire system.

3. Geographic Data Distribution

• Improvement: Implement geographic distribution of data and services to reduce latency for globally dispersed users. Use edge computing principles to bring data processing closer to the end users.
• Benefit: Reduces latency significantly and improves user experience, particularly in regions far from the central data center.

4. Enhanced Security Measures

• Improvement: Introduce more robust security protocols for data transmission and storage, including advanced encryption methods and continuous security audits.
• Benefit: Enhances the security of user data, reducing the risk of data breaches and building trust with the users.

5. Real-Time Analytics and Monitoring

• Improvement: Develop a real-time analytics engine that can process and visualize user status data as it happens, providing insights into user behavior and system performance.
• Benefit: Allows for immediate response to any critical incidents and better decision-making based on user activity trends.