Requirements

Functional Requirements:

List the key functional requirements for the system (Ask the AI for hints if stuck)...

1. inventory tracking
2. stock replenishment(add)
3. order fulfillment
4. real-time inventory updates
5. automated stock alerts

Non-Functional Requirements:

List the key non-functional requirements (performance, scalability, reliability, etc.)...

1. Scalability: start from handling 100,000-level SKU
2. Reliability: the system will continue to work if any single service instance fails. No single-point failure. And any components can recover from failure gracefully.
3. Latency: The latency of inventory updates should be minimal.
4. Consistency: all inventory updates must strictly comply with the inventory restriction. Over-booking is now allowed.

Capability estimation

1. 100,000 SKUs
2. On average, 1,000 transactions for a SKU every day.
3. Then the average update frequency is 100k * 1k / 86400s, approximately 10k/s. And I assume that the peak load is 10 times the average.
4. Every transaction record is about 1k Bytes, so every day's storage usage is approximately 100k * 1k * 1k = 100GB. And I assume that the peak load is 10 times the average. The RDB can afford such a level of requirement, but it may be a bottleneck in the future.

API Design

Define the APIs expected from the system. This is your chance to analyze and define the read and write paths to develop the high-level design...

The APIs are intuitive for this system. They provide the basic functions for this system:

create_inventory(good_id, good_name, good_price)

get_inventory_amount(good_id)

stock_replenishment(good_id, amount, price, supplier_id) -> transaction_id

order_fulfillment(good_id, amount, price, sale_channel_id) -> transaction_id

transaction_stauts(transaction_id) -> state

Database design

supplier

• supplier_id
• supplier_name
• supplier_location
• supplier_properties

sale_channel

• sale_channel_id
• sale_channel_name
• sale_channel_location
• sale_channel_properties

inventory

• good_id
• amount
• good_properties

transaction

• transaction_id
• good_id
• amount
• price
• type
• supplier_id
• sale_channel_id
• state
• transaction_properties

The tables supplier, sale_channel is rarely updated compared to inventory and transactions. I can assume that there are 100,000 supplier and 100,000 sale_channel, and every record is only 10KB, then it's a good choice to store them in RDB.

The table inventory has 100,000 records, and every record size is about 10KB, then the total storage size is only 1GB. As mentioned earlier, the transactions may be too large for RDB. And the throughput of transaction writes may be very massive.

High-Level Design

Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.

API gateway

The API gateway works as the entry point of our system, and the rate limiter can be integrated with it to avoid too high load.

Supplier service

Register supplier for inventory.

Sale channel service

Register sale channels for inventory.

Transaction service

It's the core component of the system. The transaction is sent to it for processing. And the user can check transaction status. To support high throughput, I use an event-driven approach to process transactions. When a transaction is created, the transaction service generates a UUID and inserts the transaction into the Kafka stream with the good_id as the key.

And the transaction service will check cache for quick query, only when the record doesn't exist in the cache, it will be read from the DB.

Transaction worker

The transaction worker really applys the transaction:

1. For a replenishment transaction, it will always succeed.
2. For an order fulfillment transaction, it will succeed only when its requested amount is not greater than the amount of inventory remaining.

To support high throughput with strict consistency, the transaction worker is the most crucial part of the system. An event-driven pattern implements it.

1. The transaction events from the Kafka stream are the source of truth of the current state(inventory remaining amount).
2. It stores the current state in the remote cache for quick queries (what is the current inventory remaining amount for a good). The transaction state is also stored inside the cache for quick queries.
3. Finally, it will send the final state of the transaction event into another Kafka stream consumed by the Transaction finalizer.

Transaction finalizer

The transaction finalizer is responsible for transaction state update inside the persistent DB storage for future queries and analysis. It also call the notify service to send notification for users.

Monitor service

It aggregates metrics from the transaction finalizer to count the following metrics:

1. transaction throughput
2. transaction failure rate
3. transaction latency

Then the system adminer can maintain a dashboard to visualize these comprehensive metrics to know system's current operation status and confirm it's healthy.

Notify service

It is responsible for sending notification events to the transaction owner, so that the owner can know its final state as soon as possible.

Request flow

Let me demonstrate what happens when a stock replenishment or order fulfillment is sent.

1. The transaction service generates a UUID as transaction_id for this transaction and immediately sends it to the Kafka transaction topic. The good_id of this transaction serves as the message key.
2. One transaction worker consumes transactions from the Kafka stream's good_id-related partition.
3. The transaction worker applies the transaction and updates the inventory state if it's valid. Then it produces a transaction state event into the state topic, the transaction_id serves as the message key. The inventory state will be updated in the cache. The transaction finalizer consumes the transaction state event from the Kafka stream, it will insert the transaction record with its final state into the DB. And the transaction owner will be notified.
4. The user can query his transaction's state from the transaction service, and the transaction service will fetch the transaction status from the cache or DB.

The transaction service runs asynchronously, without blocking, to maximize throughput. And the transaction worker and transaction finalizer can scale easily because fo the event-driven architecture.

Detailed Component Design

Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.

Transaction worker

The transaction worker is the most crucial part for the high throughput of this system.

And I also need to ensure high availability for it. The transactions are partitioned by the task id, and a worker may consume a group of partitions. So at the same time, one good's transactions will be operated by only one worker in the system.

I can use consistent hashing for transactional goods distribution. If any worker dies or joins, only a small group of goods will be redistributed, thereby minimizing the impact of worker failure.

Another issue for the graceful recovery of the transaction worker is the cold starting of the event-driven system. Given the initial state of one worker, and all the transaction logs it consumes, it's obvious to construct the correct state until the failure time. But this is unacceptable because there may be too many histroy transactions.

To avoid such an issue, the transaction worker can store each good's latest inventory state in the cache and the DB for persistence. This is a dict from good_id to inventory state, and when it's stored, the corresponding partition id and offset are also bound. When the worker restarts, it will first get the partition_ids it is responsible for, then fetch the dumped state record of this partition id with the latest offset. Then it can consume from the next offset of the previous dumped latest offset, minimizing the influence of restarting.

Transaction finalizer

To ensure system high availability, the transaction may be retried, so the transaction finalizer should guarantee idempotency. I will use the DB as the source of truth to guarantee idempotency.

When a transaction finalization state is consumed, the transaction finalizer will first check whether there is already a record inside the DB. If the DB record already exists, this transaction will be ignored. Otherwise, the new transaction record will be inserted, and the notification service or monitor service will be invoked.

Database and cache

To ensure high throughput, the database and cache choice is also important.

Here, I will use Non-SQL DB, for example, DynamoDB.

For the queries from the transaction service and the transaction finalizer, the transaction_id is the key. So I can use the transaction_id as the DynamoDB's primary key, to avoid the hot-spot issue of the storage, but also support quick query.

The transaction worker also needs to store its dumped state in persistent storage. I can use the partition_id of the Kafka transaction topic as the primary key, and use the offset as the sort key. And I can set an expiration policy for the worker state records because only the recent states are useful.

The cache acts for quick query of good's inventory amount, and the transaction's state. I can use Redis here for this simple key-value pattern query.

1. good_id -> inventory_amount
2. transaction_id -> transaction_state

If the records expired, the transaction worker or transaction service will re-activate it again.

Summary

As mentioned earlier, the transaction service is stateless and can be scaled independently. Using an event-driven architecture, the overall latency of transaction handling is minimized. By using consistent hashing for load distribution and periodic state dumps for the stateful transaction worker, it can recover from failures quickly and gracefully with minimal impact. Because only one worker handles the transactions of one good at the same time, there is no consistent write for one good, so the correctness and consistency are ensured.