System requirements

Functional:

• Allow customers to place orders.
• Notifies business owners or order change on the customer side (e.g. cancellation)
• Allow business to update the orders (e.g. shipping, delay, etc.), which notifies customers.
• Allow both customer and business to search orders and view order details.
• Out of scope
  • Payment
  • Allow customers to leave comments
  • Support customer and business exchange messages.

Non-Functional:

• System needs to highly available (99.99%)
• Strong consistency regarding order placement and status updates.
• Low latency regarding updates and customer/business queries.
• Security.
• Scale to 1M businesses and 100M customers, and daily order of 10M.
• Fault tolerant.

Capacity estimation

Storage

• Assume order content and all updates are 10KB each. Daily order storage = 10M * 10KB = 100GB. In one year: 100GB * 365 = 36TB. In 5 years it is ~200TB.

Network

• Assume each order gets 1 update per day, and 1 query from customers and businesses per day.
• Assume average order fullfill time is 7 days.
• Read/write qps = 10M * 7 / 86400 = 800 qps
• The number of notifications per second is similarly 800/sec

API design

// Customer order management

POST /orders -> Order

GET /orders/

GET /orders&q={filters}&pageSize={}&pageToken={}

// Business order management

GET /orders/

GET /orders&q={filters}&pageSize{}&pageToken={}

PATCH /orders/

Database design

Customer table

• id (PK)
• name
• address
• ...

Business table

• id (PK)
• name
• ...

Order table

• id (PK)
• customerId (FK -> Customer)
• itemId (FK -> Item)
• businessId (FK -> Business)
• quantity
• status // created | paid | shipped | delivered | canceled
• shippingAddress
• shippingCost
• createTime
• updateTime
• ...

Item table

• id (PK)
• businessId (FK -> Business)
• name
• price
• ...

OrderUpdate table

• orderId (FK -> Order)
• timestamp
• event // shippingLabelCreated | shipped | inTransit | ...
• description

High-level design

ClientCustomer and ClientBusiness

• Different client website and mobile applications for customer and business, respectively

APIGateway

• Handles authentication, SSL termination, rating limiting, etc.

OrderService

• Handles search, CRUD of orders

UpdateService

• Handles order updates

NotificationQueue and NotificationService

• Handles outbound notifications to customers and businesses

OrderDB

• Stores order and update information

Request flows

Customer flows

Create

• Customer sends a request to order service via APIGateway to place an order
• We omit the payment and inventory update aspect
• OrderService inserts the order and the first status update into DB, marking the order created.
• DB enables CDC, which inserts the update into notification queue.
• NotificationServices picks up the message, and sends an notification (e.g. email, mobile notification) to the customer.
• It also notifies the business that the order is created (e.g. email, webhook)

Search/view

• Customer sends a search request to order service. Order service constructs an ES query to get the matching orders.
• When customer drill into the details, OrderService retrieves the information from OrderDB, by joining the Order table with OrderStatus table

Business flows

• Business gets notified when orders are created
• It can search, list and view orders just like the customer flows
• When there is an update, it calls the UpdateService, which inserts an update into the OrderUpdate table, and sets the Order status accordingly
• Then the update is inserted into the UpdateQueue via CDC. The NotificationService handles the message and send updates to the customer.

Detailed component design

NotificationService

• This service can send to different notification targets, based on user preference. For example, email, or mobile push notifications.
• On the business side, it may have an automated order processing system, in which case we register a webhook for that business and sends the order created message.

NotificationQueue

• Anytime the order changes, or the order status changes, either the customer, or the business or both needs to be notified.
• Using CDC and inserting the updates to an update queue enables asynchronous processing of the updates, in a reliable fashion.

OrderService, UpdateService and DB

• Using an ACID database ensures strong consistency or the order system.
• When updating an order, we update the status of the order across 2 tables in an transaction. In order to ensure consistency when there are concurrent updates (assuming conflicts are rare), we can optimistically update the table by introducing a version number, and checking this version number before making the update). Otherwise we can lock the row in OrderTable when updating status.

Trade offs/Tech choices

ElasticSearch

• This is useful in supporting keyword matches for orders. Alternatively we can enable full text search on the database, which increases DB load. We should measure the load and justify the introduction of ES.

Separation of OrderService and UpdateService

• Since the two service have different access patterns, we choose to create separate services so each can be scaled differently.

Failure scenarios/bottlenecks

OrderService, UpdateService and NotificationService (w/o webhook) are stateless, and can be scaled horizontally.

With 200TB and 800 read and write qps separately, we can shard the database, based on order id. Say we create 4 shards of 50TB each.

We should also have a background process that moves older orders to cold storage so data doesn't accumulate indefinitely, if we don't want to add more data storage for DB.

The the update queue, we are looking at ~800 message/sec, with each message ~200B (or 160KB / sec). This is well within the limit of 1 kafka queue and there is no need to partition the queues.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?