Loading...
Peak concurrent users = 10M/sec
Peak concurrent order attempts = 200K+/sec
Peak reads will be around similar magnitude (Reads will be higher during the normal phase, but during the sale, everyone wants to place the order)
Total items = 50K
Orders = 50K * 500bytes/sale = 25MB/sale
Assuming 10 sales a day, data = 250MB/day
Year = 91GB (which is not a huge number)
Most of these order attempts will not reach our system. Once we are sold out, within an initial few seconds, the rest of the users will see a Sold Out page, which is highly cacheable in CDN and Redis. We can later add back products if some payments are not successful.
GET /api/v1/deals/active
Response:
{
items: {
{item_id, title, description, price, currency, rating}, {}, {}, ...
}
has_more: True
next_cursor: uuid
}
GET /api/v1/deals/{item_id}
Response:
{item_id, title, description, price, currency, rating}
POST /api/v1/cart/{cart_id}
Idempotency-Key: uuid
Authentication: Bearer
Content: "application/json"
Accept: "application/json"
{
item_id: uuid,
qty: int
}
Response:
{
success: True
cart_items: {{}, {}, ....}
}
POST /api/v1/orders/
Idempotency-Key: uuid
Authentication: Bearer
Content: "application/json"
Accept: "application/json"
{
user_id: uuid
item_id: uuid
qty: int,
}
Response (Successful order placement):
{
item_id: uuid,
order_id: uuid,
qty: int,
base_price: 200,
total_price: 200,
currency: INR,
status: "PENDING"
payment_url: string
}
Response (Sold Out):
{
item_id: uuid,
qty: int,
base_price: 200,
currency: INR,
status: "SOLD_OUT"
}
Status: 200 (Success), 429 (Too Many Request), 500 (Server Error)
POST /api/v1/orders/payments
Idempotency-Key: uuid (order_id is fine in this case, as that will be unique)
Authentication: Bearer
Content: "application/json"
Accept: "application/json"
{
user_id: uuid
item_id: uuid
total_price: 200,
currency: string
}
Response:
{
payment_id: uuid,
status: "SUCCESS"
}
GET /api/v1/orders/{order_id}
Response:
{
order_id: uuid,
item: item_id,
status: "Order Placed"
expected_eta: datetime,
shipment_checks: [],
....
}
Clients requests are routed through edge server to Load balancer to Gateway and Rate Limiter.
CDNs are useful for serving images related to products, serving the sold out response to the users, so that most of the requests do not even reach our servers
Read Service: Reads the product, order related data from Redis/Postgres and serve it to the end_user. Product details, order status for hot paths are cached in Redis. Postgres is the source of truth
Write Service: Checks the counter from Redis and decrement the available qty in one atomic process (Lua Script). If product available, we will add a row in Postgres and mark the item as hold with timer of curr_time + 10 mins.
Payment: 3rd party payment integration, we listen to the status using a webhook and update the order status based on the result.
Success -> Order placed -> Notification
Failed -> Release the Hold, update redis and postgres -> product available again.
Kafka: This is to separate out the asynchronous tasks from the primary path of Write service.
Redis Cluster: Holds product data, product available qty (acts as a primary source of qty for the high frequency order placement), if it goes down, the available qty can be re-calculated from Postgres.
Postgres: Users table, Cart Data, Order Data, Payment Data
Users Table:
(
user_id, name, address, city, age, gender, created_at, updated_at, profile_pic_url
)
Product Table (Mongo):
(
product_id, image_url, created_at, updated_at, variation: {{size, qty, price}, {}...}
)
Cart table:
(cart_id, user_id, created_at, updated_at)
Index at (cart_id, user_id), (user_id, cart_id)
Cart Items:
(cart_id, item_id, qty, created_at, updated_at)
Orders table:
(order_id, item_id, user_id, qty, price, payment_status, created_at, updated_at)
Partition by user_id if needed
Redis:
product_available:{product_id} -> int
product_details:{product_id} -> JSON
cart:{cart_id} -> JSON
Handling huge spikes: Since we have a predictable spike for the traffic (roughly T-5 mins to T+5 mins, where T is the sale start time), we can cache the product data, available qty to Redis with appropriate TTL beforehand, so that we do not call the Postgres/Mongo DB for info.
Secondly, we can increase our servers anticipating the traffic.
Handling the sale orders:
We use Redis to store the cart data of the user, so the data is available with us in single digit ms. We will also use Lua Script, to check the availability of the product and decrement if available.
On Success -> add entry to Postgres Order Table, move the product qty from available to held and wait for the payment loop to complete. A background worker will remove the hold from the order items, after 10 mins, if payment is not successful and item will be available again.
Of Out of Stock -> We will aggressively cache the data in Redis and CDN, so that the large chunk of traffic do not reach our servers and is handled through the edge servers. Redis TTL could be 2-3 mins, while CDN can cache the data for 60 secs reducing the load.
Once the order is processed, we will write to Outbox Table, from where the event is pushed to Kafka, and is used for notifications, monitoring logs etc.
Failure Scenarios: