Requirements

Functional Requirements:

We'll focus on the order placement + matching + streaming path:

• Allow traders to place buy and sell orders
• Stream live market data to traders.
• Match buy and sell orders based on price-time priority, where the best price gets priority, and among equal prices, earlier orders are matched first; confirm orders and provide trade execution reports.
• Send notifications for order status changes (filled, partially filled, cancelled), price alerts, and risk alerts, supporting real-time and periodic notifications.

9

Non-Functional Requirements:

• Buy&Sell events per day 10M to 100M (peak).
• Total Storage: 50M avg. 1Kb * 50_000_000 per day * 10 yrs = 500 TB
• Hot Storage: 50G per day
• RPS: 4_000_000 / 4000 = 1~5k RPS
• HA: 99.999% (stream, put order, check order, alert, push notification)
• Latency: Avg < 10 ms, P95 < 30 ms (put order, check order, alert, push)
• Latency: P95 < 100ms from in to out for stream
• Stream: only stream executed orders
• Support re-stream lost data by tick number range.
• Sync ops: put order, pre-order check
• Async ops: notify & alert, stream

API Design

• Register client for streaming (TCP). req: (client_ip, trader_id, market_id, etc.)
• Streaming (UDP) to registered clients. input: (client_ip, trader_id, market_id)

output: stream of [(tick_id, serial_no, asset_id, op_type, amt, market_id)]

• Put order: POST /order

Authentication: Bearer

body: market_id, asset_id (stock_id/etc.), op_type (buy/sell), amt

High-Level Design

• Order Ingest Service gets the real-time put/sell data and write to Kafka as the main event store, partition by asset_id, each market gets an exclusive topic.
• The Bid Service consumes Kafka event, we might need a large amount of Bid Service instances as this is where latency matters. For highly liquid assets, we need to use an exclusive node to handle. Bid service needs to have different cluster for different market, and in each market, each node consumes one or more assets, to ensure fairness, round robin each asset per bid. Bid Service also needs a large RAM, to store asset amt per user + balance per user to ensure the order is valid. Bid Service finalizes the bid result in async process to the Account Service.
• Account Service manages the users account balance, margin, etc.
• Post-Execute Service: does post execution check, send notifications for success/failed/timedout orders. Streams realtime data to traders. This service also has to have multi cluster setup, each cluster handles a market, each node in the cluster handles one or more assets

Detailed Component Design

• Trader registers the clients and maintain a heart beat to update the client ip
• Post-Execute Service streams the executed result (success) to traders' client
• Post-Execute Service sends the alerts/notifications via email/sms if a rule-based alert is triggered, or an order fails to execute.
• Post-Execute Service communicates to the Payment Gateway to finalize the transaction
• Bid Service maintains a Heap data structure for each asset and buy/sell op in memory, total of ~50GB for in-memory margin check fast bid:

pseudo-code:

if order.isBuy():

if order.amt >= margin[order.user_id]:

buyheaps[asset_id].push(order)

else:

return fail(order)

else:

if order.asset_amt >= assets[user_id][asset_id]:

sellheaps[asset_id].push(order)

else:

return fail(order)

execute(buyheaps[asset_id].pop(), sellheaps[asset_id].pop())