Loading...
DAU = 10M
Peak concurrent connections = 2M (users) * 3 (devices) = 6M connections
Status change per sec = 10M * 20 / day = 2000 / sec
Peak status change = 6000/sec [Assuming 3x traffic]
Fanout delivery estimate = 2K/sec * 20 (inbox contacts) = 40K deliveries per sec
Peak Fanout delivery = 120K deliveries/sec
Throughput: Heartbeats per sec = 6M /30 = 200K heartbeats / sec [Assumed one heartbeat per 30 sec]
Network Bandwidth = 200K * 50bytes = 10MB/sec = 80Mbps
Storage = 1 Row per user and we will update the status of the user in the same row. It scales with users.
At 10M DAU, and assuming the record to be of 200bytes (including metadata), storage will 10M * 200 bytes = 2GB [which is trivial and will not be the bottleneck]
Historical storage = 10M * 20 * 200bytes/day = 40GB/day = 15TB/year
/GET v1/online-presence/{user_id} # Search for user's status
Header:
Authentication: Bearer <Token>
Content: "application/json"
Accept: "application/json"
Response:
Headers:
Content: "application/json"
RateLimit-Limit: 1000
RateLimit-Reset: 60
Retry After: 60 (Only if Ratelimit is hit)
Body:
{
user_id: uuid ##The user id searched for
status: ENUM,
last_seen: timestamp ## If the user is offline
}
Status Code: 200 (all ok), 403 (Unauthorized, for non-friends), 429 (Too many request), 500 (Server Errors)
/GET v1/online-presence/friends
Header:
Authentication: Bearer <Token>
Content: "application/json"
Accept: "application/json"
Response:
Headers:
Content: "application/json"
RateLimit-Limit: 1000
RateLimit-Reset: 60
Retry After: 60 (Only if Ratelimit is hit)
Body:
{
status: [
{
user_id: uuid ##The user id searched for
status: ENUM,
last_seen: timestamp ## If the user is offline
}
],
success: True
}
Status Code: 200 (all ok), 403 (Unauthorized, for non-friends), 429 (Too many request), 500 (Server Errors)
Web Socket /v1/presence
{type: "ping",
active: true/false,
device_id: uuid,
ts: timestamp}
Clients Request is routed through Load Balancer and Gateway (also handles Rate Limiting).
Clients sends 2 types of requests: WebSocket Pings and API calls.
Web Socket Pings carry info like - {status: active/idle, device_id, client_timestamp)
WebSocket updates the info in Redis and sets the TTL (server time + 90sec).
In case of a status change (active <-> idle), it also publishes as event to kafka.
If the key expires -> Kafka listens to the event and thus we know about the client going offline.
Aggregator Service subscribes to Kafka events, batch updates the status in Postgres for persistence.
WebSocket also subscribes to the Kafka events and checks with the Bus service to know the chat servers UserA's friends are connected to and send updates to them (fan out).
Client can also ask for the status of a specific user.
Auth Service checks if the user is friend of the client and then forwards it to Status Service, which reads the status from Redis and fallbacks to Postgres (Read aside cache).
A user can have multiple devices sending heartbeats. All will update the same user key in Redis. Each heartbeat will set the TTL to +90s. So, a key will only expire when none of the devices sends a hearbeat within TTL.
Postgres
User Table:
(
User ID: UUID [PK]
Other relevant info about the user
)
Status Table:
(
id: uuid [PK]
user_id: [uuid] [FK] [index]
status: ENUM,
last_active_time: timestamp
)
Redis:
heartbeat:{user_id}:{device_id} -> {state: active/idle, last_ping_received: timestamp, device_id: uuid} [every update sets TTL to current_time + 90s]
status:{user_id} -> {state: active/idle/offline, last_ping_received: timestamp, device_id: uuid} [This is used to return the status of the user, without looking into DB for each call]
The Redis cluster will be sharded by user_id.
Status Update Logic: Each user's device sends a ping and we continuously update it.
We have a separate status:user, which is updated based on aggregate + some business logic (e.g. any ping which says user's status is idle will only update the aggregated result, if the last active_ping was atleast 60 seconds ago, but any active ping update will overwrite immediately).
The expiry of a user's hearbeat pings from all devices mean the user is finally offline, which is handled by Kafka.
Kafka is partitioned by user id. Any backpressure can be handled by adding more consumer workers.
Handling hot keys:
Lets say when a celebrity comes online, we will not fan out the status to all her subcribers/followers.
We will fan out only when the no. of followers is low (<500). For a celebrity, the users will know their status, when they visit their profile (read from Redis). We can eager fetch their status and update the Redis, or coallesce their request and let one request fetch the info from DB, while the others still send the stale state (non critical and slight delay is acceptable).
Failure scenario: