POST status/{userId}/send -> send status update to subscriber
POST status/{userId}/ (payload is OnlineStatus) -> send a single status update (login, logout) of a user from a given device
High-Level Design
DNS: route the client request to the nearest geo location
Load balancer: distribute the traffic across different API gateway of the region
API gateway: dealing with AuthN, rate limite, route the request to the right service.
Web socket: manage the real time communication between backend and client. Manage connections and heartbeats from clients.
Presence service: Handle login/logout status update, and calculate the overall status of a given user from the user's devices. Since user status might be flaky due to network, this service will have debounce concept to buffer status update for 10 seconds before really updating the storage. The aggregate logic is OR of each connected device for a given user. If one of them is online, the user is online.
Tracing service: Subscribe to message queue on user login/out updates, then store login/out traces.
Message queue: Async process for 1) status update notification event and 2) trace event based on status update from Presence service.
Database: stores 1) user login status per device, 2) raw traces
Cache: store previous calculated user status for perf.
Analytics service: Aggregate raw traces into metrics and stores in Analytics store.
Detailed Component Design
Web socket server handles client registration/unregistration on a given user's status update. It will keep the connection alive with Auth, and broadcast a user's login status update to all subscribers. Each client will register with client id, so that multiple devices of a given subscriber will receive notification for the same person. Once client registered, the client should send heartbeat signals every 10ms to the server to indicate it's liveness. If a client missing three heartbeats, it's considered offline, and the connection will be closed.
Once a user is login to the system (Login/out is out of scope of this system), presence service will be called to put a status update event on the message queue. Then another handler in the presence service will listen to the event, and do a database update for that given login from a specific device id. Once the database update is successful, it will push an aggregate status update event back to the message queue. Then another handler will pick up this event, try to find pre-existing login status for the given user from cache, if no cache hit, will query data from DB and refill cache. Once the aggregate status is changed and updated in database, another event will be put on the message queue for sending notification to subscribers.
A user's presence is calculated from all of his/her devices with OR logic. As long as there is a device online, the user is online.
When there is concurrent status write, we will use the version of the object as a lock - when an update holds different version than the latest one, the update will fail. And the caller will retry with jitter. This will guarantee eventual consistency.
When presence service received login/logout status update request, it will put a trace event to the message queue. Then Tracing service subscribe to the event will store the data in database.
The analytics data is trigger as async operation. When a status update event is put onto the message queue, the trace service listen to the event and store the raw log into the system (LogTrace). Then we will have another analytics service to batch process the raw log periodically and put into Analytics service as a multi-dimensional database for easy slice and dice data. This is how we will create the data for session, usage, ... etc. They don't have to be real time computation.
The scalability of the system comes from microservice + message queue (async process) design.
On high fan-out situation, we already have a message queue to digest burst of traffic. Besides, we can send notification to online friends only.