System requirements

Functional Requirements

• High Priority
  • FR-H1: The system shall automatically determine a user's status based on the following rules:
    • Online: The user is connected and has interacted with the application (e.g., keyboard/mouse input, screen touch, app in foreground) within the last 5 minutes.
    • Idle: The user is connected but has not interacted with the application for more than 5 minutes.
    • Offline: The user has explicitly logged out, or the server detects the WebSocket connection has been terminated (e.g., due to network loss or browser closure).
  • FR-H2: A user shall see the real-time presence status of other users who are on their contact list or part of the same active session (e.g., chat room, collaborative document).
• Medium Priority
  • FR-M1: Users shall be able to manually override their status with a selection from a predefined list (e.g., 'Busy', 'Away'). A manual status persists until the user clears it or logs out, temporarily disabling automatic status updates.
  • FR-M2: A user shall be able to subscribe to another user's presence, receiving an in-app notification when that user's status changes to 'Online'.
  • FR-M3: A user's status can only transition from 'Offline' to 'Online' upon establishing a new connection. Manual status changes are only possible when the user's status is 'Online' or 'Idle'.

Non-Functional Requirements

• High Priority
  • NFR-H1 (Scalability): The system must be designed to support up to 100,000 concurrent users.
  • NFR-H2 (Latency): Status updates shall be propagated to all subscribed clients with a 99th percentile latency of ≤500 ms.
  • NFR-H3 (Availability): The presence service shall have a minimum availability of 99.9%.
  • NFR-H4 (Security): Access to all presence information and related APIs shall be restricted to authenticated and authorized users.
  • NFR-H5 (Security): All data in transit must be encrypted using TLS 1.2+.
  • NFR-H6 (Efficiency): The communication protocol must be lightweight to minimize network bandwidth and client-side resource consumption (CPU, battery). Server-side resource allocation per user must be optimized to meet scalability targets.

Capacity estimation

General figures

First year

• 10,000 concurrent users
• Each user views ~20 statuses
• ~1% of users change status per second → ~100 status updates/s
• Notifications/s ≈ 100 updates/s × 20 recipients = ~2,000

Summary

• Concurrent users: 10,000
• Status changes/s: ~100
• Notifications/s: ~2,000

Third year

General figures

• 100,000 concurrent users
• Each user views ~20 statuses
• ~1% of users change status per second → ~1,000 status updates/s
• Notifications/s ≈ 1,000 updates/s × 20 recipients = ~20,000

Summary

• Concurrent users: 100,000
• Status changes/s: ~1,000
• Notifications/s: ~20,000

Storage

The storage calculation is based on the complete data model, including core presence data and the stream-based session lifecycle mechanism. Sizing is presented per cluster to reflect the separation between the Database cluster and the Messaging cluster.

Storage Calculation (per user):

1. Core Presence Data:
   • Status Hash (status:{user_id}): ~74 Bytes
   • Subscribers Set (subscribers:{user_id}): ~768 Bytes
   • Subscription List Set (user_subscriptions:{user_id}): ~775 Bytes
   • Sub-total: ~1617 Bytes per user
2. Session Lifecycle Data (Indexes):
   • Deadlines ZSET (z:deadlines:{shard}): ~64 Bytes per user
   • Last Seen Hash (h:last_seen:{shard}): ~54 Bytes per user
   • Sub-total: ~118 Bytes per user

Total per user (excluding stream): 1617 + 118 = ~1735 Bytes

Stream Storage (pings:{shard}):

The pings stream is a rolling log. Its size depends on the retention window, not the user count directly.

• Pings per second (3rd year): 100,000 users / 60s ping interval = ~1,667 pings/sec.
• Estimated size per ping entry: ~100 Bytes.
• Memory for a 15-minute retention window: 1,667 pings/sec * 900s * 100 B/ping ≈ 150 MB. This will be managed by setting MAXLEN or MINID on the stream.

Total Estimated RAM (Third year, 100,000 users) — per cluster

• Database cluster (core presence + indexes): 100,000 users × 1,735 B/user ≈ 174 MB (excluding streams). Add operational overhead per your deployment standards.
• Messaging cluster (Streams + Pub/Sub): ~150 MB for 15‑minute pings retention, plus allowance for Pub/Sub buffers/channels and client connections. Plan ~300–400 MB with headroom.

Assumptions (capacity):

• Devices per user: baseline 1.0; for fan‑out estimates consider 1.3× to account for multi‑device usage.
• Average follow graph ≈ 20; real workloads can be skewed (hot subjects). Apply coalescing and per‑channel rate limits to smooth spikes.

API design

Real-time Communication (WebSockets)

Envelope

• Fields: v (protocol version), event, payload, optional ts (server timestamp), optional id (message id), optional correlationId to pair responses to client commands.
• Versioning: increment v only on breaking changes; add new fields in a backward‑compatible manner.
• Request/response pairing: clients set an id on requests; the server echoes this value as correlationId on any direct response (e.g., error) or hydration message (e.g., presence:updated sent in response to subscriptions:set). Server‑initiated events omit correlationId.

Client-to-Server Messages

The client sends commands to the server.

presence:update

Updates the current user's presence status.

subscriptions:set

Replaces the entire list of users the client is subscribed to.

subscriptions:add

Adds one or more users to the subscription list.

subscriptions:remove

Removes one or more users from the subscription list.

session:ping

A heartbeat message sent periodically by the client to indicate it is still connected and active. This prevents the session from being marked as abandoned and cleaned up.

session:disconnect

A message sent by the client just before it intentionally disconnects (e.g., on logout or window close). This allows the server to perform an immediate and clean removal of the session without waiting for the grace period.

notifications:subscribe

Subscribes to a one-time notification for a specific event, like a user coming online. For scalability and performance, it will be implemented in the client application.

Client Behavior

• Heartbeats: clients send session:ping every 60s.
• Idle detection: the idle threshold is 5 minutes without user interaction.
• Reconnects: on disconnect, clients use exponential backoff with jitter (e.g., 0.5–30s) and cap attempts; honor gateway 429 with Retry-After.

Server-to-Client Messages

The server sends events to the client.

presence:updated

Notifies the client that one or more subscribed user's status has changed. Can also be sent in response to a subscriptions:set command to provide the initial state of the new subscription list.

notifications:received

Delivers a one-time notification that was previously subscribed to. For scalability and performance, it will be implemented in the client application.

error

Returns validation or processing errors for client commands.

Authorization & Subscription Rules

• Limits: enforce max subscriptions per user and cap changes per minute to prevent abuse.
• Errors & correlation: on validation/authorization failures the server returns an error event with correlationId set to the request id. On success, when a state change is relevant, the server emits presence:updated (e.g., after subscriptions:set) and includes the same correlationId.
• Rate limits: per-connection caps on message rate and payload size; the gateway enforces coarse limits, the microservice enforces fine-grained ones.

Database design

The data model is designed to be resilient and efficient. It separates the core presence data from the session lifecycle management mechanism, which is built on a reliable event stream pattern.

Core Presence Data

These structures hold the real-time state of user presence and subscriptions.

1. User Status (HASH): status:{user_id}
2. Subscriber List / "Reverse" Mapping (SET): subscribers:{user_id}
3. User's Subscription List / "Forward" Mapping (SET): user_subscriptions:{user_id}

Session Lifecycle & Cleanup Data (Stream-based)

This model ensures reliable detection and cleanup of inactive sessions.

1. Pings Stream (pings:{shard}): A durable stream where every client ping is recorded using XADD. This serves as the reliable input for session activity.
2. Deadlines (ZSET): z:deadlines:{shard}. A sorted set where the member is the user_id and the score is the Unix timestamp when the user should be considered stale if no new ping has arrived. This is updated by the consumer group.
3. Last Seen (HASH): h:last_seen:{shard}. Stores the timestamp of the last processed ping for a user. Used by the Scheduler to prevent race conditions.
4. Presence Events Stream (presence_events:{shard}): An output stream where the Scheduler publishes inactive events, allowing other systems to react.

Note: Sharding using a hash tag ({shard}) ensures related data for a user is co-located on the same cluster node, enabling efficient operations.

High-level design

This section includes a description of the components that implement the system for fulfilling the requirements.

It's complemented with the C4 Model system context and container diagrams.

A Kubernetes cluster is used to run most of the components.

Edge layer

Includes components that handle system incoming requests.

Load balancer

A load balancer ensures load distribution and supports high availability sending requests to active nodes.

A TCP load balancer (L4) is used because no complex routing logic is needed. Once the WebSocket connection has been established, it is maintained as a persistent connection. If the connection fails, due to a communication problem or one of the component fails (API Gateway pod, online presence microservice), a new connection will be established by the client application.

API Gateway

An API Gateway is used to:

• Rate limit requests
• Route new sessions to the presence microservice
• Centralized logging
• Check session authentication
• Message max length validation avoiding big problematic fraudulent requests

APISIX is the selected one because it's low latency, high throughput, high availability, stateless and extensible architecture. It is an Apache open source top project with good community and support. It will be implemented as a Kubernetes Ingress Controller.

Other good alternatives like Kong and NGINX were considered and APISIX selected because its features:

• Better performance and lower latency
• High availability and scalability (stateless data plane) by design
• Lower TCO
• Cloud-native solution
• Full dynamic configuration

Kong has a stronger plugin ecosystem. APISIX, as a top‑level Apache Software Foundation project, is growing rapidly.

Application layer

Contains the core applications and background workers that provide the business logic. All backend components will be implemented with Node.js, chosen for its scalable, lightweight, and efficient performance in I/O-bound, real-time applications.

An alternative like SpringBoot was considered but discarded due to its higher resource footprint and slower scaling for this specific use case.

Presence management

Identity and access management

Provides authentication in a specific, specialized secure solution. Keycloak will be used to authenticate users and issue JWTs for establishing the WebSocket connection.

Keycloak is selected because it's an efficient, popular solution that helps to avoid lock-in. Proprietary features will be avoided, sticking to standards.

Other alternatives considered were Auth0/Okta and AWS Cognito. These SaaS solutions provide lower TCO but higher lock‑in and were discarded.

Online presence microservice

This service's primary role is to manage the persistent WebSocket connection with clients. Its responsibilities are:

• Handling subscription commands (subscriptions:*) to modify user data.
• Acting as a producer of activity events (pings) to the session management system.
• Publishing and subscribing to the Pub/Sub broker to fan-out real-time presence updates.

The service will be implemented with Node.js because it provides a very scalable, lightweight solution ideal for I/O-bound real-time applications.

Spring Boot has been discarded because it is heavier for a real‑time application and slower to scale when needed.

Session lifecycle

Ping Consumers (Consumer Group)

A set of background workers that reliably process the activity event stream. They are responsible for updating the last seen status for each user and rescheduling their inactivity deadline. These will be implemented as a standard Kubernetes Deployment.

Scheduler

A background worker that periodically checks for users whose inactivity deadlines have passed. After verification, it publishes an "inactive" event to trigger the final cleanup. This will be implemented as a Kubernetes CronJob to ensure periodic execution.

Cleanup Worker (Event-Driven)

A background worker that listens for "inactive" events. Upon receiving an event, it performs the full, permanent cleanup of all data associated with the stale user. This will be implemented as a standard Kubernetes Deployment.

Database

A dedicated Dragonfly cluster that stores all core presence and session lifecycle data. Its in-memory, multi-threaded architecture was chosen for its low latency and high throughput, which are critical for this real-time application.

Pub/Sub Broker

A separate, dedicated Dragonfly cluster used for the real-time, low-latency fan-out of presence updates. This isolates the critical messaging fabric from the database, ensuring high availability.

Using Dragonfly for both roles simplifies the technology stack and reduces operational overhead, which was preferred over introducing a different system like RabbitMQ or Kafka.

Event Streaming

For reliable session management, we will use Dragonfly Streams. This feature provides a persistent, append-only log for processing activity events (pings) with at-least-once delivery guarantees.

The streams will be hosted on the same Dragonfly cluster as the Pub/Sub broker, creating a unified "Messaging Cluster" that is isolated from the primary database.

Analytics, Monitoring & Alerting

This layer collects and provides useful information for business analysts and SREs to monitor system health and usage.

Observability Stack: The Grafana LGTM Stack

For our observability needs, we have selected the Grafana "LGTM" (Loki, Grafana, Tempo, Mimir/Prometheus) stack. As the company has a mature platform engineering team, this stack provides a cloud-native, highly available, and scalable solution that avoids vendor lock-in. It offers an efficient TCO at scale and provides a single pane of glass for metrics, logs, and traces, which is crucial for rapid troubleshooting.

OpenSearch stack has also been considered

A key metric to monitor will be the consumer lag for the pings stream (XPENDING).

Components

• Visualization (Grafana): The central dashboarding platform for both business and technical metrics.
• Metrics (Prometheus): Scrapes and stores time-series data for monitoring and analysis.
• Alerting (Alertmanager): Manages alert deduplication, grouping, and routing based on Prometheus data.
• Telemetry Collector (OpenTelemetry Collector): Collects application metrics for ephemeral application components (Scheduler).

Logging

This layer is responsible for the collection, storage, and querying of application logs and traces to support troubleshooting and debugging by SRE and DevOps teams.

Components

We will leverage components from our chosen Shared Observability Stack:

• Visualization (Grafana): Used to search, query, and correlate logs and traces.
• Log Storage (Loki): Stores and indexes application logs.
• Trace Storage (Tempo): Stores and retrieves distributed traces.
• Log Collector (Fluent Bit): A lightweight agent to collect logs and forward them to Loki.
• Telemetry Collector (OpenTelemetry Collector): Collects application traces and forwards them to Tempo.

Observability & SLOs

• SLOs: P99 publish→client push latency ≤ 500 ms; availability ≥ 99.9%; missed-update rate ≤ 0.1%/day.
• Metrics: WebSocket connection count, send-queue depth, broker publish rate, pub/sub end-to-end latency, stream XPENDING lag, consumer throughput, error rates.
• Alerts: on sustained P99 latency breaches, rising XPENDING, consumer restart storms, and publish failures.

High Availability

This section summarizes how HA is achieved for each component. The baseline target is ≥99.9% service availability.

Kubernetes cluster

• Two availability zones (AZ-A, AZ-B)
• Managed HA control plane (or 3 dedicated control plane nodes)
• Worker nodes distributed across AZs
• Reliability measures:
  • PodDisruptionBudgets (PDBs)
  • Topology spread constraints
  • Anti-affinity to avoid colocation
• Updates and scaling:
  • Rolling updates with surge
  • Horizontal Pod Autoscaler (HPA)
  • Cluster Autoscaler enabled

L4 load balancer

• Multi-AZ nodes
• TCP health checks to API Gateway
• Connection draining during rollouts
• Routes:
  • WebSocket (WSS) traffic
  • IAM (Keycloak) traffic

API Gateway (APISIX)

• 3+ replicas, stateless data plane
• APISIX Ingress Controller: 2 replicas
• Config store (etcd):
  • HA quorum of 3 nodes across AZs
• Resilience:
  • Readiness/liveness probes
  • PodDisruptionBudgets (PDBs)
  • PreStop hooks for WebSocket connection draining

Presence microservice

• 3+ replicas, stateless
• Autoscaling:
  • HPA on CPU usage
  • HPA on custom metrics (active sessions, send-queue depth)
• Reliability measures:
  • Readiness/liveness probes
  • PDBs
  • Topology spread constraints
• Graceful termination:
  • PreStop hook
  • terminationGracePeriod for WebSocket drain

Dragonfly database (core presence)

• 1 primary + 2 replicas across AZs
• AOF enabled
• Automated failover via operator/orchestrator
• Client connections:
  • Service/virtual IP for reconnection
• SLA characteristics:
  • RPO bounded by async replication
  • RTO at seconds-level

Dragonfly messaging (Pub/Sub + Streams)

• Separate cluster, same topology (1 primary + 2 replicas, AOF)
• Pub/Sub:
  • Consumers auto-reconnect and resubscribe on fail-over
• Streams:
  • Consumers use consumer groups to resume from Pending Entry List

IAM (Keycloak)

• 2+ replicas behind the same L4 load balancer
• OIDC endpoints highly available
• Database:
  • External HA (e.g., PostgreSQL with 1 primary + replicas, or managed)
• Key operations:
  • JWKS and token introspection served via LB
  • API Gateway validates through LB to avoid single-node dependency

Observability

• Prometheus pair scraping in parallel
• Grafana:
  • 2 replicas
  • Persistent storage
• Loki/Tempo:
  • 2+ replicas for ingesters and queriers (or managed)
• Alertmanager:
  • HA mesh setup
• Dashboards & alerts:
  • Cover replica health, replication lag, failover KPIs

Operational HA practices:

• Health Checks & Timeouts: strict readiness/liveness; conservative idle timeouts on LB to sustain long-lived WSS.
• Rolling/Canary: phased rollouts for gateway and presence pods; error budgets gating production.
• Backups & DR: periodic AOF snapshots exported to object storage; documented restore runbooks and drills.
• Configuration: infra as code for quick re-provisioning; secrets stored in KMS/secret manager with replication.

Security

Communication with the client application is encrypted using WSS protocol and HTTPS (authentication flow).

In order to open a WebSocket connection and be able to access the system functionality, authentication is required with Keycloak. For increased security JWT is validated against the Keycloak token introspection endpoint.

API Manager provides rate-limiting and restricts message max size.

To protect Dragonfly unauthorized access the following measures are defined:

• Authentication: the access is protected with authentication, with credentials configured for Online presence microservices to access Dragonfly.
• Network security, like firewall or other kind of network-controls, to restrict network access to Dragonfly.
• Data is encrypted in transit with TLS.

For preventing DDoS and other malicious attacks a Web Application Firewall can be used as first line of defense.

Request flows

User login and token validation

User authenticates with IAM, client obtains JWT, presents it in WSS handshake; API Gateway validates (introspection, fallback: local signature).

See diagrams/sd-user-login-and-token-validation.puml.

Connection and initial hydration

Establish WSS, validate JWT, load subscriptions and initial statuses, send presence:updated.

See diagrams/sd-user-connect-and-subscribe.puml.

Presence change fan-out

Write status, publish on presence_updates:{user}, push presence:updated to subscribers.

See diagrams/sd-presence-update-fanout.puml.

Reliable liveness and cleanup

Produce pings, process deadlines, emit inactive, delete state.

See diagrams/sd-session-cleanup.puml.

Detailed component design

Kubernetes cluster

In order to provide high availability, the Kubernetes cluster will have nodes in 2 availability zones, and components distributed to have pods in both nodes.

Online presence microservice

The microservice is the primary interface for the client, handling the WebSocket connection and translating client commands into database operations and events.

Connection Establishment

1. Upon a new, authenticated WebSocket connection, the service retrieves the user's full subscription list from the central database: SMEMBERS user_subscriptions:{user_id}.
2. It then fetches the current status for all users in that list (e.g., via a batched HGETALL on the status:* hashes).
3. Finally, it sends a single presence:updated message to the client to fill their initial view of their contacts' statuses.

session:ping Handling

• Action: Upon receiving a ping, the service's only responsibility is to produce a message to the activity stream.
• Command: XADD pings:{shard} MAXLEN ~ <per-shard-max> * user_id {user_id} ts {epoch_ms}
• Per‑shard MAXLEN sizing: derive from target retention and shard count, e.g., total_entries ≈ pings_per_sec × retention_sec; per_shard_max ≈ ceil(total_entries × headroom / shards). Example: 1,667 pings/s × 900s ≈ 1.5M total; with 16 shards and 20% headroom → per_shard_max ≈ 112,523.
• Alternatively, use time‑based trimming with MINID to enforce a strict time window (e.g., now−15m).
• This is a lightweight, fire‑and‑forget operation that delegates all further session management logic to the background workers.

subscriptions:set Handling

This is the most complex client-facing operation, requiring updates to both the forward and reverse subscription mappings.

1. Fetch Old List: SMEMBERS user_subscriptions:{user_id}.
2. Calculate Diff: The service computes the added_users and removed_users by comparing the old list with the new one from the event payload.
3. Update Mappings (Transaction): The service executes the following in a single transaction (MULTI/EXEC):

• Overwrite the forward mapping: DEL user_subscriptions:{user_id} followed by SADD user_subscriptions:{user_id} [new_list].
• Update reverse mappings for added users: For each added_user_id, SADD subscribers:{added_user_id} {current_user_id}.
• Update reverse mappings for removed users: For each removed_user_id, SREM subscribers:{removed_user_id} {current_user_id}.

1. Respond: The service fetches the current status for all users in the new list and sends a presence:updated message back to the client.

session:disconnect Handling (Graceful Logout)

• Action: When a client sends a disconnect message, the server can bypass the normal inactivity window and trigger an immediate cleanup.
• Command: It publishes an inactive event directly to the presence_events:{shard} stream, which will be picked up by the Cleanup Worker. This is much faster than waiting for the Scheduler to detect the inactivity.

Scalability, Reliability & Backpressure

• Connections per pod: target 5k–10k concurrent WSS sessions per Node.js pod (tune based on CPU, memory, and event loop latency). HPA scales on CPU plus custom metrics (active sessions, send-queue depth).
• Coalescing: aggregate multiple updates into a single presence:updated per tick (e.g., every 50–100 ms) to reduce fan-out chatter.
• Dedupe: cache last sent status per user and suppress duplicates within a short window; include ts in broker payloads for freshness.
• Backpressure: bound per-connection send queue; drop oldest non-critical updates if the queue exceeds a threshold and send a resync snapshot when recovered.
• Resyncs: on reconnect or missed updates, send a compact snapshot of current statuses for subscribed users.

Session Lifecycle Components

The session lifecycle is managed by a set of decoupled components that implement a reliable, stream-based architecture.

Ping Consumers (Consumer Group)

These are a group of workers that process pings from the pings:{shard} stream with at-least-once delivery guarantees.

• Trigger: Reads a batch of messages using XREADGROUP.
• Logic: For each ping, it updates the h:last_seen:{shard} hash and reschedules the user's inactivity deadline in the z:deadlines:{shard} sorted set.
• Outcome: After processing a batch, it confirms with XACK. If a consumer fails, another can claim the pending messages, ensuring no pings are lost.

Scheduler

This is a background worker that periodically checks for users whose deadlines have passed.

• Trigger: Runs every few minutes.
• Logic:
  1. It queries the z:deadlines:{shard} set for users whose deadline score is less than the current time (ZRANGEBYSCORE).
  2. For each potential candidate, it performs a final check against the h:last_seen:{shard} hash to prevent race conditions.
  3. If the user is confirmed to be inactive, it publishes an inactive event to the presence_events:{shard} stream and removes the user from the deadlines set.

Cleanup Worker (Event-Driven)

This worker listens to the presence_events:{shard} stream.

• Trigger: Receives an inactive event.
• Logic: Performs the full data cleanup for the specified user (removes them from all subscribers sets, deletes their status hash and user_subscriptions set).

Database

The Dragonfly topology, for high availability, is 1 node (able to manage up to 1 TB of RAM) with 2 replicas, one in each availability zone.

If higher availability is needed in the future, a third replica could be deployed in a different region.

This configuration can scale much more than current needs.

Append Only File (AOF) persistence will be enabled for durability and being able to recover data in case of a full failure, and for faster fail-over recovery. It has small performance and throughput impact on Dragonfly thanks to its multithreaded architecture.

Pub/Sub broker (Dragonfly Pub/Sub)

The same Dragonfly topology of the database is used for the same reasons.

Channels (Topics)

• presence_updates:{user_id}
  • Purpose: low-latency fan-out of a subject user's presence change to interested viewers.
  • Producer: Presence Microservice (after updating status:{user_id}).
  • Consumers: Presence Microservice instances that currently host WebSocket sessions for viewers subscribed to {user_id}.
  • Payload (JSON): { "status": "online" | "idle" | "offline" | "Busy" , "ts": 1712345678901 }
    • The subject user is implied by the channel; include ts for client-side freshness checks and dedupe.
  • Semantics: best-effort/ephemeral pub/sub; duplicates are possible; consumers should coalesce and drop unchanged states.

Subscription Strategy

• Each service instance SUBSCRIBEs to one channel per followed subject across its connected clients.
• Subscriptions are updated when clients send subscriptions:set|add|remove and on connect/disconnect.
• Optional: if channel cardinality becomes too high, shard while keeping subject targeting, e.g., presence_updates:{shard}:{user_id}.

Operational Notes

• Avoid broadcasting via wildcard topics to limit wasted work; keep messages narrowly scoped to subjects actually followed.
• Monitor: total published messages/sec, per-channel publish rate for hot users, and delivery latency (publish → WebSocket push).
• Include memory for Pub/Sub client output buffers and channel metadata on the Messaging Cluster; monitor subscriber counts and per‑channel rates.

Event Streaming (Dragonfly Streams)

The pings and presence_events streams are critical for reliable session management and must be durable. They will be hosted on the dedicated Messaging Cluster. To ensure data is not lost in case of a full cluster failure, Append Only File (AOF) persistence will be enabled for this cluster. This provides a strong guarantee of durability with minimal performance impact.

Streams

• pings:{shard}
  • Producer: Presence Microservice on every session:ping.
  • Consumer Group: cg:pings (Ping Consumers).
  • Entry fields: { user_id: string, ts?: number } (HLD uses user_id; ts is optional and helps with observability/dedupe).
  • Command: XADD pings:{shard} MAXLEN ~ 50000000 * user_id {user_id} (optionally also ts {epoch_ms}).
  • Purpose: reliable input for liveness tracking; drives z:deadlines:{shard} and h:last_seen:{shard} updates.
• presence_events:{shard}
  • Producers: Scheduler (timeout-based inactivity), Presence Microservice on session:disconnect (graceful).
  • Consumers: Cleanup Worker (event-driven deletion of all user presence/session data).
  • Consumer Group: cg:presence-events.
  • Entry fields: { event: "inactive", user_id: string, reason?: "timeout" | "disconnect" }.

Semantics & Operations

• Delivery: at-least-once (use XREADGROUP/XACK); consumers must be idempotent (e.g., re-check keys before delete).
• Sharding: choose {shard} via a stable hash of user_id to co-locate related keys with the same hash tag.
• Retention: control size with MAXLEN/MINID; keep a modest buffer (e.g., 10–15 minutes) sufficient for operational catch-up.
• Monitoring: XPENDING lag for cg:pings and cg:presence-events, XINFO STREAM, and consumer restart recovery time.

Trade offs/Tech choices

Fan-out notifications with Dragonfly Pub/Sub

For implementing fan-out notifications two options where considered:

• Dragonfly Pub/Sub: lightweight, faster, less reliable.
• Dragonfly Streams: more reliable, scalable.

Pub/Sub is selected following efficiency and performance requirements as it provides enough throughput for the expected load. For a more reliable and scalable solution, streams can be used.

Streaming technology for session lifecycle

Dragonfly Streams has been selected instead of Kafka/Kafka Streams or Flink for managing session expiration. This choice meets efficiency goals and reduces platform complexity/TCO by reusing the existing data tier while still providing low-latency, at‑least‑once processing via consumer groups and bounded retention.

JWT validation

JWT validation is performed when the WebSocket connection is established. The API Gateway calls the Keycloak token introspection endpoint to catch revoked/closed sessions without incurring per‑message overhead. If IAM is unavailable, the system falls back to local signature validation using cached JWKS with background refresh and a short grace window.

Observability metrics database

Mimir, a standard component of the Grafana LGTM stack, is not used because:

• Moderate scale: ~1.7k pings/s, 1k updates/s, 10–30 services, 15–30s scrape interval → well within a single HA Prometheus pair’s capacity.
• Lower ops/TCO: Fewer moving parts than a full Mimir cluster (distributors, ingesters, queriers, compactor, object storage).
• Latency + simplicity: Prometheus provides fast queries and 7–14 days of retention with materially lower operational overhead for this use case.

Operational Parameters (defaults)

• Ping interval: 60s; gateway/LB idle timeout ≥ 120s.
• Idle threshold: 5 minutes without user interaction.
• Cleanup deadline: 2–3× the ping interval after the last heartbeat.
• Circuit breaker: open on ≥5% errors over 30s or p95 > 500 ms; half‑open after 30s with limited probes.
• Retry policy: bounded retries (max 3–5), exponential backoff with jitter, no unbounded loops.

Failure scenarios/bottlenecks

Reconnect storms

Regional blips/deploys cause thousands of clients to reconnect simultaneously, spiking gateway, IAM, and Dragonfly.

Mitigations:

• Exponential backoff with jitter
• 429 + Retry-After at gateway
• Autoscale gateway/microservice on reconnect rate

Dragonfly outage/latency spikes

Primary fail-over or hot keys increase latency, timeouts cascade to WS handlers.

Mitigations:

• Replicas + automatic fail-over
• AOF enabled
• Client timeouts + bounded retries + circuit breaker with half-open probes
• Alerts on p95/p99 and replication lag

API Gateway (APISIX) misconfig/outage

Bad rollout or overload blocks WS upgrades and HTTP.

Mitigations:

• Versioned config with dry-run/canary
• Fast rollback
• Autoscale on p95 latency. Exporting APISIX latency histograms to Prometheus and then driving scaling via HPA v2 (External/Custom metrics using Prometheus Adapter)

LB keepalive/idle-timeout mismatch

L4 or gateway idle timeout lower than ping interval silently drops WS.

Mitigations:

• Set ping interval < idle timeout by ≥30%
• Monitor disconnect spikes by reason
• Document client reconnect policy

Celebrity fan-out hotspot

Very large follower sets overwhelm fan-out paths.

Mitigations:

• Limit maximum followers number. 5.000 is considered a safe limit.

Consumer group lag/PEL growth

Stalled consumers cause growing lag and stale presence.

Mitigations:

• Autoscale by lag/age-of-last
• Cap retries
• Alerts on lag and Pending Event List size

IAM (Keycloak) unavailability/JWKS failures

Token checks fail, blocking connects/API calls.

Mitigations:

• Local JWT signature validation with cached JWKS + background refresh
• Short token TTLs
• Define deny-by-default with small cached-positive window
• Alerts on introspection/JWKS errors and key age

Poison messages/data corruption

Repeated failures block partitions and inflate PEL.

Mitigations:

• Validate inputs strictly
• Schema/versioning
• DLQ with quarantine and replay tooling
• Cap retries with backoff
• Dashboards on DLQ rate

Future improvements

• Multi-region presence with regional brokers and eventual global status via replication.
• Privacy controls (fine-grained visibility and “invisible” mode) and audit events.
• Client UX: display “last active” with bounded accuracy; tune idle threshold per platform.
• Optional migration path to Kafka + Flink if throughput/retention or stream processing complexity grows.