Loading...
This section outlines the refined and prioritized system requirements after a critical analysis.
Summary
Estimated growth: 10x
Summary
This section provides a detailed storage estimation for the graph database.
id (16 B) + name (avg 50 B) + preferences_reply_notifications (1 B) = ~67 Bytesid (16 B) + title (avg 150 B) + creation_date (8 B) + author_id (16 B) + author_name (avg 50 B) + last_update_date (8 B) = ~248 Bytesid (16 B) + content (avg 300 B) + creation_date (8 B) + author_id (16 B) + author_name (avg 50 B) + last_update_date (8 B) + discussion_id (16 B) = ~414 Bytesid (16 B) + login_date (8 B) + type (avg 10 B) = ~34 Bytesdate (8 B) = ~48 Bytesid for all nodes.author_id (Discussions, Comments) and discussion_id (Comments).Total Estimated Storage (Year 1): ~4.0 GB
Total Estimated Storage (Year 3): ~40.0 GB
This section defines the RESTful API for the nested comments system.
The API will use URI path versioning (e.g., /api/v1). Breaking changes will require a new version (e.g., /api/v2). Non-breaking changes (like adding a new optional field to a response) can be introduced within the same version.
PUT, PATCH, and DELETE operations are designed to be idempotent. Retrying these requests with the same parameters will not result in a different state.Authorization header.POST /api/v1/discussions{ "title": "string", "content": "string" }GET /api/v1/discussionslimit, offset.GET /api/v1/discussions/{discussion_id}DELETE /api/v1/discussions/{discussion_id}POST /api/v1/discussions/{discussion_id}/comments{ "content": "string" }last_update_date of all parent comments.GET /api/v1/discussions/{discussion_id}/commentslimit, offset.POST /api/v1/comments/{comment_id}/replies{ "content": "string" }last_update_date of all parent comments.GET /api/v1/comments/{comment_id}/replieslimit, offset.PATCH /api/v1/comments/{comment_id}{ "content": "string" }DELETE /api/v1/comments/{comment_id}403 Forbidden if the target comment has one or more replies.POST /api/v1/discussions/{discussion_id}/followGET /api/v1/user/followed-discussionslimit, offset.PATCH /api/v1/user/settings{ "enable_reply_notifications": true }200 OK: Request succeeded.201 Created: Resource was successfully created.204 No Content: Request succeeded, but there is no content to return (e.g., for DELETE).400 Bad Request: The request was malformed (e.g., invalid JSON, missing required fields).401 Unauthorized: Authentication is required and has failed or has not yet been provided.403 Forbidden: The authenticated user does not have permission to perform the requested action.404 Not Found: The requested resource could not be found.429 Too Many Requests: The user has sent too many requests in a given amount of time (rate limiting).trace_id will be included in all logs.User
Discussion
Comment
Device
id property of all nodes to ensure fast lookups.User(id)Discussion(id)Comment(id)Device(id)Discussion(author_id): To efficiently find all discussions started by a specific user.Comment(author_id): To efficiently find all comments made by a user, primarily for handling user profile updates (e.g., name change).Comment(discussion_id): To quickly fetch all comments for a given discussion and to allow for fast, synchronous updates of the root discussion's last_update_date.HAS_COMMENT
FOLLOWS_DISCUSSION
USES_DEVICE
Load balancer A L7 load balancer will spread the load between instances of API Gateway. It will provide advanced health checks. It may be integrated as a Kubernetes Ingress controller.
Will be implemented in a Kubernetes cluster.
API Gateway
Provides the following capabilities:
After aplying the proper logic, it forwards requests to the appropriate microservice.
Implemented with Apache APISix.
APISix is a high throughput, efficient, highly available and high scalable component because of its stateless data plane architecture and etcd based configuration. It will be deployed in a higly available Kubernetes cluster.
Identity and Access Management
A specific component based on OAuth2 / OpenID Connect will be used for it for several reasons:
It will be implemented with Keycloak.
Keycloack provides the following features for scalability and availability:
API implementation micro services
Provide the API business logic. Use the database for storing and retrieving the information.
Implemented as SpringBoot microservices.
Event Bus
Provides async processing of events for some tasks:
Implemented with RabbitMQ.
Last update date synchronizer
Spring Boot micreoservice that retrieves events when a comment has been created and asynchronously updates its hierarchy last update date.
Notification service
Spring Boot micreoservice that retrieves events when a comment has been created and notifies users of new replies to their comments. It takes into account user notification preferences. It notifies to all logged devices:
To support horizontal scaling, this service uses a Redis Pub/Sub layer for routing internal messages. When an instance receives a notification task from the Event Bus, it publishes the message to a user-specific Redis channel. The instance holding the actual SSE connection for that user is subscribed to the channel and delivers the message.
Real-time Routing Layer (Redis Pub/Sub)
A Redis cluster used as a high-speed, low-latency messaging layer for internal routing of real-time notifications between instances of the Notification Service. This allows the service to be horizontally scaled while ensuring messages are delivered to the correct client connection.
Database
Neo4J graph database that stores the application information. Other alternatives have been considered:
Neo4J has been selected becaues it provides a very good read performance with good write performance. It has an excelent ecosystem. Initially it will be self-hosted. If scalability starts to be difficult, it will be migratod to a fully managed cloud version.
Telemetry Collection
Grafana stack is used to provide a comprehensive solution.
Grafana As visualization platform for metrics, logs, traces. It provides dashboards both for both IT users (SRE and DevOps) and Business Analysts.
Prometheus Screaps technical and business metrics and stores them in it's TSDB. Initially no TSDB external database is used according to usage estimates. It could be changed in the future if needed.
Loki Stores application logs.
Tempo Stores application traces.
A specific component (IAM) used for increased security.
Some operations are protected and need authentication.
Operations allowed only for the author.
Encryption in transit useing HTTPS / TLS. For data at rest encryption Full-Disk Encryption (FDE) is used.
See sequence diagrams.
This section provides a detailed design for the Neo4j graph database, which serves as the primary data store for the nested comments system.
1. Data Model Summary
The database schema is designed as a graph to naturally represent the hierarchical and connected nature of the data. The core entities are:
User, Discussion, Comment, Device.(Discussion)-[:HAS_COMMENT]->(Comment)(Comment)-[:HAS_COMMENT]->(Comment)(User)-[:FOLLOWS_DISCUSSION]->(Discussion)(User)-[:USES_DEVICE]->(Device)This model was chosen because it allows for highly efficient traversal of the comment hierarchy (fetching replies) without the need for expensive recursive queries or complex joins that would be required in a relational database.
2. Indexing Strategy
To ensure low-latency queries, the following indexes are critical and will be created:
id property of all nodes (User, Discussion, Comment, Device) to guarantee fast, direct lookups.Discussion(author_id): To efficiently find all discussions started by a specific user.Comment(author_id): To efficiently find all comments made by a user. This is crucial for features like handling user profile updates (e.g., a name change).Comment(discussion_id): To quickly fetch all comments belonging to a given discussion. This is essential for the initial page load and for efficiently updating the root discussion's last_update_date.3. High Availability (HA) and Clustering
To meet the 99.9% uptime non-functional requirement, the database will be deployed in a Neo4j Causal Cluster.
4. Scalability Strategy
The database is designed to handle the projected 10x growth.
5. Backup and Recovery Strategy
A robust backup and recovery plan is essential for data durability.
neo4j-admin backup tool, which performs online backups without requiring database downtime.The Notification Service is a horizontally-scaled component responsible for sending real-time notifications to users. A key challenge in its design is handling the delivery of Server-Side Events (SSE) in a distributed environment.
The SSE Fan-Out Problem
When a user connects via SSE, their connection is maintained by a single, specific instance of the Notification Service. However, the RabbitMQ event that triggers a notification might be consumed by a different instance—one that does not hold the user's connection. The challenge is to route the notification from the processing instance to the connection-holding instance.
Solution: Redis Pub/Sub for Real-time Routing
This problem is solved by using the Real-time Routing Layer (Redis Pub/Sub) as a high-speed, internal messaging bus for the Notification Service instances.
The detailed flow is as follows:
Instance A).Instance A stores the user's connection object in its local memory.Instance A also subscribes to a unique Redis Pub/Sub channel named after the user's ID (e.g., notifications:user-123).comment_created event is consumed from the RabbitMQ bus by any available instance (e.g., Instance B).Instance B processes the event and determines that a notification needs to be sent to user-123.Instance B does not try to find the user's connection. Instead, it publishes the notification payload to the user's dedicated channel in Redis (PUBLISH notifications:user-123 '{"message": "..."}').Instance A, being subscribed, receives the message. It looks up the connection for user-123 in its memory and writes the message to the open SSE connection.Instance A removes the connection object from memory and unsubscribes from the notifications:user-123 Redis channel.This decoupled architecture allows the Notification Service to scale horizontally while ensuring that real-time messages are reliably and instantly delivered to the correct user.
A Graph database is the appropriate one for representing comments relationships and providing the reach queriesthe application needs.
Several alternatives have been considered for the Graph database: Neo4j, Amazon Neptune, ArangoDB, Dgraph
The following attributes have been evaluated: read performance, write performance, scalability, availability, operational overhead, tco (total cost), adoption & ecosystem The main candidates considered are:
Neo4j is the selected option. It has balanced qualities, open source, the best adoption, and allows to start with self-hosted solution and further scale with clud hosted option when needed.
For providing asynchronous communication between components message bus for 2 requirements is needed:
Kafka and RabbitMQ are considered
RabbitMQ has good availability and enough throughput for a significant increase in the supported user. The estimated number of notifications x second is 70 very far from the 10.000 limit RabbitMQ can provide. RabbitMQ is fairly known and has lower TCO.
RabbitMQ is selected for the current scenario.
For real time notifications sharing between the notification service instances, a more dynamic distribution based on user id is needed. When a server handles a user session it registers it in the pub/sub system and receives notifications for that user.
Redis Pub/Sub fits well for thes requierements with excellent throughput and availability.
APISix can validate JWTs locally and with the Keycloacks introspection endpoint. The latter provides additional security checking the token realtime for revocation and session logout. A balanced stratety, for reducing Keycloack requests, avoiding botlenecks, and maintaining security is:
For Grafana / Prometheus / Loki / Tempo, even though they provide high availability configuration, it won't be used initially to take advantage of the budget for other more critical components.
Those components are not critical and some outage can be accepted.
This is the main source of bottlenecks. For managing them, a circuit breaker pattern will be implemented in the API microservice, the update synchronizer and the notification services. Priority will be given to the API implementation that provides the core functionalities.
A circuit breaker will also be applied to Keycloack introspection endpoint calls with fallback to local validation.
In case pending events start to send notifications start to increase, Keda is used dynamically increase the number of pods in order to process more events concurrently.
The push notifications service can take longer to process requests. As we use an async service processing events no other system will be impacted, it will take longer to process requests. In case some user information is invalid, to avoid poison pills, an error message will be loged and the notification skiped.
When a comment is shared on a high-traffic social media platform, it can receive a massive number of requests in a short period, potentially overwhelming the backend API service and database. This is known as the "celebrity problem" or a "hotspot".
An effective way to mitigate this is by using the API Gateway (Apache APISIX) to dynamically cache only these high-traffic comments. This is achieved by combining several plugins.
The Strategy: Combining Plugins for Dynamic Caching
limit-count Plugin (The Detector): This plugin is used to track the number of requests for each individual comment ID within a specific time window.serverless-pre-function Plugin (The Brains): This plugin runs custom Lua code before other plugins. It checks the counter from limit-count. If the count exceeds a defined threshold, the script adds a special header to the request (e.g., X-Enable-Cache: true).proxy-cache Plugin (The Cacher): This plugin is configured to be "off" by default but activates when it sees the X-Enable-Cache: true header. For all subsequent requests for the "celebrity" comment, it serves the response directly from its cache, protecting the backend.Example APISIX Route Configuration
In your APISIX config.yaml or via the Admin API
routes:
uri: "/api/v1/comments/*"
methods: ["GET"]
plugins:
Step 1: Count requests for each unique comment ID.
limit-count:
count: 1000
time_window: 60
key_type: "var"
key: "uri_capture[0]"
policy: "redis" # Use Redis for shared state in a cluster
Step 2: The custom logic to decide whether to cache.
serverless-pre-function:
phase: "rewrite"
functions:
- |
local core = require("apisix.core")
local limit_count = require("apisix.plugins.limit-count")
local key = core.request.get_uri_capture()[1]
if not key then return end
local limit_count_key = "limit-count:" .. key
local count, err = limit_count.get_req_count(limit_count_key)
if err then
core.log.error("Failed to get limit-count: ", err)
return
end
local celebrity_threshold = 100
if count >= celebrity_threshold then
core.request.set_header(core.ctx.var, "X-Enable-Cache", "true")
end
Step 3: The cache plugin, which is bypassed by default.
proxy-cache:
cache_ttl: 30s
cache_key: ["uri_capture[0]"]
cache_bypass:
- expr: [['header_x_enable_cache', '!=', 'true']]
upstream:
type: roundrobin
nodes:
"your-backend-service:8080": 1
For additional security, a Web Apliction Firewall could be introduced.
Redis can also be used to share cache information between several APISix nodes.
More advanced caching strategies can be used for improving system performance in case of repetitive queries.