System requirements

Functional:

1. Handling Out-of-Order and Delayed Events:
   • The system must manage events that arrive out of order, such as an endEvent arriving before a startEvent.
   • It must also accommodate events that may arrive significantly delayed (e.g., hours or days after their actual occurrence).
2. Event Duration:
   • An event can last up to one hour.
3. APIs:
   • startEvent(event_id, timestamp): Record the start of an event.
   • endEvent(event_id, timestamp): Record the end of an event.
   • getOngoingEvents(timestamp): Return the count of events ongoing as of the specified timestamp. This can only be queried for timestamps older than 24 hours.
4. Data Storage:
   • Efficiently store large volumes of event data with appropriate indexing and sharding strategies.
   • Optimize for querying the status of events.

Non-Functional:

1. Scalability:
   • The architecture must be distributed and horizontally scalable to handle data volumes that exceed the capacity of a single server.
   • Expect to manage up to 1 million events per second.
2. Fault Tolerance and Consistency:
   • Ensure the system is fault-tolerant, continuing to operate even if parts of the system fail.
   • Support eventual consistency to accommodate distributed data storage.
3. Query Performance:
   • Although real-time query performance isn't required (especially since getOngoingEvents is for historical data older than 24 hours), the system should still provide reasonable query response times.
4. Data Retention and Management:
   • Determine data retention needs based on functional requirements, potentially keeping historical data for extensive analytics and compliance purposes.

Capacity estimation

1. Event Rate:
   • Given: The system must handle up to 1 million events per second.
   • Calculate the daily event ingest rate: 1 million events/second * 60 seconds/minute * 60 minutes/hour * 24 hours/day = 86.4 billion events per day.
2. Event Size:
   • Assume each event includes:
     • event_id: 16 bytes (UUID)
     • event_type: 4 bytes (integer or short string)
     • description: Assume average 128 bytes (could vary)
     • start_time, end_time, and arrival_time: Assume 8 bytes each (typically stored as 64-bit timestamps)
   • Total approximate size per event: 16 + 4 + 128 + 3 * 8 = 172 bytes
3. Daily Data Storage Requirement:
   • Total daily data: 86.4 billion events/day * 172 bytes/event = 14.85 TB/day
4. Retention and Historical Queries:
   • Determine how long data needs to be retained. If, for example, we need to keep data for one year:
   • Annual storage requirement: 14.85 TB/day * 365 days = 5.42 PB/year

API design

POST /api/events

Authorization: Bearer

Content-Type: application/json

{

"event_type": "USER_SIGNUP",

"source": "mobile_app",

"payload": {

"user_id": "abc123",

"timestamp": "2025-07-11T18:00:00Z"

}

}

GET /api/events/{event_id}

Authorization: Bearer

GET /api/events?type=USER_SIGNUP&start=2025-07-01&end=2025-07-10

Authorization: Bearer

POST /api/notifications

Authorization: Bearer

Content-Type: application/json

{

"user_id": "abc123",

"message": "Welcome to the platform!",

"event_id": "evt-789xyz"

}

Database design

Store event in Nosql DB, like Cassandra.

CREATE TABLE events_by_user_and_date (

user_id TEXT,

event_date DATE, -- YYYY-MM-DD, extracted from created_at

created_at TIMESTAMP,

event_id UUID,

event_type TEXT,

source TEXT,

payload TEXT, -- or JSON string

status TEXT,

PRIMARY KEY ((user_id, event_date), created_at)

) WITH CLUSTERING ORDER BY (created_at DESC);

High-level design

• Event Producer: Publishes events to the message queue.
• Message Queue: Buffers the events, providing fault tolerance, absorbing spikes, and enabling asynchronous processing.
• Event Processor: Consumes and processes events from the queue, updating the Events Database.

Request flows

1. Event Producer → Ingestion API

HTTP POST to /api/events

Payload: JSON event data

API validates and forwards the event

2. Ingestion Service → Kafka

The service publishes the event to a Kafka topic (e.g., events.incoming)

3. Kafka → Event Processor

Event Processor (Kafka Consumer):

Consumes the event

Validates/enriches/transforms it

Persists to Cassandra (events_by_user_and_date, etc.)

Publishes processed status to another Kafka topic (e.g., events.processed)

4. Event Processor → Notification Service

If event type requires notification:

Sends message to Notification Service

OR pushes to notifications.to_send Kafka topic

5. Notification Service → WebSocket Server

Sends real-time notification to connected WebSocket clients:

Message includes event ID, type, timestamp, etc.

6. Client Receives Notification

WebSocket client displays the update

Detailed component design

1. API Gateway

• Responsibilities:
  • Handles all inbound HTTP requests
  • Performs authentication (e.g. JWT validation)
  • Routes requests to internal services
• Tech: NGINX, Kong, API Gateway (AWS/GCP), Istio (for service mesh)
• Endpoints:
  • POST /api/events → Ingestion Service
  • GET /api/events → Query Service
  • POST /api/notifications → Notification Service

2. Event Ingestion Service

• Responsibilities:
  • Receives external events via REST
  • Validates payload schema (JSON Schema or Avro)
  • Publishes to Kafka (events.incoming)
  • Returns event ID to producer
• Tech: Node.js / Python / Java + Kafka producer lib
• Kafka Topic: events.incoming

3. Kafka Cluster

• Responsibilities:
  • High-throughput event transport
  • Topic-based pub/sub decoupling between services
• Key Topics:
  • events.incoming
  • events.processed
  • notifications.to_send
• Config:
  • Replication: 3x
  • Partitions: based on volume
• Tech: Apache Kafka or Confluent Platform

4. Event Processor

• Responsibilities:
  • Kafka consumer on events.incoming
  • Transforms, enriches, validates data
  • Writes to Cassandra tables:
    • events_by_user_and_date
    • events_by_type_and_date
    • events_by_id
  • Emits events.processed to Kafka
  • Triggers notification events if needed
• Tech: Java (Spring Boot), Go, or Python + Kafka Consumer
• Fault Tolerance: Uses Kafka consumer groups with auto-retry

5. Query Service

• Responsibilities:
  • Serves search/read APIs
  • Optimized queries for Cassandra based on query patterns
  • Pagination support
• Tech: Node.js/Go + Cassandra driver
• Queries:
  • GET /api/events?user_id=X
  • GET /api/events?event_type=Y

6. Notification Service

• Responsibilities:
  • Consumes from notifications.to_send or receives HTTP trigger
  • Formats notification payload
  • Sends via:
    • WebSocket
    • Email (SMTP/SendGrid)
    • SMS (Twilio, optional)
  • Logs delivery status to Cassandra
• Tech: Python (FastAPI), Node.js, or Go

7. WebSocket Gateway

• Responsibilities:
  • Accepts WebSocket connections from users
  • Authenticates users (token or session)
  • Subscribes clients to channels (e.g., user_id or topic)
  • Pushes real-time messages from Notification Service
• Tech: Socket.IO, WS, or custom using Go / Elixir / NATS
• Scaling: Horizontally scalable with sticky sessions (Redis Pub/Sub or Kafka backplane)

8. Database: Cassandra

• Schema: Time-series, user-centric
• Tables:
  • events_by_user_and_date
  • events_by_type_and_date
  • events_by_id
  • notifications_by_user
• Config:
  • Replication factor: 3
  • TTL for events (optional)

9. Schema Registry (Optional but Recommended)

• Responsibilities:
  • Stores Avro/Protobuf/JSON schemas
  • Used during ingestion and processing
• Tech: Confluent Schema Registry, Redpanda, or custom

10. Monitoring & Observability

• Logging: Fluentd, ELK, or Loki
• Metrics: Prometheus + Grafana
• Tracing: OpenTelemetry + Jaeger

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?