System requirements

Functional:

1. Handling Out-of-Order and Delayed Events:
   • The system must manage events that arrive out of order, such as an endEvent arriving before a startEvent.
   • It must also accommodate events that may arrive significantly delayed (e.g., hours or days after their actual occurrence).
2. Event Duration:
   • An event can last up to one hour.
3. APIs:
   • startEvent(event_id, timestamp): Record the start of an event.
   • endEvent(event_id, timestamp): Record the end of an event.
   • getOngoingEvents(timestamp): Return the count of events ongoing as of the specified timestamp. This can only be queried for timestamps older than 24 hours.
4. Data Storage:
   • Efficiently store large volumes of event data with appropriate indexing and sharding strategies.
   • Optimize for querying the status of events.

Non-Functional:

1. Scalability:
   • The architecture must be distributed and horizontally scalable to handle data volumes that exceed the capacity of a single server.
   • Expect to manage up to 1 million events per second.
2. Fault Tolerance and Consistency:
   • Ensure the system is fault-tolerant, continuing to operate even if parts of the system fail.
   • Support eventual consistency to accommodate distributed data storage.
3. Query Performance:
   • Although real-time query performance isn't required (especially since getOngoingEvents is for historical data older than 24 hours), the system should still provide reasonable query response times.
4. Data Retention and Management:
   • Determine data retention needs based on functional requirements, potentially keeping historical data for extensive analytics and compliance purposes.

Capacity estimation

1. Event Rate:
   • Given: The system must handle up to 1 million events per second.
   • Calculate the daily event ingest rate: 1 million events/second * 60 seconds/minute * 60 minutes/hour * 24 hours/day = 86.4 billion events per day.
2. Event Size:
   • Assume each event includes:
     • event_id: 16 bytes (UUID)
     • event_type: 4 bytes (integer or short string)
     • description: Assume average 128 bytes (could vary)
     • start_time, end_time, and arrival_time: Assume 8 bytes each (typically stored as 64-bit timestamps)
   • Total approximate size per event: 16 + 4 + 128 + 3 * 8 = 172 bytes
3. Daily Data Storage Requirement:
   • Total daily data: 86.4 billion events/day * 172 bytes/event = 14.85 TB/day
4. Retention and Historical Queries:
   • Determine how long data needs to be retained. If, for example, we need to keep data for one year:
   • Annual storage requirement: 14.85 TB/day * 365 days = 5.42 PB/year

API design

Define what APIs are expected from the system...

Database design

Store event in Nosql DB, like Cassandra.

Event table has {

event_id

description

event_type

timestamp

status

start_time

end_time

}

High-level design

• Event Producer: Publishes events to the message queue.
• Message Queue: Buffers the events, providing fault tolerance, absorbing spikes, and enabling asynchronous processing.
• Event Processor: Consumes and processes events from the queue, updating the Events Database.

Request flows

Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...

Detailed component design

Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?