Requirements

Functional Requirements:

List the key functional requirements for the system (Ask the AI for hints if stuck)...

• New sports events can be detected and registered on the platform.
• Users can see real-time scoreboards, player statistics and game timelines.
• Users should get push notifications for score changes, injuries or other major events.

Non-Functional Requirements:

List the key non-functional requirements (performance, scalability, reliability, etc.)...

• The system should be available 99.99% of time.
• The system can be eventually consistent, where different users see consistent results after small delay.
• The system should be able to easily scalable to more users.
• The p99 latency for viewing real time data should be less than 200ms.

Capacity Estimation

Estimate the scale of the system. Consider daily active users, read/write ratio, storage requirements, bandwidth, and any relevant QPS calculations...

Assuming 100 million DAU, and on peak hours, 40% of the users are online at the same time and are watching matches. While watching matches, the user checks scores/status on average once per 5 seconds.

The peak read QPS = 100 million DAU * 40% * 0.2 reads per second = 8 million

Assuming that we have at most 1000 matches concurrently. For each match, we write latest match data every 2 second, the peak write QPS = 1000 * 0.5 writes per second = 500

For network bandwidth, at peak, assuming 1kb of data is transmitted on average per user, the bandwidth needed is:

1kb * 40 million = 40GB per second

For storage, we need to store:

1. Users metadata
2. Matches metadata and timeline trends

Assuming for users, each user's metadata is 1kb, and for matches, each match metadata is 5kb.

Assuming we have 100 million users and 1 million matches.

user metadata storage = 100 million * 1 kb = 100GB

matches metadata storage = 1 million * 5kb = 5GB

Both are trivial for a modern database.

API Design

Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...

For sports event to be registered:

POST v1/register_event {

event_name: String,

event_type: String,

event_time: Timestamp,

expected_event_duration: String,

event_metadata: String

}

For real-time updates for events:

POST v1/update_event_real_time {

event_id: UUID,

event_update: String

}

To read real-time event developments and statistics:

GET v1/read_live_event_stats {

event_id: UUID

}

GET v1/player_stats {

player_id: UUID

}

GET v1/game_timeline {

event_id: UUID

}

For pushing real-time events to users:

POST v1/push_event {

event_id: UUID,

user_id: UUID,

event_update: String

}

High-Level Design

Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.

In this system, we have the write path of sports events creation, and real time events ingestion, as well as the read path of browsing events feed and reading real-time event updates.

On the write path, we have a real-time data write service that ingests sports events from external sports events vendors.

For each real-time sports match update, we trigger an event on Kafka queue.

There are 3 downstreams consumers of Kafka:

1. The game events processor, which ingests the real time events, modifies them into correct data models, and write them to the cassandra database.
2. The player stats processor, which ingests the real time statistics of each player in the game, converts data into the right data model and write them to the player database.
3. The web socket servers. For any major events, without users polling, we directly use the websocket servers to push latest changes to users' devices.

On the read path, all requests from the client go through load balancers, which balance traffic based on consistent hashing. Then they go through API gateway, which does authorization, authentication and rate limiting.

There are 2 types of reads:

1. For browsing the events feed, it requests the sports events relational database, and fetches metadata about sports events.
2. For viewing real time sports games updates, as well as player statistics. It first queries redis. If not available, it then queries the underlying cassandra database and populates the entry in redis.

In case of redis cache, we will read directly from the database. To avoid overwhelming the database, we will implement circuit breakers on the request side, and load shedding on the database side.

Given the database is eventually consistent, we will implement monotonic read on the client side. We will record the latest timestamp of events that the client has received, and sent that in the request to servers. The servers can't return an older event that is older than the client timestamp.

This approach is risky if we have client-side clock skew. In that case, we can also send vector clocks. However, that introduces additional complexity and additional storage need. So it might not be the right choice for sports events update. Timestamp might still be the better option.

In case of kafka queue being flooded with overwhelming events from multiple hot sports events, we will prioritize the high-priority events in the kafka topics, and drop low-priority events as a load shedding mechanism.

Database Design

Define the data model. Identify the main entities, their attributes, and relationships. Consider the choice of database type (SQL vs NoSQL) and justify your decision based on access patterns...

In this design, there are 3 types of data we need to store:

1. Real time events data
2. Registered sports events data
3. Real time player statistics data

For real time events data and real time player statistics data, we will use cassandra. While for registered sports events data, we will use a relational database.

For real time events, cassandra is a good fit because:

1. It is highly scalable horizontally, and can easily support huge write throughput
2. It is highly available across different geo regions

The tradeoffs are:

1. The data needs to be denormalized, which is acceptable in our use case
2. The data is eventually consistent, which is acceptable in our use case
3. No complex joins can be done, which is acceptable in our use case

Here's the sample data schemas:

table sports_game_updates {

game_id: UUID,

game_name: String,

event_update: String,

event_update_time: Timestamp

}

The partition column is game_id and the clustering key is event_update_time

table sports_game_player_statistics {

game_id: UUID,

player_id: UUID,

update_time: Timestamp,

update_event: String

}

The partition column is game_id while the clustering keys are player_id and update_time.

For the real time events, to reduce read latency, we will use a redis cluster as a read aside cache.

For writes, they are written directly to the database, and the corresponding entries in the cache are invalidated.

For reads, they access redis first, and if not available, we query the database and write the updated results to the cache.

The benefits are:

1. We don't end up cache everything in cache, saving memory
2. The implementation is in application layer, less complex
3. The read latency is low when cache hit rate is high.
4. Data is durable as we store data in the database.

The tradeoffs are:

1. The system is eventually consistent. However, we are not very likely to serve stale data since we invalidate cache in database writes.
2. The write latency is relatively high as we write to the database.

Both tradeoffs are acceptable in our use case.

For the registered sports events data, however, we want to use a relational database. The benefits are:

1. Strong ACID properties guarantees
2. The sports events data is easily represented by relational data models
3. We can do complex join queries if needed.

The tradeoffs are:

1. This table won't be easily horizontally scalable
2. The table schema isn't flexible

However, these are acceptable tradeoffs as the registered events need to be consistent, and we won't have frequent schema changes.

A sample data schema is like:

table registered_sports_events {

event_id: UUID,

event_name: String,

event_type: String,

event_time: Timestamp,

event_metadata: String

}

Detailed Component Design

Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.

In this design, there are 3 types of data we need to store:

1. Real time events data
2. Registered sports events data
3. Real time player statistics data

For real time events data and real time player statistics data, we will use cassandra. While for registered sports events data, we will use a relational database.

For real time events, cassandra is a good fit because:

1. It is highly scalable horizontally, and can easily support huge write throughput
2. It is highly available across different geo regions

The tradeoffs are:

1. The data needs to be denormalized, which is acceptable in our use case
2. The data is eventually consistent, which is acceptable in our use case
3. No complex joins can be done, which is acceptable in our use case

Here's the sample data schemas:

table sports_game_updates {

game_id: UUID,

game_name: String,

event_update: String,

event_update_time: Timestamp

}

The partition column is game_id and the clustering key is event_update_time

table sports_game_player_statistics {

game_id: UUID,

player_id: UUID,

update_time: Timestamp,

update_event: String

}

The partition column is game_id while the clustering keys are player_id and update_time.

For the real time events, to reduce read latency, we will use a redis cluster as a read aside cache.

For writes, they are written directly to the database, and the corresponding entries in the cache are invalidated.

For reads, they access redis first, and if not available, we query the database and write the updated results to the cache.

The benefits are:

1. We don't end up cache everything in cache, saving memory
2. The implementation is in application layer, less complex
3. The read latency is low when cache hit rate is high.
4. Data is durable as we store data in the database.

The tradeoffs are:

1. The system is eventually consistent. However, we are not very likely to serve stale data since we invalidate cache in database writes.
2. The write latency is relatively high as we write to the database.

Both tradeoffs are acceptable in our use case.

For the registered sports events data, however, we want to use a relational database. The benefits are:

1. Strong ACID properties guarantees
2. The sports events data is easily represented by relational data models
3. We can do complex join queries if needed.

The tradeoffs are:

1. This table won't be easily horizontally scalable
2. The table schema isn't flexible

However, these are acceptable tradeoffs as the registered events need to be consistent, and we won't have frequent schema changes.

A sample data schema is like:

table registered_sports_events {

event_id: UUID,

event_name: String,

event_type: String,

event_time: Timestamp,

event_metadata: String

}

In this system, we have the write path of sports events creation, and real time events ingestion, as well as the read path of browsing events feed and reading real-time event updates.

On the write path, we have a real-time data write service that ingests sports events from external sports events vendors.

For each real-time sports match update, we trigger an event on Kafka queue.

There are 3 downstreams consumers of Kafka:

1. The game events processor, which ingests the real time events, modifies them into correct data models, and write them to the cassandra database.
2. The player stats processor, which ingests the real time statistics of each player in the game, converts data into the right data model and write them to the player database.
3. The web socket servers. For any major events, without users polling, we directly use the websocket servers to push latest changes to users' devices.

On the read path, all requests from the client go through load balancers, which balance traffic based on consistent hashing. Then they go through API gateway, which does authorization, authentication and rate limiting.

There are 2 types of reads:

1. For browsing the events feed, it requests the sports events relational database, and fetches metadata about sports events.
2. For viewing real time sports games updates, as well as player statistics. It first queries redis. If not available, it then queries the underlying cassandra database and populates the entry in redis.

In case of redis cache, we will read directly from the database. To avoid overwhelming the database, we will implement circuit breakers on the request side, and load shedding on the database side.

Given the database is eventually consistent, we will implement monotonic read on the client side. We will record the latest timestamp of events that the client has received, and sent that in the request to servers. The servers can't return an older event that is older than the client timestamp.

This approach is risky if we have client-side clock skew. In that case, we can also send vector clocks. However, that introduces additional complexity and additional storage need. So it might not be the right choice for sports events update. Timestamp might still be the better option.

In case of kafka queue being flooded with overwhelming events from multiple hot sports events, we will prioritize the high-priority events in the kafka topics, and drop low-priority events as a load shedding mechanism.

There are several other key areas:

1. We have an idempotency store that stores the idempotency keys for a sports events update. In case of network blips that cause retries on Kafka, we have idempotency key for that event so the same event won't be processed twice.
2. In redis, we not only cache the raw event data, but also the ordered chronological timeline view of a game. So any request for viewing event timeline is fast.
3. For the web socket servers, it will use auto scaling based on CPU usage to handle spiky traffic. Same auto scaling is applied to Kafka consumers as well to avoid consumer lag.
4. The Kafka is partitioned by game_id, so all events for the same game end up ordered on the same partition. If a processor crashes, it resumes from the last committed offset.