Requirements

Functional Requirements:

• Notifications of sore updates, player stats, in-game events, and end-of-quarter/period updates
• Mobile notifications, web notifications, email
• The ability to see all stats on our website
• Different kinds of sports: soccer vs basketball

Non-Functional Requirements:

• One user should not receive duplicate notifications for events (once delievery)
• 1M concurrent users watching 50 parallel games
• Each even lasting around 2 hours
• 50 notifications per user/game
• 50 million notifications per game
• 7K events per second
• x5 peak traffic -> 35K events/sec
• Users should be notified of an event in less than a minute of it occurring.

API Design

Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...

High-Level Design

Game Data goes into the data pipeline. It's events like, "X player received a ball and succeeded in a pass" or "Y player scored a goal". Let's use protobufs. Each event has gameID, and a oneof GameEvent. This lands into a FIFO queue, where each queue represents 1 game.

From the queue begins the data processor we can either use a realtime streaming technology here, like spark/flink, but based on the speed and complexity we'd rather use servers with an Actor Model instead (like Akka). Games are stateful with rich historical context, requiring more than just a bunch of context-independent workers. Each "Game" and "player" will be actors, and the messages will be things that happen in that game or to particular players. That way, we can highly complex, low latency, inherently distributed logic about what happens in each game. It also reads and writes a historical record of every game to a document database (not relational, because of the sparseness of data and unstructured nature of supporting many kinds of sports).

Next, that fleet of Actors publishes to another queue notifications that need to happen. This is our messaging microsevice. It also writes to our a database that supports our web microservice.

Detailed Component Design

Messaging MicroService:

• A Kafka queue gets the game-event message from the GameEvent MicroService (fleet of Akka machines). Each GameID has its own queue, and it's partitioned by event types.
• An auto-scaling consumer-group pulls from that queue and determines who should receive notification
• This fans out into another queue, the "User Notification Queue". Different consumer groups handle different kinds of notifications: Web (server-side events), Mobile, Email, etc.
• Messages that cannot be sent are pushed to a dead letter queue which is closely monitored.

Web Microservice:

• Publishes realtime events using Server events. No need for websockets since this is one-way communication; just keep the HTTP(S) socket open.
• We need a mechanism to map user to connected web-server. We can use Redis or a hash-ring load balancer. The problem with reling on an hash-ring LB is that our fleet can resize, leading us into trouble for finding instances. So we'll use Redis.
• Database is another document store with leader-follower. We choose leader/follower because this service is high reads, low-writes. For redundancy we can have another DB on standby in another cloud-region, but 1 leader DB handles all the writes. MongoDB with 8vCPU can support the 35K/sec (assuming 5K/sec/vcpu) volume.
• Connected web-clients read from the "follower" DBs. 1M concurrent users all have open sockets to our web-server fleet and are receving realtime updates. The system auto-scales to handle the load.