Loading...
Let's count we have 100 million of active users.
Every user work out 2 time a week for 1 hour.
User transmits such information every second about 100 bytes:
So it's 720kb per week for one user
72 terabytes for 1 week.
150 terabytes for 1 year
300 terabytes(one copy for replication) for 1 year or 1500 terabytes for 5 years.
startTrain(apiKey,location,timestamp) starts new training.It returns trainingId.
track(apiKey, trainingId, timestamp, user metrics) passes the user metrics to server.
finishTrain(apiKey, trainingId,timestamp) finishes the current training, it returns the final result of the workout.
getSummary(apiKey, startDate, endDate) returns the workout summary for the given time interval.
The primary data model for this system is three tablets: one table is for the users, the second is for workouts, and the third is for workout data.
It doesn't require strong consistency for users and workouts. We write the user and workout data to NOSQL like AWS Dynamo.
User table:
id(10 bytes)
first and last names (200 bytes)
phone number (50 bytes)
email (100 bytes)
createdAt (8 bytes)
Workout table:
id (10 bytes)
description(300 bytes)
userId (10 bytes)
startDate(8 bytes)
endDate (8 bytes)
Workout data:
id (8 bytes)
userId (8 bytes)
trainingId (8 bytes)
latitude: (2 bytes)
longitude (2 bytes)
date (8 bytes)
heartbeat (2 bytes)
elevation (8 bytes)
Goals table:
id (10 bytes)
userId (8 bytes)
date (8 bytes)
text (1000 bytes)
Analytics table:
id (10 bytes)
userId (8 bytes)
trainingId (8 bytes)
start date (8 bytes)
end date (8 bytes)
min speed (2 bytes)
max speed (2 bytes)
average speed (2 bytes)
distance (3 bytes)
The analytics table will be updated after the user finishes the workout.
It needs to create the indexes to search in DB faster.
See the diagram from the high-level architecture.
API Gateway provides DDoS protection and TLS termination, and forwards request to the right service nodes.
The user data service processes the request about the user and her workout data. This service stores the data in an RDBMS, such as MySQL or Postgres. The tracing service receives the workout data like the user speed, distance, heartbeats, and geolocation and stores that in NO SQL, which makes writing very quick. Then workout data are ingested by the Analytics service and uses MapReduce to get workout summaries and stores that in its own No SQL.
startTrain(apiKey, geolocation, timestamp) opens a new workout in the User data service and creates new records in workout tables, the user data service returns the new unique trainingId that would be used to access workout data.
track(apiKey, trainingId, timestamp, user metrics) sends the workout data to the Tracking service from the client application. The tracking service persists those data: the user speed, geolocation, heartbeats, and distance to No SQL. In this case, No SQL is the best solution to make writing. We may use AWS Dynamo to store that data.
finishTrain(apiKey, trainingId, timestamp) finalizes the current workout and updates endDate in the user data service DB.
The Analytics service requests the workout data from the tracking service to prepare different analytics report data for the user application dashboards.
The user may use the mobile or web client application to track the workout progress.
Performance and scalability of tracking endpoint is extremely important for this system. As such, we employ No SQL to scalable user load evenly. Also, we may Kubenates horizontal autoscaler to regulate node number. No SQL uses replication and partitioning to be fault-tolerant. Using indexes (traningId, timestamp) would help for Analytics service to process the workout data quickly.
startTrain and finishTrain would be rarely used compared to track
getSummary would return the workout summary for the given interval. Most frequent data would be cached in Redis.
The client application should be developed in such a way, that if it loses the connection and restores it, it sends the accumulated workout data to the backend.
We may use WebSockets to connect the client application. It wouldn't be critical if some data got lost.
track API would be partitioned by traningId and timestamp. It gives the data even distribution. getSummary API may use userId as the partition key. It wouldn't give us even distribution but the system load is low and it doesn't hurt the system performance.
If the user starts the workout and doesn't finish it, the background job closes such workouts.
Tracking service is a stateful service and stores the websocket connection information.If clthe ient application loses the connection, it would route to the same backend node and restore the connection easily.
Because the Analytics MapReduce is long-built, we may return summary information for the last workout.
All the components - Load Balancers, Web Servers, Cache and Database should have multiple instances for improved availability. There should be robust monitoring and alerting systems on them.
All nodes can fail. Let's look at important failure cases.
If the user data servicefails (hardware failure, crash, software bug, network partition, slowness ...), it would directly impact the most time sensitive operation of this system, i.e., startTrain() and finishTrain(). To mitigate this, we should always run multiple user data service nodes. It is s stateless service, so we can have multiple nodes of the same service. We can use a coordination service, e.g., ZooKeeper, to track which nodes are alive (i.e. sending regular heartbeat to ZooKeeper), which are likely dead (i.e. not sending heartbeats for some time), which nodes should be taking requests.
These user data services are stateless and if one of them falls down, the Kubernetes we can use it to orchestrate container runs new service instances.This service uses MySQL and MySQL supports master and slave nodes.If leader is down, the one of slave nodes would be promoted to leader. To decrease the inactivity timeout for MySQL, we may support the leader-to-leader replication, when the standby leader stores updated data and may get promoted immediately to a new leader.
Losing the tracking service would also impact tracking and analytics API functionality. But the client application may mitigate the failures by accumulating workout data and after the connection is restored, the the client application will send the accumulated workout data. The tracking service will now have to access the database for each of these workouts. Increased load on the database may even have a cascading impact - the database gets slower and slower, and Tracking services retry, making the database even busier - ultimately resulting in the database crash.
We have multiple mitigations:
a. The tracking service should have a mitigation strategy to avoid overloading the database. For example, exponential backoff before retrying, rate-limiting, and circuit-breaking.
To make the system more fault-tolerant, we deploy the tracking service in a few instances. If the tracking service is stateful, it's not a problem for Kubernates to start a new instance if one of the tracking service instances falls.
No SQL is managed by AWS and it's fault-tolerant AWS service.AWS guarantees 99,9999 availabitly for AWS Dynamo.
Failure in Analytics Service.
The analytics service may fail and in this case, we may cache the most requested workout summaries. Analytics services are stateless and Kubernetes manages a few instances of those services. The analytics service will use the AWS Dynamo which will be managed by AWS. If MR(MapReduce) fails, we would restart that again because and requests necessary workout data from the tracking service.
Scalability - track and getSummary API:
As the number of write requests to track and getSummary API increases, it might put too much pressure on the database, causing slowness, errors, or even crashes.
To avoid this, we can introduce a message queue to buffer the requests. API Gateway would push a message in the message queue, representing the request. The tracking and analytics services would pull from the queue, process the request, and notify the API Gateway client requests processed with a hook. The system can inform the client with long polling.
Add the metrics, collect analytics to predict the user request peaks.