Loading...
1) Log activities, which can have associated metrics/details with them. (Time/distance for running, weight/reps/sets for weightlifting, etc)
1a) Users will be able to see their activities and view the metrics for that activity
2) Users will be able to see a recent activities list that shows their 20 most recent activities
3) Users will be able to set goals that can be related to one or more activities
3a) Users will be able to see their progress on said goals update based on their recent activities
3b) Users will be able to set the start date/end date of a goal
4) Progress monitoring
5) Users should be able to login/register and create a profile. All dataa and workouts should be synced to a user. When logging into a different device, that device should have the latest sync of data.
1) Durability - Data should retain from session to session.
2) Availability - Users should always be able to see their sessions and goals (so long as they are connected to the internet)
3) Consistency - Users should be able to see their sessions across devices regardless what server/replica they are connected to. The latency for this does not need to be extremely low, since there is no urgency to replicate data. A few minutes of waiting should be fine. The main thing is that the user should be able to see any newly logged sessions/workouts/data on the same device they logged it on (read your own writes).
4) Support for multiple platforms (mobile with iOS and Android and potentially a desktop app)
5) Data encryption and privacy. Data needs to be encrypted since a lot of health data is sensitive and needs to be unreadable even on the server end.
6) Scalability: We need to be able to support the data storage and queries/requests of many users at once. For now, let's assume a maximum of 10,000 DAU. We can of course plan for a larger user base and design around that. But this is my base assumption.
7) In terms of performance, we do want all logging operations to be done within a second (locally), propagating to database may take longer.
Estimate the scale of the system. Consider daily active users, read/write ratio, storage requirements, bandwidth, and any relevant QPS calculations...
So based on our initial 10k daily active users, let's assume that on average, each users logs 2 workout sessions per day.
For a breakdown, all of the workout data can be broken into 1KB.
At most, they have a few sentences of comments for the workout, a byte to identify what type of activity it was, and a few ints/doubles to track metrics such as start/end time, duration, effort, calories burned, reps per exercise, etc. But at worst, let's give a 50% tolerance and assume 1.5KB per exercise. Given that, each user would require 3KB per day for their workouts.
For goals, these would be less common (most likely once a week/month/year). Let's assume a monthly goal sis the most common. A goal will be composed of a few center pieces of data similar to the workouts, but also needs flexibility to maintain references to previous/related workouts. Let's allocate 2KB per goal which should give ample enough space to reference many workouts via id (4 bytes). This can be grown as well. Assuming that it's once per month and 30 days are in a month, then we get around 0.07KB per user per day.
In total, each user requires around 3.07KB per day, which we can just estimate as 3KB per day.
Assuming 10k users, we end up with 3KB *10,000 which leaves us with 30,000 KB per day which is 30MB per day.
That means per year, we'd end up with 30*300 MB per year. 9000MB -10000MB is 9-10GB per year just for the storage of the activities and the goals.
We'd also need to keep track of the user profile and other related areas. So let's go with 20GB per year. Assuming a tolerance in case we have a huge sudden influx of users, let's go with 40GB.
Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...
At a high level, we need APIs to support the functional requirements we specified earlier
POST api/new_workout
PUT api/update_workout
GET api/workout_data
POST api/create_goal
GET api/goal_progress
GET api/recent_activities
PUT api/update_user_profile
DELETE api/delete_activity
Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.
So at a high level, I've utilized an API gateway to handle and route the relevant requests to their associated service. The API gateway will also handle rate limiting and authentication for us so the services/servers don't have to explicitly handle them.
For the workflow, some data such as static assets, images, videos, etc can be stored at the CDN level for easier transmission. The CDN can also store some hotspot data as well if necessary. Any other data/requests goes to the API gateway.
Based on the request URL, the API gateway will route the requests to the appropriate service.
For now, I've created 3 microservices that handle their own set of requests. The architecture is intended to be stateless to allow for any client to query any service worker for their request and missing credentials, tokens, etc.
The activity service handles the request for recent_activities and accumulates the 20 activities at a time for a user.
The goal service handles the simple new_goal and update goal requests, but also handles the more complicated goal_progress API. At a high level, the goal_progress needs to check the workouts done recently but after the last time the goal was queried. Based on the results from the query, it then needs to update the goals table with the progress calculated (if there is any change). Because of the necessary querying and accumulation, I thought about separating the server into services instead. After calculating, the goals services will update the goals table with the updated progress and the last queried attribute. The workout service would send raw numbers and the client would render the graph. It would not make sense for the server to render the graph and send that graph to the client. The client would query for the goal data and whether or not it has been changed, the service would send back the % of each subsection of the goal. For now, I've kept it simple to just % of the goal achieved. The client would get this info and render the graph.
The workout service is used to query, update, and add new workouts.
All Get requests can be first routed to the cache to see if the relevant response exists there to relieve load on the database.
Define the data model. Identify the main entities, their attributes, and relationships. Consider the choice of database type (SQL vs NoSQL) and justify your decision based on access patterns...
So here, a SQL/relational database would be better suited over NoSQL. the number of requests and necessary storage can easily fit inside a single database node. We'll of course add read replicas (one at first) as well as replication across datacenters to ensure durability. The data is also more suited for 'relationships' due to the association of users and unique workouts together. Allowing for joins via tables gives us an easier time reconcile data. In the future, if necessary, we can scale this up by adding more database nodes and partitioning if necessary.
The SQL Table Design would look something like this
Users Table
user_id
user_name
name
date_of_birth
Workouts Table
workout_id
user_id (foreign key)
workout_type (RUNNING,SWIMMING, ...)
date
Exercise Table
workout_id
number_of_reps
weight
duration
effort
Goals Table
goal_id
user_id
start_date
end_date
workout_type
last_queried
progress
User Preferences Table
preference_id
user_id
preference_
preference_value
In terms of the ratio of reads to writes, I'm expecting there will be a lot more reads than writes as they just want to see progress on goals and review their past progress.
For faster access, we'd generally want to optimize queries on user_id. Adding an index in the workouts or goals tables will allow for sorting and querying data more easily. For workouts, we also want to query for workout_type as well for the goals progress.
Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.
Most of the APIs/workflows for getting and updating values are relatively straightforward, so I won't go too deep into them.
One of the more important workflows is how to actually process the "goal progress". As mentioned earlier with the high level design, the goals service needs to query all the workouts for the user and then check against the goal and then return the relevant % accomplished. The main challenge is that the user can add new workouts which can increase the % or they can delete workouts which may lower the %. For example, if the user wants to get to 25 lbs for bicep curls, they can add a new workout which specifies 5lbs with 10 reps and 10lbs with 5 reps. At a very simplified rendition, we'd present 40% finished. If the user adds another workout of 15 lbs, then we'd need to query and update the goals database to reflect 60% there. The challenge comes when users delete workouts and the relevant rows in the workout are removed. In the high level, I mentioned that we can avoid querying prior timestamped rows based on the 'last updated' timestamp in the goal service. The query would look something like
SELECT exercise.weight
FROM exercise JOIN workout
where user_id = specified_user_id AND workout_type = goal.workout_type AND date > goal.last_queried.
After receiving the results, we'd get the MAX of the queries and use that to judge the progress.
But as I mentioned earlier, the user can delete rows, which invites challenges. A tentative solution would be to remove the last queried parameter and always query all activities since the start date of the goal, but that would cause extra processing for all cases, not just the edge case of the user removing a workout. Another possible solution would be to to have a stale flag on the "goal table" as an extra attribute. When a workout is deleted, we can join the workouts and goals table. If the workout deleted matches with the goal, then mark the stale flag as true. When querying the goals with the stale flag, then we re-query all exercises from the start date of the workout and then set the stale flag back to false. This adds extra processing to the 'delete' case, but workout deletion should be a rarer case that we can compromise on for the more common case of querying for goal status.
With the service architecture, we do need a way to persist requests. Each service may be busy at the time a new request is sent/received. In that case, we don't want to lose the request and want to process subsequent requests in an orderly manner. Here, we can create a queue for each service that operates on a FIFO basis. Requests are added via the API gateway and then popped by the service. This way, if the service goes down, then the requests still persist and can continue to be processed when the service is spun up. This also allows us to horizontally scale based on the size of the queue. If the queue is above a threshold (let's say 70%), then we can spin up another instance of the service to process requests.
On the cache side, we can keep workout and activities cached with a TTL of 1 day. We don't want to specify too long of a TTL since the user will not update these often. Once they are created, it's unlikely they are updated. If they are updated, then we can use a write through caching strategy to also update the queue. For the goal progress, we can follow a similar approach but with a 1-2 hours TTL since goals are updated by activities.
For data encryption, all data should be encrypted. For analysis, the server should have the key (especially for goals). When sending said data, the client should encrypt all fields relevant to health. The server would use their copy of the key to decrypt and operate on that data. When sending said data back, that data is encrypted using the same key and then the client will decrypt it using its copy of the key.
We could have it so when the user creates an account, the server associates a secret key with it or we could have it as a per-device key. For the single user key approach, the client would maintain a copy for decrypting the data from the server responses and the server maintains a copy for decrypting and encrypting the results/query response.
In terms of database failure/redundancy, we can utilize a single-leader replication model based on the data constraints that we have. With a single leader, we can route writes to the single leader and have 1-2 synchronous followers and a few asynchronous followers. Reads will go to read replicas depending on the scenario. If reading after a write, we may want to route only to the synchronous followers or the leader to ensure they can see the most recent write they have done. Otherwise, reads can go through any of the replicas. In case the leader goes down, we can utilize a consensus algorithm to determine who the new leader is. In general, we can use a heartbeat mechanism to keep track of instance health. If heartbeats haven't been received from an instance in a while, then we can mark it as suspicious. If further heartbeats are missed, then we can mark it as dead. For the leader, we may have a partitioned leader and end up electing a new leader. If the old leader comes back, then we need to ensure that we don't have 2 nodes thinking that they're both the leader. In some cases, we can manually shut down the old leader. But a more automated way is to use an epoch number. When the old leader sends a write with an older epoch #, it'll be rejected. When teh request is rejected, the leader will realize it's no longer the leader and step down.
Future Requirements/Enhancements