System requirements

Functional:

• Users must be able to enter several different types of activities. Let's start with running and weightlifting.
• Users must be able to see a nice visualization of the activities they've done, and how they compare to previous activities
• Users must be able to enter goals for certain activities and also see progress towards those goals.
• Users must be able to update goals and activities.
• (future) App must integrate with third party tracking services to facilitate activity entry, biometrics tracking
• (future) users must be able to share their activities with others.

Non-Functional:

• The app must respond quickly. Users should be able to see activities locally before they get synced to our backend.
• Users must be able to see all of their data from either mobile or desktop, with reasonable latency for full consistency
• The app must be highly available
• The app must have be able to tolerate a weak network connection as people will often be uploading while off wifi. If an activity is unable to be uploaded immediately, uploads will be retried with backoff.
• The app must durably store a users data forever.

Capacity estimation

• 100k MAU uploading activities (2 ^ 17)
• Each user uploads 8 activities a month on average
• Let's assume each activity will have 1MiB of data (high end - but covers things like runs where heart rate data, stride count, breath rate, etc is recorded)
• (2 ^ 17) * (2 ^ 3) * (2 ^ 20) = 2 ^ 40 bytes/month of storage -> 2 ^ 10 GiB -> 1 TiB month
• Let's assume we care about storage costs on a 20 year horizon at this steady state, we're looking at 1 TiB * 12 months/yr * 20 yr = 240 TiB storage. Assume a replication factor of 3x -> 720 TiB. Not too bad, even if we keep this on SSD it's still very cheap, and negligible on HDD.
• For network ingress costs: we have (2 ^ 3) * (2 ^ 17) = 2 ^20 uploads a month, 2 ^ 12 * 2 ^ 4 * 2 ^ 5 seconds per month... so right around 0.1 upload a sec. (2 ^ 21/ 2 ^ 20) Which means we're looking at 100 KiB/sec = 800Kib/s. very cheap.
• Network egress traffic will be assumed to be 3x, we'll assume each user loads a full activity 3 times, and viewing other stuff on our app is negligible cost. 2400 Kib/sec egress.
• I'll assume a user opens the app 10x per activity uploaded, which lands us at 1 QPS to load activities and goals, as we'll assume we do that once per app open. This might be a bit low but it's probably reasonable.

API design

// Activities

UploadActivity(UserId user_id, Activity a); // need timestamp in activity as upload might not be instant

LoadActivities(UserId user_id); // Lightweight view of all activities

LoadActivity(UserId user_id, ActivityId aid); // Loads a full individual activity and progress against

// goals. Only called if this device does not have the activity cached locally, which is unlikely

// Goals

SetGoal(UserId user_id, Goal goal);

ViewGoals(UserId user_id); // Gets full set of goals. Can load all here as goals will be small

UpdateGoal(UserId user_id, Goal goal);

Database design

This is going to be heavier on writes than most workloads. I anticipate that almost all detailed reads of individual activities will be from a cache on the user device.

This data is going to require some notion of interconnectedness for goal tracking (we'll use an aggregation on all previous activities to show progress towards the goal). We definitely don't want a database system that is going to make that incredibly painful unless we're working at a scale that necessitates that, so I'm going to lean towards a standard RDBMS like Postgres.

We're not worried about hot shards/users as there is no activity sharing at this point in time.

Examining a couple of our other goals:

• we can't lose user data, so I'm going to replicate it in at least three places.
• We need low latency for serving ListActivities, and given that we're only looking at 1 QPS of traffic, with no hot shards, we can just put all activities from the past month in a materialized view. Let's assume we have 1KiB metadata/activity * 8 activities/month * 100k MAU, we are looking at 100 * 2 ^ 3 * 2 ^ 17 = 100 * 2^20 = 100MiB in materialized views. Super cheap.
• For ListGoals we'll do a similar calculation and assume we want to store all active goals. We'll assume a user has 8 active goals at a time so we can just copy the above. Another 100 KiB in materialized views.

I'm going to assume our users aren't generally jet-setting so we don't need to worry about frequent cross continent replication. We'll do cross continent replication for disaster aversion but it doesn't have to be frequent as not doing it doesn't come with a hit to the performance of the things we care about.

I'll plan to have a replica in us east, us west, and europe central to start.

I'm not concerned about other performance issues yet as we're not allowing users to share activities with others.

As far as actual DB schema goes, we'll have a user table, and a table for each type of activity we support, keyed by userid. Table schemas per activity will vary, and store raw activity data. For example, for running you might upload a set of timestamped coordinates from the run, and then the duration. Each activity needs a start time, duration, and owning user id, but will likely diverge from there.

High-level design

We want to never, ever lose a user activity (obviously impossible to 100% guarantee but we want to get as close to this as possible). As soon as it is recorded in the app, it needs to get written to a file, so even if the app crashes it is still there. Failure to write it to the file should not crash the app, we should handle that gracefully.

We'll store activities for the app locally in a SQLite database, the standard for mobile apps.

This is very straightforward otherwise, there is no need for a large scale up due to low QPS. In each region, we'll have three active app servers, an active-passive load balancer, and two frontend service replicas to handle TLS termination and communication with app servers. Sophisticated load balancing is not required as traffic is so low. We'll use round robin for now.

Request flows

Every request will always reach the database. That is totally acceptable with this low of traffic.

The general design is covered in the diagram.

Detailed component design

This scales well, easily to 10x-100x users, if we don't add the ability to share activities.

Sharing activities invites a host of new problems: handling hot users, significantly increase read traffic, handling likes and comments, generating feeds for users etc.

Trade offs/Tech choices

Postgres and SQLite are obvious choices given our requirements for relational queries and low traffic.

Failure scenarios/bottlenecks

If a DB dies, we will have backups but it'll take some time to bring one back online. For that scenario, assuming we can reasonably afford it, we'll run an active-passive configuration in each region so that we don't have to failover traffic to a replica in another region except in the case of a region-level failure.

To bring up DBs replicas faster, we'll have regular backups (every hour or so) and keep a write ahead log of each transaction, so that we can just restore the backup and then replay the log in order to get the new DB ready.

Future improvements

User sharing as discussed above. Adding photo capabilities, third party integrations, as well as advanced analytics and the abilities for users to export/delete data.