Loading...
Show insights and analysis of detailed historical activity logs
Track fitness goals and progress
Receive data from wearables and GPS
Should be able to process hundreds of events per minute produced by devices
Should have good consistency for events and progress
Should be able to work offline or during connectivity issues
1M DAU
Which evenly distributed along the day with 30% during peak hours producing at most 100 events per second are:
1M*.3*100 = 3*10^7
Which batched per minute are be 3/60*10^7 = 5*10^5
get_analytics()
Input: user id, time period : { start, end }
Output: a series of data analytics
push_data()
Input: all Events[] with timestamp, type during a minute
Goals - CRUD endpoints. per user always active with a type and a target per time period as well as progress
Events
Id
userId
timestamp
value
Goals
Id
userId
startTimestamp
target: number
period: 'Month' | 'week' |'day'
type
Goal_Progress_Step ( one of these is created after a session of events to mark progress for a goal in a particular period)
Id
amount: number
Timestamp
PeriodNumber - (an easy way to group progress per periodNumberd for easy consultation and not needing to compute from timestamps)
We have a mobile app with a local database which stores high frequency events from devices like a fit bit and the phone itself. These events are batched to a service that inserts them into a time series database, and creates goal_progress records. An analysis service runs when the training session is over and updates analytics for time periods including this data. These analytics are stored in a documentDB to allow for values constituting of complex values without schema updates. With a pub sub service we push analytics into the device which keeps them in a local cache for offline usage.
The user creates a goal for a time period and an activity by calling the API from his mobile device app.
The user starts an activity which starts logging events into their mobile device. This device starts batching them into the time series writing service.
When the time for analyzing the time period is over, goal_progress_steps are created and related analysis are updated, an analysis for the day is created. This data is stored in a document DB and all the new data is sent to the device.
If the user requests analytics, this data is served from a service fetching then from the documentDB if not present in their local device.
The document DB analysis database is shared by userID and a time period giving predictable database storage related to time. It also has read replicas to offload reads and allow writes to be scaled based on the analysis service.
The analytics service which is triggered after a training session runs is kept alive by a sentinel which makes sure that the process completes or reports failures if it fails after some retries.
The time series database deletes batches of old data particularly those that have already been processed for analytics and are very old.
The time series database gives us great time related queries and aggregation on time periods but sacrifices other querying capabilities offloading this to the analysis service. It also gives us fast insertion.
The mobile device batching increases mobile storage while lowering rps on the time series write service. It also adds reliability by storing data and making sure it was transmitted.
The document DB allows us for acid processing of complex objects which are the analysis per userID per time period avoiding complex joins and giving us good read performance. It sacrifices though partial concurrent updates and reads and relations which we are not currently using.
Triggering the analytics service at the end of the training session and updating analytics incrementally. Allows for almost instant and performative analysis, but limits the kinds of analysis that can be executed, specially those that might not be incremental which would deter the performance a lot or suffer a lot of re-processing.
If the time series write service fails, devices can store trainings for further transfers.
If the analysis service fails, old analysis can still be loaded.
If the documentDB fails, users can view their local copies of analysis
If users are offline, training data is saved and uploaded at a later time.
Add a secondary cron job service for analysis types that are not incremental.