Loading...
Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...
read paths:
write paths:
Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.
client - mobile phone aggregates data in memory in our specified format then each minute it push the data to our health app.
server - gets the data from the client. Store it - generates a normalized values for whole day.
each minute is saved as separate row record then we have daily-data which is normalized data for the whole day.
data aggregation pipeline - has an endpoint /push-data where a client sends the data from trainings and the data aggregation pipeline. Imediatelly saves the data to database for the minute records. And then we have a hourly and daily aggregations. as the hourly data are calculated based on the input data on the go. we always process the minute data imediatelly and then create a job into queue to aggregate the hourly and daily metrics. This means if we have calories burned the app will imediatelly have the data and then a separate worker that accepts messages from queue will recalculate hourly data and daily data.
WE HANDLE DUPLICATION the way that we set the ZULU time and if we already have the data for a specific time record we just upsert the newly received data. its called LAST-WRITE-WINS approach.
offline sync - client keeps records in ZULU time per each minute and then sync it after its online. It has a local storage that will send it to the POST /push-data after its online.
database - includes a one minute point of data and then hourly and daily normalized data.
partitioning by time. each month / day / year of data based on users = if we have too much users we will need more granular time sharding.
queue worker - gets jobs to aggregate data into hourly/daily sync. We can always create a timeout to when pickup the message to sync the daily data/hourly data. So we dont sync every minute for the hourly/daily data but most likely only at the end of the day at the end of the hour.
caching - layer to cache frequently accessed data = eviction policy could be least frequently accessed. TTL for daily data could be whole day.
Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.
Queue - this queue will create a messages that a queue worker can take and work on. I think a good idea would be some time-outs to sync data so we dont sync hourly and daily aggregations when a user has activity ongoing it will take too much compute power. so basically once a hour ends = sync data it will never change. The same thing will happen with daily data dont sync on each data push. investigate if we dont have any active activity and sync at the end of the day or after activity is finished. we can have this status on activity record and once its states as finished we can create a sync job message and push this to the queue.
database - we can use noSQL for high throughput and low latency some dynamoDB to store the one minute records - it will keep the data working on scale because dynamo handles geo-replication and the partitioning on scale. and for reading the daily aggregation and hourly aggregation we can use a sql postgres with citrus to keep the data and timeseries with postgres data that will get synced data from workers.
data aggregation pipeline - we can have a pipeline that will sync the data each second or 5 second or minute based on the granularity we need. if its each second I would preffer streaming the data, if its more than 5seconds I think its enough to sync the data with a post requests. Each time data aggregation pipeline will accept data it will populate this data to the noSQL database and later we can create a job after its saved to nosql database to create a message it will use the automatic sliding window approach to not sync every change but only once we have more info/more data to do it in bulk. The bulk or normal upload is handled by separate workers the workers will be spawned based on messages in the queue.
caching - layer to cache frequently accessed data = eviction policy could be least frequently accessed. TTL for daily data could be whole day. Or we can have a partitioning on tables and dont care about the caching because the SQL is made to be read-heavy and efficient on retrieval data.