Loading...
TrackUserActions (clicks, page visited)
ReadActions
Availability --> as a customer, I want the dashboards system to be up at all times, plus, the tracking of actions to not suffer from downtime
Scalability --> I want to be able to ingest as many user actions as possible
Latency --> Mainly for the when we are reading data to fill the dashboard. <1 sec is good, preferably <200ms
Given a platform that targets stores growing organically:
Lets say we have: 10000 customers
For each customer: 1000 store visitors
Amount of writes to the system: 10 actions per store visit per customer
Total Writes = 100.000.000 actions per day
Each action is 1KB
Storage needed = 100GB a day * 2 years = 73TB
Lets round it to 100TB (with some buffer for growth) of storage that is needed for 2 years.
logAction:
Input --> actionId, actionType, timestamp
Output --> 201 created
readAllActions:
Input --> userId, actionType
Output --> list of all actions by type
User: id, storeUrl
Action entity: id, name, type, timestamp, owningUser
Relationship: a user owns none, one or many actions. An action is owned by a user.
Access Pattern: given a User --> retrieve all Actions of given type
I would say since the system is write heavy, choosing a NoSQL DB to store data would be the right choice.
NoSQL DB would also give us the flexibility of adding new action types that might have different structures compared to existing ones.
NoSQL DB fits the NFR well since they can be horizontally scaled and we don't need strong consistency of the data.
I would pick Cassandra over MongoDB because Cassandra is optimized for write heavy workloads, all nodes are able to make write operations and we avoid having to create a sharded datastore if we used MongoDB given that only the master node can write.
As for reads, we would be better off if we use a database system that would allow for better querying capabilities. An Analytical warehouse with SQL support makes sense here.
So, the actions would be saved initially into Cassandra and we can have some sort of data management/transformation tool that asynchronously reads that data and inserts it into our warehouse. Given that the application would need to run complex JOIN logic to be able to fullfill the asks of our different use cases (existing and future ones) data can be precomputed/materialized to whatever the expected shapes are. In this way, we would (initially at least) avoid any sort of caching needs.
You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...
logActions flow:
readAllActions flow:
DataTransferService: this service is a third party tool that is scheduled to perform async polls to read data from Cassandra into our Data Warehouse. There is no need for any transformation at this step, just data basic management of removing unnecessary columns for example if there is any. By giving away such simple task to a third party service we can trade in a bit of extra cost for less complexity, an alternative would creating a service that reads data as it comes and queues it into a message broker for next steps to come.
DataTransformationService: I am gonna be using DBT here (Data Build Tool) which is an open source data transformation tool existing on the market for this kind of workloads. The way to setup this is to create a separate repository that hosts the query that will consume whatever is in the warehouse and perform the transformations that result in tables with precomputed data in our SQLDB. Saving the data precomputed allows the ApplicationService that serves the frontend to access information without running in real time complex queries, improving performance for reads.
Explain any trade offs you have made and why you made certain tech choices...
Try to discuss as many failure scenarios/bottlenecks as possible.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?