Loading...
We assuming DAU or 20,000, with each user logging activities once a day on average.
Accounting for spikes in usage, for capacity planning, lets assume all updates show up in the same hour. We have about 10 updates per second, which is well within the capabilities of nearly any NoSQL or SQL database.
We should also size what is needed for the social sharing features. Assuming 10 activities per second being shared with 10 friend each
PUT /activity/:activity-id -> Shareable-URL
{
type (e.g. "running")
duration
distance
}
POST /weekly-goal/:goal-id
{
type (e.g. "running")
duration
distance
start_date
end_date
}
GET /activities?start=:starttime&end=:endtime -> Activity[]
Activity {
type
duration
distance
upload-date
}
GET /weekly-goals?start=:starttime&end=:endtime -> Goal[]
Goal {
type (e.g. "running")
duration
distance
start_date
end_date
}
User
Goal
Activity
Use Postgres for storage
Client connects to her service via an API gateway and load balancer that are responsible for authentication and rate limiting.
FrontEnd terminates the connections from client and serves the API. It stores User, Activity and Goal data in Postgres.
In enques content generation requests to the Shared Page Generator.
The Shared page generator generates content that can be shared, and uploads it to blob storage, and puts the URL in Postgres.
Once the client has shared the URLs with their friends, their friends can access the content via the blob storage directly
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
Given the scale of the system, the storage can be handled by post progress both for the entities such as users and for goals and activities. The content itself, though should be in blob storage where it can be access directly by friends of our users without hitting our postgres storage.
For decoupling the front end from the workers that generate content we're using redis streams as a queuing mechanism. It's possible to put a cash in front of our postgres storage both for the front end and for the shared page generators to do reads, but it doesn't seem it's necessary here due to the scale and due to the usage pattern
The first thing to talk about is the communication between the clients and the backend in this approach. Most of the functionality is actually in the client. We are only using the backend for the social sharing features and as a backup in case the data on the client is lost as such since the client powers most of the features if there is any loss in communication loss and connectivity we just rely on the client to retry pushing the data when the connection is reestablished. The client also makes a decision of when it has lost data and it must get the data from the backend.
The front ends are fairly stateless. They terminate the connections of the clients. If the front end fail, we can just move the clients to another front end. We'll have multiple instances of the front end.
As far as red stream goes, we can use master slave replication with lettuce in case the master fails. The stream is being backed up to the slave, and the slave can be promoted. The scale of data doesn't seem to suggest that we would need sharding
The shared page generators are based on pulling requests from radius. They should only acknowledge the requests after the processing is complete in case of any failures. There are operations on reduce streams for another consumer to claim those requests.
Post supports replication as well with strong consistency and durability. The scale of the data doesn't seem to suggest we would need sharding here.
The shed page generator is uploading generated content to blob storage. We are taking on a dependency on this blob storage service, but this service itself should be providing for its own high availability.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?