Loading...
MVP
design an Instagram like system
Users can upload photos/images
Users can follow other users
In the homepage, the app will display a timeline consists of photos/images posted by the subscribed users
Non MVP
comments, likes, stories
Scalable - be able to handle large traffic
Available - Should be accessible anytime/anywhere
Performance - the latency should be low
Durability - photos/images should be persisted once get uploaded
400 million DAU
100 million photos uploads daily
Upload QPS
100 0 00 000 / 100 000 = 1000 QPS
peak QPS * 3 => 3000 QPS
Read QPS
assume each user views 5 posts per day
2000 million views/day
2 0000 00 000 / 100 000 = 20000 qps
peak QPS * 3 => 60k qps
Storage, assume 500k / photo
100 000 000 * 500k = 50 TB /day
replica factor * 3 (replications, different resolutions)
150 TB /day
*top account 100 million followers
*4 billion likes everyday
POST /$user_id/media
add image's data/post, metadata like title, to the request body
returns: $post_id
GET /$user_id/timeline?from=
returns: a list of posts
users table (NoSQL)
user_id (primary/partition key)
DOB
creation_timestamp
last_active_timestamp
subscription_table (NoSQL)
user_id (partition key)
following_user_id (clustering key)
since: timestamp
primary_key: (user_id, following_user_id)
But this users_follow_table design will make it harder to look up the followers for a given user
we have to create another table for user_id to followers, which would increase complexity
or we can create secondary index, but that may impact the table performance
Another option is to use other databases, like SQL or graphDB where relationships are a first-class citizen
Post_table (NoSQL)
post_id (primary key)
author_id
creation_timestamp
file_url
Upload path
Users upload images and create posts through the post service
post service will upload the image to a storage service like s3, write an entry to post table, and insert an event into a queue, then returns 200 and post id to the user
We can have image process service that further processes the uploaded image like compression, generate copies with different resolutions, etc.
The reason to have a queue here is to separate the process of creating post and fan-out processes
The consumers of the queue service will read the subscription table, find all subscribers and fan out the post to each subscriber's timeline cache
We will only maintain the timeline cache for recent active users
The fanout workflow should only be triggered for general users. For celebrities, we will use pull mode instead
Redis cache can handle millions of keys per instance, and the values for each key is not large in our case (just post_id and timestamp), so we should be able to fit the timeline info in the cache
We can further improve the read latency by distributing images with CDNs
Read Path
users open the app, which triggers a call to timeline service, the timeline service will fetch the precomputed timeline from the timeline cache (user_id as key, members are the list of posts), as well as checking the posts from subscribed celebrities, then present the timeline to the user
Most components in our service can be horizontally scaled
Timeline cache is stored in redis and DB, so in case of cache crash, we can recreate the timeline from the DB
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
Explain any trade offs you have made and why you made certain tech choices...
Try to discuss as many failure scenarios/bottlenecks as possible.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?