Loading...
Posting tweets
Open a user page and see their posts.
Possibility to react and answer to a post
Subscription to an account.
Home feed.
Low latency
Availability
Scalability
20 million daily active users = 100 million tweets/day
200 million daily passive users = read-only
1 billion total users
A globally read-heavy structure mainly.
The user id is always passed implicitly as an argument.
GET /users/id/posts ?limit=10&cursor=...
-> in reverse chronological order
GET /feed ?limit=10&cursor=...
POST /tweets {"content":"..."; "parent_id":...}
DELETE /tweets/id
POST /tweets/id/like
DELETE /tweets/id/like
POST /users/user_id/follow
DELETE /users/user_id/follow
GET /subscriptions
cf diagram.
SQL
User table
user_id; username; creation_date; nb_followers; nb_posts
Posts table
post_id; author (~user_id); content; nb_likes; parent_post; enclosed
Subscription
account; follower
(there can be several times the same value in each column, but no duplicate rows)
Storing post parent-child relationship as tree in NoSQL
If files enclosed to the post, a link in posts.enclosed column refers to the actual file stored in an AWS S3 bucket.
I don't really see how to store likes per post in a way that's efficient to retrieve
Client connects to server via HTTPS. We use a content delivery network (CDN) for faster access and reduce overload by bringing servers closer geographically, and a TLS terminator for further security in case the encryption breaks.
Then the request goes through an API load balancer: depending on the request of the user, we want to split the load between the different services.
We have three main services: tweet service (handling the tweets themselves and their content, the media being handled through URLs to an Amazon S3 bucket), the user service handling any request regarding users (updating profile, following someone, etc), and the engagement service (handling likes, etc).
When a new tweet is posted, the user calls the API endpoint POST /tweets.
The tweet database is updated with a new line including author id and, if relevant, the parent post. Some metadata (e.g, time of posting) is added.
If there is a parent post, it is added in a Graph DB (parent -> children) for quick retrieval upon reading.
Since there is only a limited amount of writes compared to an insane amount of reads, we want the feed to load quick, and thus to use a fan-out-on-write structure. When a user posts a tweet, the timeline of the users following him is updated progressively, using a fan-out queue handled, for example, by Kafka. Except for celebrities with millions of followers. In their case only, their new posts are fetched upon each user opening their app.
When a user subscribes to someone, a row is added to the follow graph database, including the id of the user the person is now following, and the id of the person himself.
When someone opens the app, the feed being loaded from the feed database, we have quick reads. To make them even faster, we could use a Redis cache, at the cost of delayed updates.