Requirements

Functional Requirements:

• Creating own tweets
• View others tweets on mainline
• Follow/unfollow tweet
• Appreciate other updates

Non-Functional Requirements:

• high-availability
• System must scale horizontally as users and content grow
• low-latency on read path - to ensure well user experiance

Capacity Estimation

• Assume 1000000000 registered users.
• The 20% of them is actively reading. - 200 000 000 DAU
• 500 000 000 new tweets per day - 6k/s up to 3x in spike - 18k/s
• 300 followings on average
• 500B on tweet

500M tweets/day * 500B on tweet = 250GB/day = 90TB/year = 3MB/s

The each tweet is copied to the followers timeline, so there are 3MB/s*300 - 900MB/s.

Also it's 500M * 300 = 150B timeline insterts/day and 1.7M inserts.

Timeline reads: ~1M/s (each DAU reads ~5 timelines/day → 1B reads/day) Likes: ~50k/s · Follows: ~20k/s

There is also graph of followers - 300 * 1B = 300B edges. About 50B per edge so it sums to 15TB

Likes 1B/day.

API Design

• POST /api/tweets
  • create a new Tweet owned by logged User
  • body: {"text": <string>, "imageUrl": <imageUrl>}
• GET /api/timeline?cursor=<cursorId>&limit
  • retrieves timeline for logged User
  • <cursorId> is stored in session databasewith TTL (24h)
• POST /api/followed
  • Subscribes current User to followed User
  • Body: {"followedId":<UserId>}
• DELETE /api/followed/<followedId>
  • Unsubscribes to followed User
• POST /api/tweets/<tweetId>/likes
  • Appreciate Tweet
  • Body: {"followedId":<UserId>}
• DELETE /api/tweets/<tweetId>/likes
  • Removes Tweet appreciation

High-Level Design

User loads app by reaching Frontend cached by CDN.

Using loaded app, User sends request to fetch timeline.

User is unauthenticated due to logout.

When user passes authentication, the timeline is fetched.

General requests schema:

Frontend sends request to LoadBalancer/Reverse Proxy. It's responsible for ssl-termination, load-balancing Then request is routed to Api-gateway is an entrypoint does rate-limiting, authorization (if neccessary reaching Auth Service to process authentication), and routing requests to services.

Timeline:

• Api-gateway routes request to Timeline Service. the whole timeline for logged user is read from database. To make timeline fast timeline must be preprocessed and ready, when user accesses it.
• Timeline is divided to fragments, to make timeline chunks small enough to be fetched and not bandwith consuming.
• Likes are fetched for all fetched tweets by quering Like Service with all ids.

Tweet creation:

• Api-gateway routes request to Tweet Creation Service. The image attached to Tweet should be preprocessed (e.g. minified) before publication, so it's placed on preprocessing queue. After processing, the Tweet must be propagated to followers' timelines, so for each timeline it is asynchroniously (via queue) putted into database.

Following:

• Api-gateway routes request to Follower Service. Then it reaches database and stores information about newly followed person and who follows this person.

Unfollowing:

• Api-gateway routes request to Follower Service. Then it reaches database with relations between users and removes information about followed person. Optionally it should also schedule process of removal followed person for given timeline.

Likes/Unlikes:

• Api-gateway routes request to Likes Service. Then it reaches Likes database and update infromation current user's Like.

Database Design

Main Database - Mysql (replicated):

• timeline_entries table
  • id BIGINT PK
  • publisher_id BIGINT FK
  • publication_time TIMESTAMPTZ
  • timeline_owner_id BIGINT FK
  • text VARCHAR(150)
  • image_id BIGINT FK
• tweets:
  • id BIGINT PK
  • publisher_id BIGINT FK
  • publication_time TIMESTAMPTZ
  • text VARCHAR(150)
  • image_id BIGINT FK
• users:
  • id BIGINT PK
  • name VARCHAR(100)
  • profile_id BIGINT FK referencing image
• followers
  • follower_id FK
  • followee_id FK
  • primary key on (follower_id, followee_id)
• likes:
  • tweet_id
  • liker_id
  • primary key on (tweet_id, liker_id)
• images:
  • id BIGINT PK

Redis as a timeline cache (sharding)

• timeline_entries - (key - timeline_owner_id) sorted_set (sorted by creation_time)
  • JSON with:
  • publisher_id
  • publication_time
  • text
  • image_id

RabbitMQ as message broker (sharded):

• image_processing
• timeline_inserter

Detailed Component Design

Common path:

• client (using previously described CDN, frontend) sends request to our service. The SSL termination and routing to one of multiple stateless Api-Gateways instances.
• Api-gateway performs authentication, authorization and rate-limiting. The rate-limiting data is hold on shared Redis instance (or Cluster for better HA)
  • Authentication uses JWT tokens as auth-holder, so it do not requires Database
  • The manually expired (in case of logout) JWTs are hold in Redis with TTL calculated to expire token expiration date
• The Api-gateway routes request to one of selected service.
  • Every service has multiple instances, so to route the request to services fairly, we route to internal proxy (e.g. Haproxy) that balances the load and performs service-discovery (maintain alive services list, performs healthchecks)

Tweet Creation path:

• Api-gateway routes the request to one of Tweet Creation service instances.
  • if Tweet contains image, the new ID (UUID) is saved to Database
  • The image processing is scheduled (async)
  • The Tweet is stored in Database and retrieves also the UUID assigned to this tweet
  • The Tweet processing is schedule (async)
• The response with stored values is returned to current user. However, for now, only user can view newly created Tweet
• Image processing:
  • There are multiple workers that can handle and can be scaled up/down basing on pending messages metric
  • A worker performs resize its and compress image in multiple versions (to adjust to various screens)
  • After the worker does it's job it updates image table in database
• Tweet processing:
  • Tweet needs preprocessed image, so it awaits for image processing to finish using Enrich Pattern (may reach Database Replica for it)
  • To make tweet inserted only once, we have unique key on (follower_id, tweet_id)
  • Now the processor performs batch insert to all followers timelines (using INSERT INTO select followers SQL query)
  • Query for subscribed followers. Information about tweet is also propagated to Redis, so we need to add it also into followers timeline sorted_set.

Timeline retrieval:

• Api-gateway routes the request to one of Timeline Services.
• If timeline has the opaque_id, we need to access Redis to deserialize it to offset_id.
• The timeline service reads from Redis another available entries
• if there are no more entries in Redis, we query Database for elder entries.

The TTL for Tweets on timeline is 7 days from creation.

Edge cases:

• for very famous creators (e.g. celebrities) the other path is required - pull-based.