Design Twitter

A user opens Twitter and sees a personalized feed in under 500ms. Behind that instant response is a system that pre-computes feeds for 500M users, fans out each tweet to millions of followers, and handles celebrities whose single post triggers writes to millions of caches. The core challenge is not storing 280-character messages. It is delivering the right tweets to the right users fast enough that the feed feels instant.

Functional

• Post tweets up to 280 characters of text.
• Follow and unfollow other users.
• Like tweets.
• Home feed showing a personalized, ranked stream of tweets from followed users.
• Hashtag search to view tweets containing a specific hashtag.

Non-Functional

• Read-heavy workload. Each user tweets about twice a day but reads roughly 100 tweets. With 500M DAU, this produces a read-to-write ratio near 50 to 1, which means the system must be optimized for reads above all else.
• Low latency. The home feed must load within 500ms. Users who wait longer leave for competitors.
• High availability. Target 99.99 percent uptime. A feed that is occasionally stale is acceptable; a feed that is down is not.
• Eventual consistency. A tweet appearing in a follower's feed a few seconds after posting is acceptable. Strong consistency per keystroke is unnecessary and would bottleneck the write path.
• Scale. 500M daily active users, 1B tweets per day, 50B feed reads per day.

Out of Scope

Direct messages, image and video upload pipelines, advertising, content moderation algorithms, and trending topics.

The numbers below reveal why this system needs pre-computed feeds instead of on-the-fly computation. Pay attention to the read-to-write ratio, which justifies every caching decision in later sections.

Users and Activity

• 500M DAU.
• Each user tweets an average of 2 times per day, producing 1B tweets per day.
• Each user views roughly 100 tweets per day, producing 50B feed read operations per day.
• At peak, 20 percent of DAU (100M users) interact with the system concurrently.

Throughput

• Write throughput. 1B tweets per day divided by 86,400 seconds equals roughly 12,000 tweets per second. At peak (3x average), about 36,000 tweets per second.
• Read throughput. 50B feed reads per day equals roughly 580,000 reads per second. At peak, about 1.7M reads per second.
• Fan-out writes. Each tweet fans out to an average of 200 followers. 1B tweets multiplied by 200 fan-out writes equals 200B cache writes per day, roughly 2.3M cache writes per second.

Storage

• Tweet size. Each tweet record is approximately 512 bytes (ID, author ID, timestamp, content, media link, like count, hashtags).
• Daily storage. 1B tweets per day multiplied by 512 bytes equals 512GB per day.
• Two-year projection. 512GB per day multiplied by 730 days equals roughly 365TB. With growth buffer, plan for 500TB.
• Feed cache. 500M DAU multiplied by 20 tweets per feed multiplied by 512 bytes equals 5.12TB. At 128GB per Redis server, this requires roughly 40 cache servers.

Why These Numbers Matter

The 580K reads per second versus 12K writes per second (a 48-to-1 ratio) is the single most important number. Computing a ranked feed on every read would require running the scoring algorithm 580K times per second. Pre-computing feeds on write reduces this to 12K scoring operations per second, each fanning out to an average of 200 cache entries. The math makes the architectural choice obvious.

Four endpoints define the system. The most important design decision here is pagination. The home feed is a continuously updating stream, and offset-based pagination breaks when new tweets arrive between page requests. Cursor-based pagination solves this by anchoring each page to a specific tweet.

Core Endpoints

• POST /tweets with body containing the content. Creates a tweet. Returns the new tweet ID. Server-side parsing extracts hashtags and user mentions from the content.
• POST /users/{user_id}/follow with body containing the target user ID. Creates a follow relationship. Idempotent: following a user you already follow returns success without duplicate entries.
• POST /tweets/{tweet_id}/like. Records a like. Increments the like counter (via sharded counter for popular tweets).
• GET /users/{user_id}/feed?cursor={last_tweet_id}&limit=20. Returns a JSON object containing an array of tweets and a next_cursor value. The cursor is the ID of the last tweet returned, so the next request picks up exactly where the previous one left off.

Why Cursor-Based Pagination

Offset-based pagination (page 1, page 2, page 3) breaks in a feed that updates continuously. If 5 new tweets arrive between the user requesting page 1 and page 2, offset-based pagination skips or duplicates tweets. Cursor-based pagination anchors to a specific tweet ID, so the next page always starts after the last seen tweet regardless of new arrivals.

Error Handling

Standard HTTP codes. 429 (Too Many Requests) is especially important for the feed endpoint because it is the most expensive operation. When a client receives 429, it should back off for a few seconds before retrying. 404 for deleted tweets, 403 for private accounts.

Three data stores, each chosen for a different access pattern. Tweets go to a document database for flexible schema and horizontal scaling. Feeds live in Redis for sub-millisecond reads. The follows graph uses a separate collection to support efficient bidirectional queries.

Tweet Collection (MongoDB)

• tweet_id (primary key, auto-generated).
• author_id (indexed for author lookups).
• timestamp (indexed for chronological queries).
• content (up to 280 characters).
• media_url (link to image or video in object storage).
• like_count (periodically flushed from sharded counter).
• hashtags (array of hashtag strings extracted at write time).

User Collection

• user_id (primary key).
• email (unique index).
• name, display_name.
• follower_count, following_count (maintained via async counter updates).

Follows Collection

• follower_id + followed_id (compound index).
• timestamp.

Separating follows from the User document enables efficient bidirectional queries. Given a user, list everyone they follow (query by follower_id). Given a user, list all their followers (query by followed_id). Embedding this in the User document would make one direction fast and the other a full collection scan.

Hashtag Collection

• hashtag string (primary key).
• tweet_ids (array of tweet IDs containing this hashtag).

This denormalized design makes hashtag lookups fast: one read returns all tweet IDs. The trade-off is that deleting a tweet requires removing it from both the Tweet collection and the Hashtag document's array.

Sharded Counter for Likes

When a celebrity's tweet receives millions of likes, a single like_count field becomes a hot spot. The solution is a sharded counter with 100 sub-counters stored in Redis, each handling 1/100th of the write traffic. A Counter Service randomly assigns each like to a sub-counter. Every 5 minutes, the service sums all sub-counters and flushes the total to the tweet's like_count field in MongoDB.

Two paths define this architecture. The write path stores a tweet and fans it out to followers' feeds through a message queue. The read path retrieves a pre-computed feed from cache in under 10 milliseconds. Separating these paths lets you optimize each independently: writes go through async message queues while reads hit pre-computed caches.

Write Path

The client sends a tweet to the API Gateway. The gateway authenticates the request, applies rate limiting, and forwards it to the Tweet Service. The Tweet Service stores the tweet in MongoDB and publishes an event to the Message Queue. Three async consumers process the event independently: the Tweet Analyzer extracts hashtags and user mentions, the Content Analyzer checks for policy violations, and the Home Feed Generator computes a score and pushes the tweet into every follower's Redis Sorted Set.

Read Path

The client requests its home feed through the API Gateway. The Home Feed Service checks the user's Redis cache first. On a cache hit (the common case), it returns the pre-computed top-K tweets in under 10 milliseconds. On a cache miss, it falls back to MongoDB, queries for followed users' recent tweets, computes scores, populates the cache, and returns the results in 100 to 500 milliseconds.

CDN

Static content (profile images, uploaded media, video thumbnails) is served through a CDN. These assets are written once and read millions of times, making them ideal CDN candidates.

Two flows define this system. The write flow shows how a single tweet reaches millions of feeds through fan-out. The read flow shows how pre-computation turns a complex ranking problem into a simple cache lookup. Together, they demonstrate why separating write and read paths makes this system tractable at scale.

Write Flow (tweet)

1. The client sends POST /tweets with the tweet content to the API Gateway.
2. The API Gateway authenticates the request, applies rate limiting, and forwards to the Tweet Service.
3. The Tweet Service validates the content (280-character limit), generates a tweet ID, and stores the tweet in MongoDB.
4. The Tweet Service publishes a "new tweet" event to the Message Queue.
5. The Tweet Analyzer consumes the event, extracts hashtags and user mentions, and updates the Hashtag collection.
6. The Content Analyzer consumes the event and checks for policy violations.
7. The Home Feed Generator consumes the event, looks up the author's follower list, computes a score, and inserts the tweet into each follower's Redis Sorted Set.

Read Flow (home_feed)

1. The client sends GET /users/{user_id}/feed to the API Gateway.
2. The Home Feed Service receives the request and checks the user's Redis Sorted Set.
3. Cache hit path (over 99 percent of requests). The service retrieves the top-K tweets by score from the Sorted Set, resolves tweet content from a tweet cache or MongoDB, and returns the response in under 10 milliseconds.
4. Cache miss path. The service queries MongoDB for the user's followed accounts, fetches their recent tweets, computes scores, inserts results into the Redis Sorted Set (backfilling the cache), and returns results in 100 to 500 milliseconds.

The question "how do you serve a personalized feed to 500M users within 500ms?" is the question an interviewer is really asking when they say "design Twitter." The fan-out strategy and the ranking algorithm are what make this problem unique. Every other component (the database, the API, the message queue) exists to support this core mechanism.

Fan-Out on Write

When a user tweets, the Home Feed Generator looks up their follower list and inserts the tweet (with its computed score) into every follower's Redis Sorted Set. For an average user with 200 followers, this means 200 cache writes per tweet. Across 1B tweets per day, this produces 200B fan-out writes per day.

This works because the writes are asynchronous and distributed across many Redis servers. The tweeting user does not wait for fan-out to complete. The Message Queue absorbs traffic bursts. And 200 writes per tweet is a small, predictable cost.

Scoring Formula

The ranking score determines which tweets appear in a user's top-K feed. A simple weighted formula balances multiple engagement signals: Score = W_like * likes + W_followers * author_followers + W_retweet * retweets. Time decay ensures older tweets gradually lose ranking position. The weights are tunable: the product team can prioritize engagement (higher W_like) or authority (higher W_followers) without changing the architecture.

Redis Sorted Set for Top-K Storage

Each user's feed is stored as a Redis Sorted Set with the key user:{id}:feed. Tweet IDs are the members, ranking scores are the scores. Retrieving the top-K tweets is a single ZREVRANGEBYSCORE call that returns the highest-scoring tweets without modifying the data structure.

This is fundamentally better than a heap for this workload. A heap requires popping K items to read the top-K, destroying the structure each time. With feeds read far more often than updated (50 reads per write), a non-destructive read operation is essential.

Pre-Generation Economics

Storing 20 tweets per feed for 500M DAU requires 5.12TB (500M users multiplied by 20 tweets multiplied by 512 bytes). At 128GB per Redis server, roughly 40 servers cover the entire feed cache. This is a modest infrastructure cost compared to the alternative of computing rankings on every read at 580K requests per second.

Two sharding decisions cascade through the entire system. The tweet database and the feed cache use different shard keys, each optimized for its dominant access pattern. Getting either one wrong creates hot spots that undermine the entire architecture.

Tweet Database Sharding (by Tweet ID)

Sharding the tweet database by tweet ID distributes writes evenly regardless of author popularity. A celebrity and an average user both generate tweets with randomly distributed IDs, so no single shard becomes a hot spot.

The trade-off is that querying "all tweets by user X" requires scatter-gather across all shards. But this is acceptable because hashtag lookups (the more common query pattern) are fast because the Hashtag document stores tweet IDs directly, and the system can route each ID to its specific shard.

Feed Cache Sharding (by User ID)

Sharding the Redis feed cache by user ID means every feed read hits exactly one shard. The Home Feed Service hashes the user ID, finds the shard, and retrieves the Sorted Set. Each user's feed is roughly the same size (20 tweets), so the load distributes evenly.

The trade-off is that the Home Feed Generator must write to many shards during fan-out. A tweet from a user with 1,000 followers requires writes to potentially hundreds of different cache shards. But writes are async and batched, making this manageable.

Pre-Generation vs On-The-Fly

Pre-generation costs 5.12TB of Redis storage and 200B cache writes per day. On-the-fly computation eliminates storage but requires 580K ranking operations per second. At a 48-to-1 read-to-write ratio, pre-generation is the clear winner because it shifts computation from the frequent read path to the infrequent write path.

A celebrity with 10M followers tweets. The Home Feed Generator must update 10M Redis Sorted Sets, a burst of 10M cache writes from a single event. This is the celebrity fan-out problem, and it traces directly to the pre-generation strategy from earlier sections. Every failure scenario in this system connects back to a core architectural decision.

Celebrity Fan-Out

The Home Feed Generator pushes each tweet to every follower's cache. For a celebrity with 10M followers, a single tweet triggers 10M cache writes, overwhelming the Redis cluster and causing a backlog in the Message Queue that delays all other fan-out processing.

The solution is a hybrid approach. Set a threshold (users with over 1,000 followers) above which tweets are not pre-generated into follower feeds. At read time, the Home Feed Service pulls from a dedicated celebrity tweet list and merges it with the pre-computed regular feed. The celebrity list is refreshed every 5 minutes to reduce load.

Redis Cluster Failure

If the Redis cluster goes down, all feed reads fall back to MongoDB. Latency jumps from under 10ms to 100-500ms per request. At 580K reads per second, MongoDB cannot sustain this load. Mitigation includes Redis replication (primary plus replica with automatic failover), partitioning across multiple independent Redis clusters so a single cluster failure only affects a fraction of users, and circuit breakers that serve degraded feeds (recent tweets without scoring) rather than timing out.

Message Queue Backlog

If the Home Feed Generator consumers cannot keep up with incoming tweets, the Message Queue backs up. Feed caches become stale: users see old tweets and miss recent ones. Mitigation includes horizontal scaling of consumer instances, monitoring queue depth with alerts on lag exceeding 30 seconds, and a priority lane for high-engagement tweets so that viral content is processed first even during backlogs.

Hot Hashtag

A viral event causes millions of users to search the same hashtag simultaneously. The Hashtag document's tweet_ids array becomes a hot read. Mitigation includes caching popular hashtag results in Redis with a short TTL (30-60 seconds) and serving the cached result for all concurrent requests.

Four extensions beyond the core design address real-time delivery, smarter ranking, global distribution, and content safety. Each builds on the existing architecture without requiring fundamental redesign.

Real-Time Push via WebSocket

Currently, users see new tweets when they refresh their feed or the client polls. Adding a WebSocket connection for active users enables instant push when a followed user tweets. The Home Feed Generator, after inserting into the Redis Sorted Set, also pushes a lightweight notification through the WebSocket channel. This reduces perceived latency from "next poll interval" to "instant."

ML-Based Ranking

The static weighted formula (likes, followers, retweets) treats all users identically. A machine learning model trained on individual engagement history can personalize ranking per user. Features include time of day, content type preference, social graph proximity, and historical click-through rate. The model replaces the static weights in the scoring step without changing the Sorted Set storage or fan-out architecture.

Multi-Region Feed Cache

Replicating the Redis feed cache to regional data centers reduces read latency for users far from the primary region. The write path stays centralized (or uses conflict-free fan-out), while each region serves reads from its local cache. This is especially impactful for a global user base where cross-continent latency adds 100-200ms.

Automated Content Moderation

The Content Analyzer currently checks for policy violations asynchronously. An ML-powered classifier can detect hate speech, misinformation, and spam in near real-time, removing violating tweets from feeds before users see them. This is a proactive approach (pre-delivery filtering) rather than reactive (post-report takedown), significantly reducing user exposure to harmful content.