Requirements

Functional Requirements:

• Users can find tweets by text, time range, and/or author.
• Users can create & read tweets.
• Users can like tweets.

Non-Functional Requirements:

• The p99 latency for finding tweets by text, time range, and/or author should be less than 100ms.
• The p99 latency for tweet loading should be and liking tweets should be less than 100ms.
• The p99 latency for creating tweets should be less than 500ms.
• The system can be eventually consistent. Different users can search for same keyword but don't see same results immediately.
• The system should be easily scalable to support more users and tweets.
• The system should be available 99.99% of time.

Capacity Estimation

Estimate the scale of the system. Consider daily active users, read/write ratio, storage requirements, bandwidth, and any relevant QPS calculations...

We need to estimate QPS and storage.

For twitter, we have 1 billion DAU. For each user, they view 10 tweets per day, search for 3 tweets and create 1 tweet per day on average. Peak QPS is average QPS * 2, so:

1. Peak view QPS = 1 billion * 10 / 24 / 3600 * 2 = 232K
2. Peak search QPS = 1 billion * 3 / 24 / 3600 * 2 = 69K
3. Peak write QPS = 1 billion / 24 / 3600 * 2 = 23K

Given we have 1 billion DAU, and on average we create 1 tweet per day. For each tweet, we assume 1KB of contents + metadata storage, and 1MB for media storage.

So we need 1TB of ADDITIONAL storage for tweet and metadata per day, and 1PB of ADDITIONAL storage for tweet media per day.

We store the tweets that were read in the past year in database. For cold and stale tweets, we store them and their associated media in cold storage.

We also need to estimate the search index:

• Raw tweet text + metadata: ~1 KB (as specified)
• ES overhead (stored fields, doc values, term dictionary, postings, norms, field data): ~2–3× raw size

Calculation:

• 1B tweets/day × 1 KB = ~1 TB/day raw data
• ES inflated size: 1 TB × ~2.5× overhead = ~2.5 TB/day
• 1-year retention: 2.5 TB × 365 = ~912 TB ≈ 0.9 PB
• 2-year retention: ~1.8 PB

Shard planning:

• With 50 GB per shard (good ES practice): ~18,000 shards for 1 year
• With 10 nodes, 36 shards/node → ~4–5 TB/node of indexed data

API Design

Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...

For users to search for a tweet:

GET v1/search_tweet {

user_id: UUID,

search_keyword: String,

time_range: String,

author: String

}

For a user to post a tweet:

POST v1/post_tweet {

user_id: UUID,

created_at: Timestamp,

tweet_text: String,

tweet_image_url: String,

hashtags: String

}

For a user to like a tweet:

POST v1/like_tweet {

user_id: UUID,

tweet_author_id: UUID,

tweet_id: UUID

}

For a user to read a tweet:

GET v1/read_tweet {

user_id: UUID,

tweet_author_id: UUID,

tweet_id: UUID

}

High-Level Design

Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.

For all traffic to the system, it goes through API gateway, which handles rate limiting and authentication, as well as load balancers, which routes requests to different servers based on consistent hashing of user_ids.

When users post a tweet, there are 3 data sources we need to write to:

1. We write the contents of the tweet to the redis cluster, and synchronously write the new tweet data to the cassandra cluster. The synchronous update between redis and cassandra might increase write latency. However, it helps us to preserve durability of data in case of cache failure, as well as consistency between cache and underlying database.
2. We upload the tweet media data to S3 object storage. Later, they are also written to local CDN when they are accessed.
3. We trigger an event on Kafka queue, which gets consumed by the index writer, and write to ElasticSearch index. We avoid directly writing to ElasticSearch index, to prevent high write throughput from overwhelming the index. The tradeoff is that data won't be immediately available in EalsticSearch index. However, since search is an approximate feature, real-time accuracy is not guaranteed.

When users read a tweet,

1. We query from the redis cluster, and if not available, query the underlying cassandra database. We also fetch media from CDN, and if not available, the S3 object storage.

When users search for a tweet:

1. For trending search terms, the results are cached in the redis cluster. If the results are not in the cache, we query the underlying ElasticSearch index, and find associated tweets.
2. For trending tweets, we need to prevent thundering herd when popular keys expire at the same time. In this case, we will use TTL jittering and stale-while-revalidate.
3. In case of cache cluster failure, ElasticSearch cluster will try to absorb all the traffic, with load shedding if traffic is overwhelming. If the ES cluster fails, the redis cache cluster will temporarily handle incoming traffic for popular search terms.

When users like a tweet:

1. The like count of a tweet is stored in redis. It is a distributed counter. We increment the tweet like count. Every once in a while, the like count for a tweet is batched, aggregated, and written to underlying cassandra database as well as ElasticSearch.

Database Design

Define the data model. Identify the main entities, their attributes, and relationships. Consider the choice of database type (SQL vs NoSQL) and justify your decision based on access patterns...

For data storage, we store below information:

1. Metadata and text about a tweet
2. Media associated with a tweet

For media information, we want to store them in object storage like S3, cached in local CDNs.

The metadata and text will be stored in 2 places:

1. A distributed noSQL database like cassandra. This is used for users to browse their tweet feed, open and view a specific tweet etc.
2. A search index like ElasticSearch, which indexes all the tweets, with their text and metadata. This is used for users to search for tweets based on filters and text.

For viewing and browsing , here's why we want to use cassandra.

1. We need to support very high write throughput across multiple regions
2. We don't need complex joins or flexible schema for our browsing use cases. We can use denormalized data.

Here's tradeoffs:

1. We will need to accept eventual consistency. Updates to a tweet or posting of a tweet might not be immediately available to everyone.

For cassandra, we will use user_id as the sharding key, so tweets from the same user end up on same shard. tweet_id and created_at can be used as the clustering columns.

Here's the detailed table schemas:

table tweets {

tweet_id: UUID, (clustering column)

author_id: UUID, (partition key)

like_count: Integer,

repost_count: Integer,

tweet_text: String,

tweet_media_links: List,

tweet_created_at: Timestamp, (clustering column)

}

For searching, we want to use an ElasticSearch cluster to store and index all the words in a tweet. ElasticSearch uses an inverted tree to index documents. It is scalable and optimized for handling search at our throughput.

We will shard the index to different shards because of the huge data size. The sharding key will be each term query. For example, for the tweet "I am Iron Man", we will store each word in a different shard based on hashing of the search term.

If we use user_id as sharding key, it might create a hot spot problem where some shards receive way more traffic than others.

If we use tweet_id as the sharding key, it does avoid the hotspot problem. However, for any query like "World Cup", we still need to hit all shards, which is less than ideal.

With sharding by search term query, we only need to hit 1 or at most a few shards per search query, which reduces latency.

To improve availability and reliability in case of huge traffic, we will also create replicas for each ElasticSearch shard. The read replicas will receive updates asynchronously. It is an eventually consistent system, which is an acceptable tradeoff for lower latency and higher availability.

Also, other 2 problems with the system are:

1. 5K QPS write spike could overwhelm the relational database cluster
2. Search traffic is heavily skewed to recent tweets

To address 1, we will use sharding and replication to scale up the database cluster, and to reduce the throughput per shard. Cassandra is a poor choice due to our need for consistency, as well as need for complex join queries across tables.

To address 2, we will have a redis cluster layer which stores trending/real-time terms and associated search results. This will reduce query latency on the critical path/

Detailed Component Design

Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.

For data storage, we store below information:

1. Metadata and text about a tweet
2. Media associated with a tweet

For media information, we want to store them in object storage like S3, cached in local CDNs.

The metadata and text will be stored in 2 places:

1. A distributed noSQL database like cassandra. This is used for users to browse their tweet feed, open and view a specific tweet etc.
2. A search index like ElasticSearch, which indexes all the tweets, with their text and metadata. This is used for users to search for tweets based on filters and text.

For viewing and browsing , here's why we want to use cassandra.

1. We need to support very high write throughput across multiple regions
2. We don't need complex joins or flexible schema for our browsing use cases. We can use denormalized data.

Here's tradeoffs:

1. We will need to accept eventual consistency. Updates to a tweet or posting of a tweet might not be immediately available to everyone.

For cassandra, we will use user_id as the sharding key, so tweets from the same user end up on same shard. tweet_id and created_at can be used as the clustering columns.

Here's the detailed table schemas:

table tweets {

tweet_id: UUID, (clustering column)

author_id: UUID, (partition key)

like_count: Integer,

repost_count: Integer,

tweet_text: String,

tweet_media_links: List,

tweet_created_at: Timestamp, (clustering column)

}

For searching, we want to use an ElasticSearch cluster to store and index all the words in a tweet. ElasticSearch uses an inverted tree to index documents. It is scalable and optimized for handling search at our throughput.

We will shard the index to different shards because of the huge data size. The sharding key will be each term query. For example, for the tweet "I am Iron Man", we will store each word in a different shard based on hashing of the search term.

If we use user_id as sharding key, it might create a hot spot problem where some shards receive way more traffic than others.

If we use tweet_id as the sharding key, it does avoid the hotspot problem. However, for any query like "World Cup", we still need to hit all shards, which is less than ideal.

With sharding by search term query, we only need to hit 1 or at most a few shards per search query, which reduces latency.

To improve availability and reliability in case of huge traffic, we will also create replicas for each ElasticSearch shard. The read replicas will receive updates asynchronously. It is an eventually consistent system, which is an acceptable tradeoff for lower latency and higher availability.

Also, other 2 problems with the system are:

1. 5K QPS write spike could overwhelm the relational database cluster
2. Search traffic is heavily skewed to recent tweets

To address 1, we will use sharding and replication to scale up the database cluster, and to reduce the throughput per shard. Cassandra is a poor choice due to our need for consistency, as well as need for complex join queries across tables.

To address 2, we will have a redis cluster layer which stores trending/real-time terms and associated search results. This will reduce query latency on the critical path/

For all traffic to the system, it goes through API gateway, which handles rate limiting and authentication, as well as load balancers, which routes requests to different servers based on consistent hashing of user_ids.

When users post a tweet, there are 3 data sources we need to write to:

1. We write the contents of the tweet to the redis cluster, and synchronously write the new tweet data to the cassandra cluster. The synchronous update between redis and cassandra might increase write latency. However, it helps us to preserve durability of data in case of cache failure, as well as consistency between cache and underlying database.
2. We upload the tweet media data to S3 object storage. Later, they are also written to local CDN when they are accessed.
3. We trigger an event on Kafka queue, which gets consumed by the index writer, and write to ElasticSearch index. We avoid directly writing to ElasticSearch index, to prevent high write throughput from overwhelming the index. The tradeoff is that data won't be immediately available in EalsticSearch index. However, since search is an approximate feature, real-time accuracy is not guaranteed.

When users read a tweet,

1. We query from the redis cluster, and if not available, query the underlying cassandra database. We also fetch media from CDN, and if not available, the S3 object storage.

When users search for a tweet:

1. For trending search terms, the results are cached in the redis cluster. If the results are not in the cache, we query the underlying ElasticSearch index, and find associated tweets.
2. For trending tweets, we need to prevent thundering herd when popular keys expire at the same time. In this case, we will use TTL jittering and stale-while-revalidate.
3. In case of cache cluster failure, ElasticSearch cluster will try to absorb all the traffic, with load shedding if traffic is overwhelming. If the ES cluster fails, the redis cache cluster will temporarily handle incoming traffic for popular search terms.

When users like a tweet:

1. The like count of a tweet is stored in redis. It is a distributed counter. We increment the tweet like count. Every once in a while, the like count for a tweet is batched, aggregated, and written to underlying cassandra database as well as ElasticSearch.

Additional things we need to be mindful for:

 Text Normalization

Text normalization. The ElasticSearch index uses the standard analyzer by default, which applies case folding and stopword removal at index time. For search, the same analyzer is applied to the query string via the match query, ensuring case-insensitive matching. Stemming (e.g., "running" → "run") is handled via a language-specific stemmer in the analyzer chain.

 Exact Phrase Search

Phrase search. ES stores term positions in the inverted index during indexing. When a user wraps terms in quotes, a match_phrase query checks both term presence and positional ordering. This is implemented at the query layer — no architectural changes needed.

Time-Filtered Queries

Time-based index rollover. ES indices are rolled hourly (e.g., tweets-2025-01-15-14, tweets-2025-01-15-15). A search with a 2-hour time window targets only 2 indices via the _search API's index pattern filtering. This avoids scanning the full 2-year corpus and enables efficient index-level TTL deletion of data older than retention policy.