Requirements

Functional Requirements:

• Users can find tweets by text, time range, and/or author.
• Users can create & read tweets.
• Users can like tweets.

Non-Functional Requirements:

• 500 Million DAU
• Each user tweets 1x per day and searches 10x per day
• 500 Million writes per day
• 5 Billion reads per day
• 1 tweet is around 280 bytes of text and also metadata, so lets assume 1KB. This means .5TB of data created per day, 912TB of stored tweet data over 5 years
• Search index size of 100 bytes, 92TB over 5 years of search index data
• New tweets can probably take a bit longer to appear in our searches - on the order of a minute or two. This means our writes can be eventually consistent.
• Searches should be quick, 100-300 hundred milliseconds
• Search should be highly available and redundant (p99.9)
• Writes should be durable. They can take some time to be replicated, but should never be lost.

API Design

Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...

An API for creating a tweet:

POST /tweet

{

"text"

"userID"

}

An API with different query parameters for searching:

GET /tweets?from={t}&to={t}

GET /tweets?term={s}

GET /tweets?user={u}

where t is a timestamp, and s is a search term, and u is a userID

A basic API to retrieve a single tweet:

GET /tweet/tweetID

An API to increase like count:

PUT /like/tweetID

High-Level Design

Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.

We have a few key pieces of the design:

• An API for creating tweets that writes to a highly available, partitioned database. I think we can partition by user and then use consistent hashing to colocate some users and facilitate scale ups. We then use a CDC pipeline to replicate data from the tweet DB over to our search index.
• A search API, that gets data from a search index. It can do additional filtering, aggregating, and return aggregated data to the user. For our search index, we can use something like ElasticSearch, which utilizes a ton of caching for high availability and speed. Generally, we want to store lots of different search indexes only for our most popular searches.
• A like database, which has a redis cache in front and slower replication to the actual DB (because for some posts like write count will be very high). Then uses CDC to replicate over to our search index for any searches that need to be sorted by like count.
• An API gateway for routing traffic, load balancing, health checking.
• A CDN for highly cacheable GET results.
• When the fast search path fails to find anything, we will simply return an empty search until CDC populates something in ES. We can have a secondary fallback "cold storage" for indexes that are less hot as well (so we only store the more popular ones in ES). This will not fall back to the tweet database, so needs to be complete with any search terms.

Detailed Component Design

Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.