System requirements

Functional:

• Twitter users will use this service to search for tweets based on specific keywords, hashtags, or users

Non-Functional:

• Users expect search results quickly. Returning results in a few seconds is sufficient
• New tweets need to be incorporated into the search index quickly
• Scalable, resilient, etc.

Capacity estimation

Estimate the scale of the system you are going to design...

• 500 million DAU users
• 10% DAU perform searches daily: 50 million users
• Average # of search requests per user per day: 5
• Total search requests per day: 50 million * 5 = 250 million
• Requests per minute: 250 million / (24 hours * 60 minutes) ≈ 173,611 requests per minute
• If a user tweets 5 tweets a day, 50 million * 5 = 250 million new tweets per day. We'll store metadata, the tweet content, hashtag, so we might have 200 bytes per tweet. That's 50 GB of data generated daily.

API design

Define what APIs are expected from the system...

Search endpoint:

GET /search

• User will use at least one of the following query parameters
• userId: string
• keyword: string
• hashtag: string
• offset: int
• limit: int

TODO: add more details about pagination

Database design

Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...

We can consider using a distributed NoSQL database for the tweet store like MongoDB. NoSQL databases are easy to scale and have high availability. They are also flexible, so we can change schema design as needed to accommodate different tweet content and attributes like hashtags.

High-level design

You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...

The design can be broken into two main services -- one for handling search queries and another for indexing tweeted content.

High-level flow:

1. Users send query requests that are directed to the Elastic Load Balancer (ELB).
2. Statelesss web servers behind the ELB handle routing, request processing, and authentication.
3. Web servers forward the query requests to the Query Service for searching the index and retrieving relevant tweets.
4. Simultaneously, the Indexing Service crawls through the Tweet Store, indexes tweeted content based on user ID, hashtag, and content words, ensuring that new tweets are searchable.

Request flows

Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...

1. User sends GET requests to the search service. This request is handled by the stateless web servers.
2. The traffic load is distributed among the web servers and it sends a query to the query service.
3. The query service queries elastic search to find the most relevant tweets given the user request parameters. We can get the results in reverse chronological order. If the search result is quite larger, we can paginate the query using offset limit or cursor pagination.
4. The results are returned to the users.

Detailed component design

Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...

We can consider using ElasticSearch with full-text search to help index and search tweet content based on keywords, hashtags, and user IDs. We'll leverage MongoDB as the primary database for storing tweet information. We will have a separate indexing service to batch indexing. The service will be scheduled so we can group new tweets into larger indexing operations to reduce overhead and improve efficiency of index updates. We can adjust this indexing frequency based on the traffic load on the service or time intervals or both. We can optimize for indexing most recent tweets, since users probably want to view the most recent events.

Data indexing approach

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

Redis vs ElasticSearch

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

• ElasticSearch indexes are crucial to the service, so if the clusters fail and are expensive to rebuild, users will experience a downtime or slowness in search speed.
• the NoSQL database can be a single point of failure, but it is important for both search and content storage.
• Index corruption with hardware failures, so data integrity could be at risk and could affect search results
• Data replication across multiple data center locations could cause slowness in the search service. Data synchronization can also happen if there was network connectivity issues.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?

• Replicate the NoSQL database to ensure resilience
• Implement monitoring tools to track cluster health and performance metrics of the ElasticSearch clusters. Consider using replicas for index data to ensure high availability in case of node failures.
• Introduce more layers of caching for the popular users or viral tweets. For trending content, we should consider using a Redis cache to store the most queried content in a given timeframe.
• Automatically scale up the query service when traffic increases I
• implement a robust data synchronization mechanism between the indexing service and ElasticSearch to ensure real-time updates without compromising search performance.