System requirements

Functional:

User should be able to search public websites based on text.

The system should be able to crawler public website and store results.

The search must be real time.

Return auto-complete/typeahead while user typing the search.

Non-Functional:

Low latency

Scalability

Eventually consistency

Security

Capacity estimation

Assume 1B daily active user

Read QPS: 1B / 100k = 10k

Peak read QPS: 2 * 10k = 20k

Storage estimation:

Assume the system crawler 100B website in total. Each website will need to store 100k information for text and images. Storage need: 100B * 100k = 10PB

Each data store needs to have 2 more replicas. In total the storage need is 3 * 10PB = 30PB.

API design

GET getAutoComplete(text)

return a list of strings for auto complete/typeahead. Or return error message if any error.

GET getSearchResults(text)

return the list of website info which matches the search text. Or return error message.

Database design

See diagram

High-level design

See diagram

Request flows

See diagram

Detailed component design

See diagram

Trade offs/Tech choices

See diagram

Failure scenarios/bottlenecks

Our non-functional requirements are scalability, fault tolerance, availability and low latency. Let's discuss how the proposed system fulfills these requirements:

1. Scalability: The proposed system is scalable to handle an increasing number of requests. The required resources, including load balancers, services, DB Cache, SQL DB, NoSQL DB could be horizontally scaling. They will be sharded by itemId and userId. They will be added/removed on demand.
2. Fault tolerance: The DB Cache, SQL DB, NoSQL DB will be sharded by itemId and userId. Each shard will have at lease 3 replicas. The replicas will be located in different data centers in case the whole data center is down. Moreover, the redundant resources are always there to handle the failure of a server or its component. Monitoring service is used to enhance reliability by continuously observing system health, detecting issues early, providing insights for optimization, and assisting in timely incident response.
3. Availability: The system is highly available by providing redundant servers and replicating data on them. When a user gets disconnected due to some fault in the server, the session is recreated via a load balancer with different and redundant database clusters, which provides high availability and durability.
4. Low latency: We can minimize the system's latency. Because we use DB cache as the read replica, the read request could read from cache. Also the recommendations are pre-calculated by ML models and store to a NoSQL DB for fast ready. The recommendations will be eventually consistent in return of low latency.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?