High-Level Design

seed URLs are the predefined urls needed to start crawling;

a URL reaches the URL frontier which has a prioritizer which pushes to multiple front queues the URLs. the queues contain URLs from the same domain inside each one. these queues are responsible for prioritization (eg home pages are prioritized over blog unofficial pages).

afterwards, messages from front queues are consumed by the front queue selector which has a higher probability to consume from a queue with a higher priority. it forwards the url to the back queue router, which also checks a mapping table which maps the host to its back queue, this way it ensures that each back queue contains urls from its own domain only.

back queues are then called; they contain urls from the same host; they are used for politeness.

back queue selector is selecting the queue that will be consumed from. worker threads (HTML Downloaders) start consuming from back queues, they only download from one request from each host at a time to ensure politeness, here we can also enforce a delay in consuming from the same host (mapped to a back queue).

this entire logic is enforcing crawling only one url at a time from the same host.

each HTML downloader needs to call DNS to first fetch the corresponding IP of the domain and caches the result. when fetching again, it can look firstly in the cache and only if it does not find it, it makes a request to DNS)

the request then reaches Content Loader, which is responsible for cheching that the same content wasn't already processed before under a different URL. in order to do this, it checks the content DB which stores the hashes of the pages' content and we compare the current page hash with the existing ones, if we find one matching than we stop the crawling as we already processed this content. if we didn't find any match than we save the content and its hash to content DB and then we reach link extractor which extracts all the links within the page. URL filter than checks if there are any blacklisted domains/links set up by the admins, also checks the length limit set and if it passes all the filters, it reaches URL finder which checks in cache firstly and in case it didnt find it, in the URL db, if the found URL wasnt already processed before, if it wasnt it pushes it to frontier and starts proccesing it.

the prioritizer checks multiple signals and computes a score based on them. the signals are:

page types - they are more prioritar than blog ones or articles
url slashes - pages with less / have higher priority (less nested)
domain - known trusted domains, such as wikipedia or news websites have higher prio
freshness
location
backlink count (pages with more inbound links)

the html downloader implements retrials with exponential back off, if the download failed with 4xx code, than it adds it back to the processing queue with a delay; if it is of type 5xx then it saves the url in the db and marks it accordingly (as failed or corrupted), if the max retries were exceeded then it adds it to dlq and it will be manually handled.

Under high load, the front queue selector adjusts consumption ratios dynamically. Normally it might consume 60% from high-priority queue, 30% medium, 10% low. Under load, it shifts to 80/15/5 to ensure important URLs are crawled first.

each component is decoupled with queues and backpressure metrics, for queue depth monitoring - when a queue reaches 80% capacity, the upstream producers slow down; pull-based consumption - workers pull at their own pace from queues; circuit breaker - if content db or dns is slowed down, we pause fetching to avoid cascading failures.

the system handles partial failures: in case the dns stops working, we can use cached IPs from DNS cache and skip temporarily the failing domains; if content db is down, we use temporarily the data stored on disk and in memory; if some workers crash, the remaining ones will continue working independently; if an entire domain queue is failing, all the others will continue to work

Submissions

Displaying Solutions with Scores Ranging from 1 to 10