Design a Web Crawler

A web crawler is a program that systematically downloads web pages by following hyperlinks. It sounds simple, fetch a page, extract links, repeat. But at scale, every step hides a distributed systems problem. How do you avoid crawling the same page twice when a thousand machines are discovering URLs simultaneously? How do you respect each website's rate limits when you are targeting millions of domains? How do you store billions of pages without drowning in storage costs?

Functional

• Crawl web pages across millions of domains: Starting from a set of seed URLs, the crawler follows hyperlinks to discover and download pages across the open web. The scope is not a single site but the entire reachable web. Each fetched page is stored for downstream processing, search indexing, data mining, or archival.
• URL deduplication: The web is a graph with cycles. Page A links to B, B links to C, C links back to A. Without deduplication, the crawler loops forever. Before fetching, the system checks whether a URL has already been visited. At 5 billion URLs, this check must be sub-millisecond and memory-efficient.
• Respect robots.txt: Every domain publishes a robots.txt file specifying which paths crawlers may or may not access, and optionally a crawl delay. The crawler must fetch and cache robots.txt for each domain before crawling any page on that domain. Ignoring robots.txt gets the crawler IP-banned and violates web etiquette.
• Handle redirects and URL normalization: HTTP 301 and 302 redirects are common. The crawler must follow redirect chains (up to a configurable limit) and normalize URLs to a canonical form, lowercasing the scheme and host, removing default ports, sorting query parameters, stripping fragments. Without normalization, the same page appears as multiple distinct URLs.
• Recrawl pages to detect changes: Web content changes constantly. The crawler periodically revisits pages to detect updates. Change detection compares content checksums, if the page has not changed, no storage write is needed. Recrawl frequency is adaptive: pages that change often are recrawled more frequently.

The numbers drive every architectural decision, how many nodes, what storage backend, whether the Bloom filter fits in memory, and how long the initial crawl takes. Getting these estimates wrong means building for the wrong problem.

Scale

• Total pages: 5 billion crawlable pages across the open web. This is a fraction of the estimated 50+ billion total pages, but covers the most-linked and most-updated content.
• Average page size: 100 KB of HTML (ranges from 5 KB for simple pages to 500 KB+ for content-heavy sites). With gzip compression at roughly 5:1, stored pages average about 20 KB each.
• Crawler nodes: 1,000 worker nodes, each running 50 concurrent fetchers using async I/O. Each node can sustain roughly 50 pages/sec (limited by network, DNS resolution, and politeness constraints rather than CPU).

The web crawler is primarily a batch system, not a request-response service. But it still needs an API for operators to manage crawl jobs, submit URLs, and retrieve crawled content.

Crawl Management

POST /v1/crawl/start

Schema

Storage Engine Choices

The architecture separates concerns into stateless workers, a distributed URL frontier, and durable storage. Every component is horizontally scalable and independently deployable. The key insight is that making workers stateless (storing all state in external systems) enables effortless scaling and recovery.

Architecture Overview

Crawl Flow: Seed to Storage

1. Seed URLs submitted: An operator starts a crawl job with seed URLs (popular sites, web directories, or a list from a previous crawl). Seeds are added to the URL frontier with high priority scores.
2. Worker pops a URL: A fetcher worker pops the highest-priority URL from its assigned frontier partition via ZPOPMIN. The URL is now in-progress.
3. robots.txt check: The worker looks up the URL's domain in the robots.txt cache. If no cached entry exists (or the cache has expired), the worker fetches robots.txt from the domain first and caches the parsed rules. If the URL's path is disallowed, the worker marks it as robots_blocked in Cassandra and moves to the next URL.
4. Dedup check: The worker hashes the normalized URL and checks the partitioned Bloom filter. If the Bloom filter says definitely not seen, proceed. If possibly seen (roughly 1% of checks), check Cassandra for an exact match. If already crawled and not due for recrawl, skip.
5. DNS resolution: The worker resolves the domain to an IP address via the local DNS cache. Cache hits are sub-millisecond. Cache misses trigger an async DNS lookup (10-100 ms), the worker processes other URLs while waiting.
6. HTTP fetch: The worker sends an HTTP GET request to the resolved IP. For recrawls, it includes If-Modified-Since to save bandwidth (the server responds 304 Not Modified if unchanged). The worker follows redirects up to a configurable limit (typically 5 hops). On timeout or HTTP 5xx, the URL is re-queued with exponential backoff.
7. Content processing: The worker computes a SHA-256 hash of the fetched HTML. If the hash matches the stored content_hash in Cassandra, the page has not changed, update only the timestamp. If different, store the compressed HTML in S3 with the content hash as key.
8. Link extraction: The worker parses the HTML to extract hyperlinks. Each discovered URL is normalized (lowercase host, strip fragments, resolve relative paths) and checked against the domain whitelist/blacklist. URLs passing all filters are pushed to the appropriate frontier partition based on consistent hashing.
9. Metadata update: The worker updates Cassandra: status=crawled, content_hash, storage_path, last_crawled_at. The Bloom filter is updated to include this URL.
10. Repeat: The worker pops the next URL. The cycle continues until the frontier is empty or the crawl is paused. The entire loop, from popping a URL to pushing discovered links, takes roughly 200-500 ms per URL when DNS and robots.txt are cached. The bottleneck is the HTTP fetch (network latency to the target server), which is why each worker runs 50 concurrent fetchers using async I/O.

The URL frontier is the core challenge of a web crawler. It must simultaneously optimize three competing objectives: crawl the most important pages first (priority), never overwhelm any single host (politeness), and never crawl the same URL twice (deduplication). Getting the frontier wrong means either incomplete crawls, angry site operators, or wasted resources on duplicate work.

URL Frontier: The GATE Challenge

BFS vs DFS vs Best-First

BFS (breadth-first) explores all pages at link depth N before moving to depth N+1. Starting from seeds, BFS discovers a wide variety of pages quickly. After a few levels, most of the web's highly-linked pages are found. The cost: BFS requires a large frontier (millions of URLs at each depth level), consuming significant memory.

With 1,000 nodes running continuously for days, failures are not exceptional, they are routine. The system must handle failures automatically without human intervention and without losing crawl progress.

Fetcher Node Failure

A fetcher node crashes. URLs that were in-progress on that node are stuck in in_progress status in Cassandra. A background sweeper runs every 60 seconds, queries Cassandra for URLs where status = 'in_progress' AND updated_at < now() - 60s, and re-queues them to the frontier. The URLs are re-assigned to a different worker by the consistent hashing ring (ZooKeeper has already removed the dead node). Total impact: 30-60 seconds of lost progress for URLs in the dead node's range. No URLs are permanently lost.

JavaScript Rendering

Many modern websites load content dynamically via JavaScript. A standard HTTP GET returns a minimal HTML shell with JavaScript that fetches and renders content client-side. The crawler's simple HTTP fetch misses this content entirely. Adding a headless browser pool (Puppeteer or Playwright) enables JavaScript rendering: after fetching the initial HTML, the page is loaded in a headless browser that executes JavaScript, waits for content to load, then captures the rendered DOM. This is expensive, roughly 2-5 seconds per page versus roughly 100 ms for static HTML, so it should be applied selectively to sites known to require JavaScript rendering.

ML-Based Crawl Prioritization