Design a Web Crawler That Will Crawl Wikipedia

Functional

A web crawler for Wikipedia systematically fetches, stores, and indexes every article on the site. Think of it as a bot that starts at a few seed pages, follows every hyperlink, and saves each page's content, eventually visiting all ~6.5 million English Wikipedia articles. The core challenge is doing this efficiently across thousands of machines without crawling the same page twice and without overwhelming Wikipedia's servers.

• Crawl Wikipedia pages: Fetch and store the HTML content of every Wikipedia article. The crawler follows hyperlinks within Wikipedia (internal links) to discover new pages. Stored content is used downstream for search indexing, analytics, or offline access.
• URL deduplication: Never crawl the same URL twice. Before fetching, the system checks whether the URL has already been visited using a hash-based lookup. This applies to both exact URL matches and URL variations that resolve to the same page (e.g., trailing slashes, query parameter order).
• Recrawling and change detection: Wikipedia articles are edited frequently (~600 edits/minute across all languages). The crawler periodically re-fetches pages to detect changes by comparing content checksums. Only genuinely changed pages trigger a new storage write.
• Retry failed requests: Network errors, timeouts, and HTTP 5xx responses are inevitable at scale. Failed fetches are re-queued with exponential backoff. After a configurable number of retries, permanently failing URLs are moved to a dead-letter queue for manual inspection.

Scale

• Pages: English Wikipedia has ~6.5 million articles. With disambiguation pages, talk pages, and category pages, the total crawlable URL space is ~60 million pages.
• Page size: Average Wikipedia article HTML is ~100KB (varies from 5KB stubs to 500KB+ featured articles). Including images, scripts, and metadata: ~200KB average. For text-only crawling (HTML without media): ~100KB.
• Nodes: 1,000 crawler nodes. Each node can sustain ~20 concurrent HTTP fetches with async I/O. Effective per-node throughput: ~50 pages/sec (limited by network and parse time, not CPU).

Throughput

Crawl Management APIs

POST /v1/crawl/start

1{
2  "seedUrls": ["https://en.wikipedia.org/wiki/Main_Page"],
3  "config": {
4    "maxDepth": -1,
5    "politenessDelay": 200,
6    "maxRetries": 3,
7    "recrawlIntervalHours": 24,
8    "contentType": "html",
9    "domainWhitelist": ["en.wikipedia.org"]
10  }
11}

Schema

Storage Engine Choices

Components

URL Frontier: A distributed priority queue partitioned across multiple Redis instances. Each partition is a Redis sorted set holding URLs for a consistent-hashing range. When a URL is discovered, hash(url) determines which partition it belongs to, and the URL is added with a priority score (page importance, recrawl urgency). Workers are assigned to partitions via the hashing ring and pop URLs via atomic ZPOPMIN from their assigned partition, atomicity ensures each URL is fetched by exactly one worker.

Crawl Flow: Seed to Storage

1. Administrator submits seed URLs (e.g., Wikipedia Main Page) via POST /v1/crawl/start. The seed URLs are added to the URL frontier in Redis with high priority scores.
2. A crawler worker node pops the highest-priority URL from the frontier via ZPOPMIN. The URL is now "in-progress."
3. Dedup check: Worker hashes the URL and checks the Redis Bloom filter. If the Bloom filter says "definitely not seen," proceed to fetch. If "possibly seen," check Cassandra's url_metadata table for an exact match. If already crawled and not due for recrawl, skip the URL.
4. Rate limit check: Worker queries the Redis rate limiter for the target domain (en.wikipedia.org). If a token is available, acquire it and proceed. If rate-limited, return the URL to the frontier with a delayed priority score (current_time + politeness_delay).
5. Fetch: Worker sends HTTP GET to the URL. If the page has been crawled before, include If-Modified-Since header to save bandwidth (Wikipedia responds 304 Not Modified if unchanged). On success, receive HTML content. On failure (timeout, 5xx), increment retry_count and re-queue to the frontier.
6. Content hash: Compute SHA-256 of the fetched HTML. Compare with the stored content_hash in Cassandra. If identical, update only last_crawled_at timestamp, no storage write needed (page unchanged). If different, proceed to store.
7. Store: Write the compressed HTML to S3 with the content hash as key: s3://crawl-bucket/{content_hash}.html.gz. Update Cassandra: content_hash, storage_path, last_crawled_at, status = "crawled."
8. Extract links: Parse the HTML to extract all internal Wikipedia hyperlinks. Normalize URLs (resolve relative paths, strip query parameters, decode percent-encoding). For each discovered URL, add to the frontier if it passes the domain whitelist filter.
9. Repeat from step 2. The frontier grows as new URLs are discovered and shrinks as pages are crawled. The crawl completes when the frontier is empty.

URL Frontier and Deduplication: The Core Challenge

The URL frontier is the brain of the crawler. It decides what to crawl next, prevents duplicate work, and enforces politeness. Getting it wrong means either missing pages (incomplete crawl) or wasting resources on duplicate fetches.

BFS vs DFS Traversal

BFS (chosen): Breadth-first traversal discovers pages at each link depth before going deeper. Starting from the Main Page, BFS first crawls all pages linked from the Main Page, then all pages linked from those pages, and so on. This ensures broad coverage early, after the first few levels, BFS has discovered most of Wikipedia's 6.5M articles. The trade-off: BFS requires a large frontier (millions of URLs in memory at once).

DFS (rejected for Wikipedia): Depth-first follows a single link chain deep before backtracking. It finds some pages very quickly (deep in the graph) but takes much longer to achieve broad coverage. For Wikipedia, where the goal is to crawl all pages, BFS's breadth-first coverage is more useful. DFS is better for focused crawling (e.g., "crawl all pages in the 'Computer Science' category tree").

Crawler Node Failure

A crawler node crashes. Impact: URLs assigned to this node are temporarily unprocessed. URLs that were mid-fetch are lost (the HTTP response never completes). Recovery: ZooKeeper detects the missed heartbeat within 30 seconds. The coordinator removes the node's virtual nodes from the consistent hashing ring. Adjacent nodes inherit the dead node's hash ranges. Any URLs in "in-progress" status for longer than a timeout (60 seconds) are detected by a background sweeper and re-queued to the frontier. Total impact: ~30-60 seconds of crawl progress for URLs in the dead node's range. No URLs are permanently lost.

ML-Based Crawl Prioritization

Current priority scoring uses static signals (inbound link count, time since last crawl). An ML model can predict which pages are most likely to have changed since the last crawl. Training data: historical crawl results (page X was crawled at time T1 and T2; did the content hash change?). Features: time since last edit, edit frequency over the past year, page category (current events vs. historical biography), day of week, recent Wikipedia edit activity. The model outputs a "change probability" score that replaces the static priority. This focuses recrawl bandwidth on pages that actually changed, reducing the 95% wasted fetches (unchanged pages) to ~30%.

Intelligent Recrawl Scheduling