Design a Large-Scale Web Crawler
Last updated: September 4, 2025
Quick Overview
Index billions of pages with crawl scheduling, politeness controls, duplicate detection, content extraction, and freshness policies.
Perplexity
System Design
Software Engineer
Perplexity
September 4, 2025Software Engineer
System Design Round
System Design
Hard
14
4
3,377 solved
Index billions of pages with crawl scheduling, politeness controls, duplicate detection, content extraction, and freshness policies.
Perplexity needs fresh web content to power its AI search. This question tests distributed systems fundamentals applied to web-scale data collection.
What the Interviewer Expects
- Design crawl scheduling with domain-level politeness
- Implement near-duplicate detection at scale
- Address content extraction and boilerplate removal
- Design freshness policies for different content types
- Plan for robots.txt compliance and ethical crawling
Key Topics to Cover
Distributed crawling
URL frontier
Content fingerprinting
Politeness policies
Page rank prioritization
How to Approach This
- Start by clarifying functional and non-functional requirements with the interviewer.
- Estimate the scale: QPS, storage, bandwidth. This drives your design decisions.
- Draw a high-level architecture first, then deep dive into 1-2 critical components.
- Discuss trade-offs explicitly (e.g., consistency vs availability, SQL vs NoSQL).
- Address failure scenarios, monitoring, and how the system handles 10x traffic spikes.
Possible Follow-up Questions
- How do you prioritize which pages to crawl first?
- How do you handle JavaScript-rendered pages?
- What is your strategy for detecting and handling crawler traps?
Sharpen Your Skills on Codemia
Practice similar problems with our interactive workspace, get AI feedback, and track your progress.
Practice System Design ProblemsSample Answer
Requirements
- Functional Requirements:
- Crawl Scheduling: Implement a distributed system to manage crawl requests, prioritizing based on page rank, freshness, and politeness.
- *Politeness Controls...
Capacity Estimation
Assuming Perplexity aims to index 1 billion pages and each page is crawled every 30 days:
- Pages per Second: To maintain freshness, we need to crawl approximately 1 billion pages / 30 days = 1.16...
Submit Your Answer
Markdown supported