Design a Document Ranking and Reranking Pipeline
Last updated: September 4, 2025
Quick Overview
Build a three-tier reranker with recall, precision, and LLM-judged stages. Include feature engineering, latency budgets, and A/B testing.
Perplexity
System Design
Software Engineer
Perplexity
September 4, 2025Software Engineer
System Design Round
System Design
Hard
12
11
4,874 solved
Build a three-tier reranker with recall, precision, and LLM-judged stages. Include feature engineering, latency budgets, and A/B testing.
Ranking quality directly determines answer quality. This tests understanding of modern information retrieval at production scale.
What the Interviewer Expects
- Design a multi-stage retrieval pipeline with clear latency budgets
- Implement feature engineering for relevance scoring
- Address online versus offline evaluation methodology
- Design A/B testing infrastructure for ranking changes
Key Topics to Cover
Multi-stage ranking
Cross-encoder reranking
NDCG and MRR metrics
A/B testing
Feature engineering
How to Approach This
- Start by clarifying functional and non-functional requirements with the interviewer.
- Estimate the scale: QPS, storage, bandwidth. This drives your design decisions.
- Draw a high-level architecture first, then deep dive into 1-2 critical components.
- Discuss trade-offs explicitly (e.g., consistency vs availability, SQL vs NoSQL).
- Address failure scenarios, monitoring, and how the system handles 10x traffic spikes.
Possible Follow-up Questions
- How do you handle position bias in evaluation?
- What features are most predictive of relevance?
- How do you detect ranking degradation in production?
Sharpen Your Skills on Codemia
Practice similar problems with our interactive workspace, get AI feedback, and track your progress.
Practice System Design ProblemsSample Answer
Requirements
Functional Requirements:
- Document Ingestion: Ability to ingest a large volume of documents continuously from various sources.
- Multi-Stage Ranking: Implement three tiers of ranking: ...
Capacity Estimation
Back-of-Envelope Calculations:
- User Queries: Assume 1 million queries per day, averaging 11 queries per second.
- Documents to Process: For each query, assume we initially retrieve 10,00...
Submit Your Answer
Markdown supported