Design a Document Ranking and Reranking Pipeline
Last updated: September 4, 2025
Quick Overview
Build a three-tier reranker with recall, precision, and LLM-judged stages. Include feature engineering, latency budgets, and A/B testing.
Perplexity
September 4, 202512
11
4,874 solved
Build a three-tier reranker with recall, precision, and LLM-judged stages. Include feature engineering, latency budgets, and A/B testing.
Ranking quality directly determines answer quality. This tests understanding of modern information retrieval at production scale.
What the Interviewer Expects
- Design a multi-stage retrieval pipeline with clear latency budgets
- Implement feature engineering for relevance scoring
- Address online versus offline evaluation methodology
- Design A/B testing infrastructure for ranking changes
Key Topics to Cover
How to Approach This
- Start by clarifying functional and non-functional requirements with the interviewer.
- Estimate the scale: QPS, storage, bandwidth. This drives your design decisions.
- Draw a high-level architecture first, then deep dive into 1-2 critical components.
- Discuss trade-offs explicitly (e.g., consistency vs availability, SQL vs NoSQL).
- Address failure scenarios, monitoring, and how the system handles 10x traffic spikes.
Possible Follow-up Questions
- How do you handle position bias in evaluation?
- What features are most predictive of relevance?
- How do you detect ranking degradation in production?
Sharpen Your Skills on Codemia
Practice similar problems with our interactive workspace, get AI feedback, and track your progress.
Practice System Design ProblemsSample Answer
Three-Stage Pipeline
Stage 1 (Recall, <20ms): BM25 + dense retrieval, top 200 candidates. Stage 2 (Precision, <50ms): Cross-encoder reranker on top 50 candidates, scoring ...
Evaluation
Offline: NDCG@10 and MRR on human-annotated query-passage relevance judgments. Online: A/B test with metrics like answer acceptance rate, citation cli...
Submit Your Answer
