Design an LLM Serving System at Low Latency
Last updated: September 4, 2025
Quick Overview
Serve millions of LLM queries per day with sub-second time-to-first-token. Cover batching, KV-cache management, model sharding, streaming delivery, and autoscaling.
Perplexity
System Design
Software Engineer
Perplexity
September 4, 2025Software Engineer
System Design Round
System Design
Hard
11
7
4,150 solved
Serve millions of LLM queries per day with sub-second time-to-first-token. Cover batching, KV-cache management, model sharding, streaming delivery, and autoscaling.
Perplexity serves tens of millions of queries daily. LLM inference latency directly impacts user experience and retention.
What the Interviewer Expects
- Design batching strategies for GPU utilization
- Implement KV-cache management for memory efficiency
- Address model sharding and tensor parallelism
- Design streaming token delivery via SSE
- Plan autoscaling under variable load patterns
Key Topics to Cover
LLM inference optimization
KV-cache
Tensor parallelism
Speculative decoding
GPU autoscaling
How to Approach This
- Start by clarifying functional and non-functional requirements with the interviewer.
- Estimate the scale: QPS, storage, bandwidth. This drives your design decisions.
- Draw a high-level architecture first, then deep dive into 1-2 critical components.
- Discuss trade-offs explicitly (e.g., consistency vs availability, SQL vs NoSQL).
- Address failure scenarios, monitoring, and how the system handles 10x traffic spikes.
Possible Follow-up Questions
- How do you handle cold start latency?
- What is your strategy for model updates without downtime?
- How do you manage cost per query?
Sharpen Your Skills on Codemia
Practice similar problems with our interactive workspace, get AI feedback, and track your progress.
Practice System Design ProblemsSample Answer
Requirements
- Functional Requirements:
- Serve millions of LLM queries with sub-second time-to-first-token latency.
- Implement batching strategies to optimize GPU utilization.
- Manage KV-cache to impr...
Capacity Estimation
Assuming Perplexity aims to handle 50 million queries per day:
- Queries per second (QPS): 50 million / (24 * 60 * 60) = ~578 QPS.
- Time-to-first-token: Targeting sub-second latency (500 ms)....
Submit Your Answer
Markdown supported