Design an LLM Serving System at Low Latency
Last updated: September 4, 2025
Quick Overview
Serve millions of LLM queries per day with sub-second time-to-first-token. Cover batching, KV-cache management, model sharding, streaming delivery, and autoscaling.
Perplexity
September 4, 202511
7
4,150 solved
Serve millions of LLM queries per day with sub-second time-to-first-token. Cover batching, KV-cache management, model sharding, streaming delivery, and autoscaling.
Perplexity serves tens of millions of queries daily. LLM inference latency directly impacts user experience and retention.
What the Interviewer Expects
- Design batching strategies for GPU utilization
- Implement KV-cache management for memory efficiency
- Address model sharding and tensor parallelism
- Design streaming token delivery via SSE
- Plan autoscaling under variable load patterns
Key Topics to Cover
How to Approach This
- Start by clarifying functional and non-functional requirements with the interviewer.
- Estimate the scale: QPS, storage, bandwidth. This drives your design decisions.
- Draw a high-level architecture first, then deep dive into 1-2 critical components.
- Discuss trade-offs explicitly (e.g., consistency vs availability, SQL vs NoSQL).
- Address failure scenarios, monitoring, and how the system handles 10x traffic spikes.
Possible Follow-up Questions
- How do you handle cold start latency?
- What is your strategy for model updates without downtime?
- How do you manage cost per query?
Sharpen Your Skills on Codemia
Practice similar problems with our interactive workspace, get AI feedback, and track your progress.
Practice System Design ProblemsSample Answer
Inference Architecture
Deploy models across GPU clusters using tensor parallelism (split model layers across GPUs) for large models and data parallelism (replicate models) f...
KV-Cache and Memory
KV-cache stores attention keys and values from previous tokens, avoiding recomputation. For long contexts, this can consume significant GPU memory. Im...
Submit Your Answer
