Design an AI Answer Caching System
Last updated: September 4, 2025
Quick Overview
Design a caching layer that reuses LLM answers for semantically similar queries while handling time-sensitive invalidation.
Perplexity
System Design
Software Engineer
Perplexity
September 4, 2025Software Engineer
System Design Round
System Design
Medium
11
4
3,415 solved
Design a caching layer that reuses LLM answers for semantically similar queries while handling time-sensitive invalidation.
LLM inference is expensive. Semantic caching can dramatically reduce cost and latency for repeated or similar queries.
What the Interviewer Expects
- Design both exact-match and semantic similarity caching
- Implement time-based invalidation for different query types
- Address cache warming and eviction strategies
- Discuss cost savings versus answer freshness trade-offs
Key Topics to Cover
Semantic caching
Embedding similarity
Cache invalidation
LLM cost optimization
Query classification
How to Approach This
- Start by clarifying functional and non-functional requirements with the interviewer.
- Estimate the scale: QPS, storage, bandwidth. This drives your design decisions.
- Draw a high-level architecture first, then deep dive into 1-2 critical components.
- Discuss trade-offs explicitly (e.g., consistency vs availability, SQL vs NoSQL).
- Address failure scenarios, monitoring, and how the system handles 10x traffic spikes.
Possible Follow-up Questions
- How do you set the similarity threshold for semantic cache hits?
- How do you prevent serving stale answers for trending topics?
- What is the expected cache hit rate?
Sharpen Your Skills on Codemia
Practice similar problems with our interactive workspace, get AI feedback, and track your progress.
Practice System Design ProblemsSample Answer
Requirements
Functional Requirements
- Exact-match Caching: Store and retrieve answers for queries that exactly match previous submissions.
- Semantic Similarity Caching: Leverage embedding models to...
Capacity Estimation
To estimate capacity, let's assume Perplexity handles approximately 1 million queries per day, with an average of 10 queries per second during peak hours.
Back-of-Envelope Calculations
- **Daily ...
Submit Your Answer
Markdown supported