Design an LLM Inference Service

An LLM inference service takes a prompt plus sampling parameters and streams generated tokens back to the client. On the surface this looks like any other stateless web API. It is not. The work per request is measured in GPU-seconds rather than milliseconds, the hardware is 10 to 100 times more expensive per hour than CPUs, and a single distracted scheduling decision can leave a 40K dollar accelerator idle. Every requirement we write has to be read through that lens.

Functional Requirements

• Completion API: accept a prompt plus sampling parameters (temperature, top-p, max tokens, stop sequences) and return generated tokens. Support both streaming (SSE, token-by-token) and non-streaming (single JSON response) modes. Streaming dominates for interactive chat because time-to-first-token is what users perceive as latency.
• Multi-model serving: host 7B, 70B, and 400B+ model variants side by side. A model router picks the tier based on price point, requested capability, and per-tenant configuration. Tenants can pin a specific checkpoint version for reproducibility.
• Function calling and structured output: tool invocations with typed arguments, JSON-schema-constrained responses. All implemented via constrained decoding at the logits layer, not post-hoc validation.
• Long context: frontier models today accept 128K to 1M token contexts. Code analysis, legal review, and agentic traces routinely hit these ceilings. The service must handle them without falling over, though it can price and rate-limit them differently.
• Per-request cost and usage: every response reports prompt tokens, completion tokens, and (optionally) reasoning and cached tokens. Billing rolls those up per tenant per billing period.

Capacity estimation for an LLM service starts from tokens, not requests. Tokens connect directly to GPU-seconds, which connects directly to dollars. Every other number — RPS, bandwidth, storage — falls out of token-level accounting.

Traffic shape

A single cluster serves around 50K requests per second at the API level. The token mix matters more than that number:

The API surface is shaped by three realities: streaming is the default, every request costs real money per token, and users call the API from dozens of SDKs over dozens of years. Backward compatibility matters more than elegance.

The completion endpoint

A single primary endpoint accepts a messages array plus sampling parameters:

An inference service is mostly stateless — the expensive state lives on GPUs for seconds at a time. But the service still owns several persistent stores, each shaped by very different access patterns.

Tenant and API key registry

Small, read-mostly. Postgres is a fine fit. The API key is hashed (HMAC-SHA256 with a server-side pepper) before storage; the raw key is never persisted after creation. The hot path reads this table on every request, so it is cached in the gateway with a 60-second TTL keyed by the key hash.

Zoom all the way out and the service is a pipeline: a request enters at the gateway, gets classified, routed to the right GPU worker pool, scheduled into a batch, run through the model, and streamed back.

Client-facing layer

Walk through the two flows that dominate load: a streaming chat request and a large batch request. Everything else is a variant.

Flow 1: streaming chat completion with prefix cache hit

The core of this service is a set of tightly-coupled GPU-aware mechanisms. Each one is simple in isolation but they interact in ways that shape the whole system. I'll walk through the four that matter most.

Continuous batching

Most of the interesting choices here are not database-vs-database picks but scheduling and precision trade-offs that directly shape cost and latency.

Continuous batching vs static batching

Already covered: continuous wins for general-purpose serving. Static batching still has a niche in offline evaluation where determinism across runs matters more than utilization.

Three failure modes to internalize: GPU loss, KV cache exhaustion, and tokenizer skew. Each has a surprising failure pattern.

Worker crash mid-stream

Covered in flows. Key points: the KV cache of the crashed worker is gone, all in-flight requests on that worker fail, replacing the worker takes 1 to 3 minutes (pull weights, warm up), and other workers in the pool absorb traffic. If the pool is sized for N-1 redundancy (typical), TTFT rises 10-20% during the gap. If multiple workers crash simultaneously (bad deploy, hardware rack failure), admission control kicks in and rejects requests with HTTP 529 rather than queueing unboundedly.

Where does this system go in the next 12 to 24 months?

Disaggregated prefill and decode

Prefill and decode have very different compute profiles. Prefill is compute-bound (dense matmul on full prompt), decode is memory-bound (one token at a time, weights stream from HBM). Running them on the same GPU in the same pool wastes the compute imbalance — decode sees underutilized FLOPs, prefill queues behind decode. Disaggregated architecture: dedicated prefill workers feed a shared KV cache (via RDMA or NVLink over fabric), decode workers read from that cache and generate tokens. Published results are dramatic: DistServe (OSDI 2024) reports up to 7.4x more requests served within the same TTFT+TPOT SLO on mixed workloads; Moonshot's Mooncake reports up to 525% throughput improvement on long-context chat traffic in production. The operational cost is a shared KV fabric — not trivial — but the economics tip favorable above 10K-GPU scale.