Design an LLM Inference Service

Design a multi-tenant service that serves LLM completions to millions of concurrent requests across a fleet of GPUs. Clients send prompts and receive generated tokens back, either streaming one token at a time or batched into a single response. The service runs multiple model sizes (7B to 400B parameters), balances GPU utilization via continuous batching, manages KV cache across requests, supports speculative decoding and quantization, routes requests to the right model tier, and enforces per-tenant SLOs and cost budgets. The hardest constraints are latency sensitivity (time-to-first-token under 500ms p50), GPU cost (GPUs are 10-100x more expensive than CPUs), and the tight coupling between scheduling decisions and model-level mechanics like attention computation.