Requirements

Functional Requirements:

• Receive requests - API requests/validate/assign req ids and hold client connections or streams
• Batch - group compatible requests by model/hardware/region/tenant policy and token budget
• Dispatch - send batches of GPU workers and track requests inside the batch
• Route responses - map each output back to the correct user stream or response object
• Model lifecycle - support model registry versioning, A/B tests, canaries, rollback, warmup and artifact integrity
• Multiple workloads - support LLMs, vision models, embedding, rerankers and classic DL models with different batch constraints

Non-Functional Requirements:

• Latency - bound queue wait time and optimize time to first token for real time users
• Scale across A10/A100/H100 style GPU pools with prewarming and scale to zero where safe
• Prevent noisy tenants from starving others and isolate customer data in shared batches

API Design

Customers will call the normal model API, HTTP/SSE

High-Level Design

API GatewayAuth, quotas, request validation, request ID, streaming connection lifecycle.

Admission ControllerRejects or queues based on tenant quota, model availability, priority, and overload state.

Batch SchedulerGroups compatible requests by model, region, hardware type, token budget, and SLO deadline.

GPU WorkerRuns prefill/decode batches, reports health, and returns per-request outputs.

Model RegistryTracks aliases, versions, artifacts, signatures, runtime requirements, and rollout weights.

Control PlaneOwns deploys, canaries, warm pools, autoscaling policy, scale-to-zero, and rollback.

Response RouterMaps outputs from batch results back to the correct request stream/client.

State StoreTracks request status, idempotency, stream ownership, batch membership, and retry attempts.

Route H100s and A100s by capability and cost

Different GPUs have different memory, throughput, and supported batch/token budgets. The scheduler should maintain capacity profiles and route requests accordingly.

Follow-up answer: “I’d keep per-GPU-type capacity models: max tokens, expected latency, cost, and supported models. Large contexts and high-priority requests may go to H100s; smaller or lower-priority batches can go to A100s.”

Routing inputs:

• Model version and quantization/runtime compatibility.
• Prompt length, max output tokens, and KV-cache memory requirements.
• Tenant priority tier and latency SLO.
• GPU type, current queue depth, health, memory pressure, and observed tokens/sec.
• Cost policy: avoid using scarce H100s for small low-priority requests if A100s meet SLO.
• Regional/data residency constraints.

Scale by queue pressure, tokens, and GPU saturation

CPU-style autoscaling on request count is not enough. GPU serving should scale based on queue wait, token throughput, memory pressure, model residency, and SLO burn.

Interview answer: “I would scale hot pools on queue wait and GPU saturation, keep warm capacity for critical models, and use scale-to-zero only for low-priority or rare models where cold-start is acceptable.”

Autoscaling design:

• Signals: p95 queue wait, TTFT, tokens/sec, GPU memory pressure, KV-cache usage, pending tokens, and error-budget burn.
• Tech: Kubernetes Cluster Autoscaler/Karpenter, Ray autoscaler, KServe/KEDA, or custom GPU capacity controller.
• Prewarming: keep minimum replicas per hot model/version/GPU type to avoid cold-start latency.
• Scale-to-zero: okay for offline, batch, or rarely used models; risky for interactive models with strict SLOs.
• Placement: use node labels, taints/tolerations, Ray placement groups, or custom scheduler plugins for A10/A100/H100 pools.
• Cost guardrail: cap max replicas per model/tenant unless incident override or purchased capacity allows more.

Weights, KV cache, and eviction decide capacity

For LLMs, GPU memory is often the bottleneck. The scheduler must account for model weights, KV cache growth, context length, output length, and fragmentation.

Follow-up answer: “I would batch by token budget because KV cache grows with sequence length. If memory pressure rises, the scheduler reduces batch size, routes to larger GPUs, or evicts lower-priority sessions.”

Memory controls:

• Keep hot model weights resident; avoid loading/unloading per request.
• Track prefill tokens, decode tokens, max output, and KV-cache footprint per request.
• Use runtimes with paged/managed KV cache where possible, such as vLLM PagedAttention or TensorRT-LLM KV-cache management.
• Bucket requests by context length to reduce padding and fragmentation.
• Evict or cap long-running low-priority generations before they starve interactive traffic.
• Use quantization or tensor/model parallelism when model size exceeds one GPU's memory.