Design a Distributed Tracing System

Functional

This problem is fundamentally different from most system design interviews. You are not designing a user-facing application; you are designing observability infrastructure. The "users" are engineers debugging production incidents. Every component you discuss, context propagation, span collection, sampling, trace reconstruction, exists to answer one question: "What happened to this request as it traveled through 20 services?"

A distributed tracing system captures the end-to-end journey of a request through a microservice architecture. Each service creates a span: a timed record of work, and the system stitches these spans into a trace that shows the complete call graph with latency breakdown per hop.

Span Volume

• Services and traffic: 500 microservices, each handling an average of 200 RPS. Total platform throughput: 100,000 RPS.
• Spans per request: An average request touches 10 services, generating 10 spans per trace. Total span volume: 100,000 RPS x 10 spans = 1,000,000 spans per minute = 16,700 spans per second.
• Peak multiplier: During traffic spikes (sales events, incident cascades), volume doubles to 2 million spans per minute. The ingestion pipeline must handle this without dropping data.

Storage

The API serves two audiences: instrumented services that emit span data (write path), and engineers who query traces for debugging (read path). The write path uses gRPC for efficiency; the read path uses REST for browser compatibility.

Span Ingestion (Write Path)

gRPC: SpanService.Export (preferred for high-throughput service-to-collector communication)

Span Schema

The span is the fundamental unit of storage. Each span represents a single operation within a trace:

1CREATE TABLE spans (
2  trace_id      FixedString(32),    -- 128-bit hex, globally unique
3  span_id       FixedString(16),    -- 64-bit hex, unique within trace
4  parent_span_id FixedString(16),   -- empty string for root span
5  service_name  LowCardinality(String),
6  operation_name String,
7  start_time    DateTime64(6),       -- microsecond precision
8  duration_us   UInt64,
9  status        Enum8('OK'=0, 'ERROR'=1, 'TIMEOUT'=2),
10  tags          Map(String, String),
11  logs          Array(Tuple(DateTime64(6), String))
12)
13ENGINE = MergeTree()
14PARTITION BY toDate(start_time)
15ORDER BY (trace_id, span_id)
16TTL start_time + INTERVAL 30 DAY

Five-Layer Pipeline

The architecture follows a five-layer pipeline pattern, where each layer has a single responsibility:

Write Path: Span Ingestion

1. A user request arrives at the API gateway. The gateway's tracing agent creates a root span with a new trace ID (128-bit random) and span ID (64-bit random). The trace ID is injected into the traceparent HTTP header for downstream propagation.
2. The gateway forwards the request to the user service. The user service's agent extracts the traceparent header, reads the trace ID and parent span ID, and creates a child span with a new span ID. The parent_span_id of this child span is set to the gateway's span ID.
3. The user service calls the payment service. The same propagation repeats: traceparent header carries the trace ID and user service's span ID. The payment service creates another child span.
4. As each service completes its work, the agent records the span's duration and status. Completed spans are placed in the agent's in-memory buffer.
5. Every few seconds (or when the buffer reaches a threshold), the agent batch-exports buffered spans to a collector instance via gRPC.
6. The collector validates the batch (drops malformed spans), enriches spans with collector metadata, and publishes to a Kafka topic partitioned by trace_id hash.
7. A storage consumer reads from the Kafka partition, deduplicates by span_id, and writes the spans to ClickHouse with the appropriate indexes.

Context Propagation (GATE)

This is the single most important mechanism in distributed tracing. Without context propagation, you have isolated spans with no way to connect them into traces. Every other component, collection, storage, querying, depends on trace context flowing correctly across service boundaries.

Sampling vs Full Collection

Full collection records every span from every request. Advantage: complete visibility, no trace is ever missing. Engineers can find any request, including rare edge cases that sampling would drop. Disadvantage: at 700 GB/day, storage and infrastructure costs are 10x higher than 10% sampling. Query performance also degrades with 10x more data to scan.

Sampling (head-based at 10%) reduces volume by 90% but blindly drops traces without knowing their value. A 1-in-a-million race condition that only manifests on specific code paths might never be captured.

Agent Crash

An agent process crashes or the host machine fails, killing both the application service and its tracing agent.

Impact: All spans buffered in the agent's memory (typically a few seconds of data) are lost. Production traffic is unaffected, the application service was already down due to the host failure, and the tracing agent crash is a side effect, not a cause.

ML-Based Anomaly Detection

Replace static threshold alerts (P99 latency above 2 seconds) with machine learning models trained on span distributions. The model learns each service's normal latency profile by hour, day, and traffic volume. Anomalies are detected when observed latency deviates significantly from the predicted baseline. This eliminates manual threshold tuning (which requires domain knowledge per service) and catches subtle degradations that fixed thresholds miss, such as a service whose P99 slowly drifts from 200ms to 400ms over two weeks.

The model can also cluster traces by call pattern and flag new patterns that have not been seen before (potential regressions from new deployments). Input features: span duration, service name, operation name, timestamp, status code, tag values. Output: anomaly score per trace.