Requirements

Functional Requirements:

• Trace requests across services to provide end-to-end visibility into latency breakdown per hop.
• Collect and store span data, capturing timing, metadata, and parent-child span relationships for querying.
• Visualize traces in a web dashboard, showing waterfalls, dependency graphs, and latency breakdowns.
• Allow users to tag and annotate traces with metadata (user ID, service version) for filtering and categorization.
• Integrate traces with logging and monitoring for unified observability.

Non-Functional Requirements:

• Trace ingestion should be async, should not block the target system's data flow.
• Volume: 1B traceIds ingested per day
• Latency: from ingest to viewable on the dashboard, a latency of around 500ms ~ 1s should be reasonable
• HA: 99.9%
• Scalability: should be easy to scale to handle burst volume

API Design

SDK: Provide an SDK for the services to generate traceIds and emit them.

POST /trace

req: service_name, start_ts, end_ts, traceId, meta)

resp: acknowledge success

GET /trace?service={service}&traceId={traceId}

resp: list[Trace]

GET /trace/list?service={service}&page_no=1

resp: list[Trace] + page_no + total_cnt

High-Level Design

Data Model:

Trace:

start_ts, end_ts, traceId (uk), spanId, meta

Service Metadata:

id (pk), service_name, owner, org_id, etc.

How to Ingest:

The traceId is generated inside the service, and will flow along with the request via threadlocal or explicit context passing. The SDK provides the capability to support multi-thread/coroutine passing.

Trace Ingestion Data Flow:

• ServiceA generates a traceId&spanId, emits a trace (serviceA, start_ts, end_ts, traceId, spanId, meta)
• ServiceB gets the traceId&spanId, emits a trace (serviceB, start_ts, end_ts, traceId,spanId, meta)
• ServiceC: (serviceC, start_ts, end_ts, traceId, spanId, meta)
• ...

Main Trace Processing Data Flow:

Trace Ingestion Gateway -> Kafka -> Cassandra DB -CDC-> Kafka -> Trace Processing Service -> ElasticSearch

Storage Est.

Kafka as storage: 1000_000_000 * 500 bytes * 400 = 800TB

MongoDB: 1000_000_000 * 100 bytes * 400 = 40 TB

Scalability:

Trace Ingestion Gateway will push trace data tuple (service_name, start_ts, end_ts, traceId, spanId, meta) to Kafka, partition by the traceId, so the Kafka can be scaled out easily

Detailed Component Design

Service

• Trace Ingestion Gateway: ingests the trace and emits it into Kafka, translates service_name to service_id. Kafka services as the main raw event store here for later replay.
• Trace Processing Service stores the trace info to ES for searching by traceId, spanId, and service_id purposes
• Flink subscribes from Kafka, updates it's internal data structures for the service dependency relationships, and updates the MongoDB service dependency relationship periodically.

How to generate the traceId and spanId:

The entry service generates the traceid, and propagates to the downstream participant services. the traceId will be propagated in the reques via the network.

When a service gets a request, it checks if a traceid exists in the header or a conventional field, if not, generate a new one, if yes, use and propagate it.

Each service will also generate a spanId.

When to emit the trace info:

Each service will emit the trace info via the API, when it finishes the request. The lifecycle will be: generate/take traceId -> request downstream services and propagate traceId -> prepare response and emit trace info via API.

Fault tolerant:

All traceId emit should be async, the SDK controls the traceId/spanId generation and the emit logic. If failed, the SDK discards the trace info.

The SDK can also be dynamically configured to sample and emit the trace at a specific rate (1%, 10%, 100%, etc).

How to reconstruct the span from trace:

Users mainly queries by traceId, spanId is for distincting between services, each service gets a unique spanId. When querying for trace, ES returns a list of (traceId, spanId, service_name, start_ts, end_ts), and the UI reconstructs the data flow: (start_ts, serviceA (spanId_1), end_ts) -> (start_ts, serviceB (spanId_2), end_ts) -> (start_ts, serviceC (spanId_3), end_ts)