Requirements

Functional Requirements:

• Trace requests across services to provide end-to-end visibility into latency breakdown per hop.
• Collect and store span data, capturing timing, metadata, and parent-child span relationships for querying.
• Visualize traces in a web dashboard, showing waterfalls, dependency graphs, and latency breakdowns.
• Allow users to tag and annotate traces with metadata (user ID, service version) for filtering and categorization.
• Integrate traces with logging and monitoring for unified observability.

Non-Functional Requirements:

• Trace ingestion should be async, should not block the target system's data flow.
• Volume: 1B traceIds ingested per day
• Latency: from ingest to viewable on the dashboard, a latency of around 500ms ~ 1s should be reasonable
• HA: 99.9%
• Scalability: should be easy to scale to handle burst volume

API Design

SDK: Provide an SDK for the services to generate traceIds and emit them.

POST /trace

req: service_name, start_ts, end_ts, traceId, meta)

resp: acknowledge success

GET /trace?service={service}&traceId={traceId}

resp: list[Trace]

GET /trace/list?service={service}&page_no=1

resp: list[Trace] + page_no + total_cnt

High-Level Design

Data Model:

Trace:

start_ts, end_ts, traceId (uk), spanId, meta

Service Metadata:

id (pk), service_name, owner, org_id, etc.

How to Ingest:

The traceId is generated inside the service, and will flow along with the request via threadlocal or explicit context passing. The SDK provides the capability to support multi-thread/coroutine passing.

Trace Ingestion Data Flow:

• ServiceA generates a traceId&spanId, emits a trace (serviceA, start_ts, end_ts, traceId, spanId, meta)
• ServiceB gets the traceId&spanId, emits a trace (serviceB, start_ts, end_ts, traceId,spanId, meta)
• ServiceC: (serviceC, start_ts, end_ts, traceId, spanId, meta)
• ...

Main Trace Processing Data Flow:

Trace Ingestion Gateway -> Kafka -> Cassandra DB -CDC-> Kafka -> Trace Processing Service -> ElasticSearch

Storage Est.

Cassandra: 1000_000_000 * 500 bytes * 400 = 800TB

MongoDB: 1000_000_000 * 100 bytes * 400 = 40 TB

Detailed Component Design

• Trace Ingestion Gateway: ingests the trace and emits it into Kafka, translates service_name to service_id.
• Cassandra DB: stores the trace as a raw data source, and the tuple (service_id, traceId, spanId, start_ts, end_ts)
• Trace Processing Service stores the trace info to ES for searching by traceId, spanId, and service_id purposes