Design a Log Collection and Analysis System

A log collection and analysis system is the central nervous system of any large infrastructure. Think of it as Datadog or Splunk: every server, every microservice, every network device constantly emits log lines, and your job is to collect all of them, make them searchable within seconds, and fire alerts when something goes wrong. The core challenge is not collecting logs; it is doing so at hundreds of thousands of events per second without dropping any, while keeping search latency low enough for engineers to debug production incidents in real time.

Functional

Log aggregation from all sources: The system collects logs from application servers, infrastructure services, and network devices. Sources include structured JSON logs from microservices, syslog from Linux hosts, and access logs from load balancers. Every log event includes at minimum a service ID, timestamp, log level (ERROR/WARN/INFO/DEBUG), and a message body.

The capacity numbers for a log system are driven by one insight: the steady-state volume is large, but incident-driven spikes are the real design challenge. A system that handles the average falls over when it matters most.

Traffic

Steady state: 200K log events per second: A large infrastructure with 10,000+ microservices, each emitting 20 log events per second on average. This is 200K events/sec or roughly 17 billion events per day.

The API has two primary paths: ingestion (writing logs) and querying (searching logs). A smaller third surface handles alert configuration. The ingestion API prioritizes throughput; the query API prioritizes flexibility.

Log Ingestion

1POST /v1/logs
2Headers: X-Tenant-ID, X-Request-ID (idempotency key)
3Body: {
4logs: [
5{
6service_id: string,
7timestamp: ISO-8601,
8level: "ERROR" | "WARN" | "INFO" | "DEBUG",
9message: string,
10metadata: { host: string, trace_id: string, ... }
11}
12]
13}
14Response: 202 Accepted { batch_id: string, count: number }

The storage architecture uses three tiers, each optimized for a different access pattern. The interesting design decision is not the schema but why the system needs three separate storage systems instead of one.

Elasticsearch: Hot Search Index

1Index: logs-{service_id}-{YYYY.MM.DD}
2Mapping:
3service_id keyword (term filter, not analyzed)
4timestamp date (range queries, sort key)
5level keyword (term filter: ERROR/WARN/INFO/DEBUG)
6message text (full-text search, analyzed)
7host keyword (term filter)
8trace_id keyword (term filter for distributed tracing)
9metadata object (nested fields, selectively indexed)

The architecture is a pipeline with four stages: collection, buffering, processing, and storage. Each stage is independently scalable, and the message queue between stages absorbs load differences. This decoupling is the fundamental design principle: no component directly depends on the throughput capacity of another.

Two request flows define the system: the ingestion flow (writing logs) and the query flow (searching logs). Understanding each step and where failures can occur reveals why the architecture is structured with buffers and fallbacks at every stage.

Log Ingestion Flow

The core engineering challenge (the GATE of this problem) is building a log ingestion pipeline that reads events from Kafka, transforms and normalizes them, evaluates alert rules in real time, and indexes them in Elasticsearch for near-real-time search, all at 200K+ events per second without losing or duplicating any logs. Every component in this section exists to solve some aspect of that challenge.

Stream Processing Pipeline

Every architectural decision in this system involves a trade-off between cost, latency, and completeness. Understanding these trade-offs and articulating them clearly is what distinguishes a strong system design answer.

Elasticsearch vs. Custom Storage Engine

At extreme scale, companies like Datadog and Splunk moved away from Elasticsearch and built custom storage engines. Why? Elasticsearch stores full inverted indexes for every field, which consumes significant memory and disk. At 50 TB/day, the storage and operational cost of Elasticsearch clusters becomes a major expense. Custom engines can optimize specifically for log-shaped data: append-only writes, time-partitioned storage, aggressive columnar compression, and query patterns tailored to timestamp + keyword filters rather than general-purpose full-text search.

The system's reliability comes from its ability to handle component failures without losing log data. Each failure scenario maps to a specific architectural decision made earlier in the design.

Elasticsearch Cluster Outage

Scenario: The Elasticsearch cluster becomes unavailable due to a network partition or node failures.

These improvements extend the system without changing the core pipeline architecture. Each builds on the existing Kafka and Flink infrastructure.

ML-Based Anomaly Detection

Rule-based alerting (error rate exceeds 5%) catches known failure patterns but misses novel anomalies. An ML model trained on historical log patterns detects unusual behavior automatically: a sudden change in log volume distribution, new error message patterns, or latency shifts that do not cross fixed thresholds. The model runs as an additional operator in the Flink pipeline, scoring each event window with an anomaly probability. Events above a configurable threshold trigger investigation alerts.