Loading...
High throughput ingestion: The system handles 100K-500K log events per second in steady state, with spikes to 1M+ during production incidents when error logging increases 10-100x. No events are dropped during spikes.
High availability with fault recovery: 99.99% uptime target. Log collection continues even when individual components fail. If the search index goes down, logs buffer safely in the message queue until it recovers. No log data is lost.
Low query latency: Search queries return within 50-500ms for recent data (last 30 days in the hot index). Queries against archived data (months or years old) may take longer but must still complete within seconds.
Estimate the scale of the system. Consider daily active users, read/write ratio, storage requirements, bandwidth, and any relevant QPS calculations...
Steady state: 200K log events per second: A large infrastructure with 10,000+ microservices, each emitting 20 log events per second on average. This is 200K events/sec or roughly 17 billion events per day.
Peak during incidents: 2M events per second: When a cascading failure hits, error logging increases 10x across affected services. The system must absorb this without dropping events.
Query traffic: 1,000-5,000 queries per second: Engineers and dashboards querying the search index. During major incidents, query traffic also spikes as multiple teams investigate simultaneously.
Raw log volume: Each log event averages 500 bytes (service ID, timestamp, log level, message, metadata). At 200K events/sec: 100 MB/sec or 8.6 TB/day. With 30-day hot retention in Elasticsearch: roughly 260 TB of indexed data (raw plus inverted index doubles the footprint to approximately 520 TB).
Kafka buffer: 200K events/sec at 500 bytes = 100 MB/sec. With 7-day retention: approximately 60 TB. This is the durable buffer that absorbs spikes and enables replay.
Cold storage: After 30 days, logs move to S3 with Parquet compression (5-10x compression ratio). One year of archived data: 8.6 TB/day x 365 days / 7x compression = roughly 450 TB in S3. Significantly cheaper per GB than Elasticsearch SSDs.
Each Kafka partition handles roughly 10K-20K messages/sec with standard configuration. For 2M peak events/sec: 2M / 15K = roughly 130 partitions minimum. Use 200 partitions for headroom during consumer rebalancing and uneven service distribution.
Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...
The API has two primary paths: ingestion (writing logs) and querying (searching logs). A smaller third surface handles alert configuration. The ingestion API prioritizes throughput; the query API prioritizes flexibility.
POST /v1/logs
Headers: X-Tenant-ID, X-Request-ID (idempotency key)
Body: {
logs: [
{
service_id: string,
timestamp: ISO-8601,
level: "ERROR" | "WARN" | "INFO" | "DEBUG",
message: string,
metadata: { host: string, trace_id: string, ... }
}
]
}
Response: 202 Accepted { batch_id: string, count: number }
The batch format is intentional: agents buffer logs locally and send them in batches of 100-1,000 events. This amortizes HTTP overhead and reduces connection count on the ingestion service. The 202 response means the batch was accepted into the pipeline, not that it has been indexed yet. The client does not wait for Elasticsearch indexing to complete.
The X-Request-ID header enables idempotent ingestion. If an agent retries after a timeout, the ingestion service recognizes the duplicate batch and returns 202 without re-publishing to Kafka.
GET /v1/logs?service_id=payment-service&start=2026-03-01T00:00:00Z&end=2026-03-01T01:00:00Z&level=ERROR&q=timeout&limit=100&offset=0
Response: 200 OK {
logs: [...],
total: number,
next_offset: number
}
The query API maps directly to Elasticsearch's query DSL: service_id and level are term filters, start/end are range filters on the timestamp field, and q is a full-text search on the message field. Pagination uses offset-based cursors for simplicity.
POST /v1/alerts
Body: {
name: string,
condition: "error_rate > 5%",
window: "1m",
services: ["payment-service"],
notify: ["pagerduty://team-payments", "slack://ops-channel"]
}
Response: 201 Created { alert_id: string }
Alert rules are stored in a metadata database and pushed to the stream processor (Flink). When the Flink job evaluates log events, it checks them against active alert rules and fires notifications when conditions are met.
Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.
Log agents (Fluent Bit, Vector, Filebeat): Lightweight agents installed on every server read log files, parse structured formats, and forward events to the ingestion pipeline. Fluent Bit is preferred for its low memory footprint (roughly 1 MB per instance). Agents buffer locally on disk when the downstream pipeline is unavailable, preventing log loss during brief outages.
Direct API ingestion: Serverless functions and containerized services that cannot run a sidecar agent send logs directly via the HTTP ingestion API. The API accepts the same batch format as agents.
Kafka: The durable message queue that decouples collection from processing. Logs are published to Kafka topics partitioned by service_id, ensuring all logs for a given service land on the same partition. Kafka provides three critical guarantees: durable storage (7-day retention), ordered delivery per partition, and replay capability for recovery.
Why Kafka and not a simpler queue like RabbitMQ? At 200K events/sec steady state, Kafka's append-only log structure provides 10-100x higher throughput than traditional message brokers. Kafka also retains messages after consumption, enabling replay, which is impossible with RabbitMQ's consume-and-delete model.
Flink stream processor: Reads from Kafka, transforms logs (normalizes fields, enriches with metadata), evaluates alert rules, computes aggregations (error rates, log volume by service), and routes processed logs to the appropriate storage tier. Flink's exactly-once processing guarantees, combined with Kafka's replay capability, ensure no logs are lost or duplicated in the index.
Elasticsearch: Indexes processed logs for fast full-text search. The primary query interface for engineers investigating incidents.
S3 with Parquet: Archives logs beyond the hot retention window. Cost-efficient long-term storage queryable via Presto/Trino.
Alert service: Receives alert triggers from Flink and dispatches notifications to PagerDuty, Slack, email, or webhooks. Decoupled from the processing pipeline so notification delivery failures do not block log processing.
Two request flows define the system: the ingestion flow (writing logs) and the query flow (searching logs). Understanding each step and where failures can occur reveals why the architecture is structured with buffers and fallbacks at every stage.
Log ingestion: application emits log, agent collects and forwards to Kafka, Flink processes and indexes in Elasticsearch, raw logs archived to S3.
Step 1: Application emits log event: A microservice writes a structured log line (JSON with service_id, timestamp, level, message). The log is written to stdout or a log file, depending on the deployment model.
Step 2: Agent collects the log: Fluent Bit tails the log file (or captures stdout in a container environment), parses the structured format, and batches events. The agent buffers up to 1,000 events or 5 seconds before flushing.
Step 3: Agent sends batch to Kafka: The agent publishes the batch to the appropriate Kafka topic, partitioned by service_id. The producer uses acks=all to ensure the batch is durably replicated before acknowledging.
Step 4: Flink reads from Kafka: The stream processor reads events in micro-batches, normalizes field names (different services may use different log formats), enriches with metadata (host region, deployment version), and evaluates alert rules.
Step 5: Flink writes to Elasticsearch: Processed logs are bulk- indexed into the appropriate daily index. Bulk indexing (batches of 5,000-10,000 documents) amortizes the indexing overhead per document.
Step 6: Raw logs archived to S3: Simultaneously, Flink writes raw log data to S3 in Parquet format for long-term archival. This happens in parallel with Elasticsearch indexing so archival does not slow down the search indexing path.
Step 7: Flink commits Kafka offset: After successful writes to both Elasticsearch and S3, the consumer commits the Kafka offset. This ordering ensures at-least-once processing: if Flink crashes between steps 5 and 7, events are replayed from the last committed offset.
Step 1: Engineer submits search query: Via the Grafana/Kibana dashboard or the query API: "show me ERROR logs from payment-service in the last hour containing timeout."
Step 2: Query API parses and routes: The API determines whether the time range falls within the hot tier (last 30 days in Elasticsearch) or the cold tier (older data in S3). If the range spans both, it issues parallel queries to both tiers and merges results.
Step 3a: Hot query to Elasticsearch: The API translates the query to Elasticsearch DSL: term filter on service_id and level, range filter on timestamp, full-text match on message. Elasticsearch returns matching documents sorted by timestamp descending.
Step 3b: Cold query via Presto/Trino: For archived data, the API submits a SQL query to Presto, which reads Parquet files from S3 with partition pruning (only scanning the relevant service and date directories).
Step 4: Results returned to user: Matching log entries are returned with highlighting on the search terms. Response time is 50-200ms for hot queries and 2-10 seconds for cold queries.
Define the data model. Identify the main entities, their attributes, and relationships. Consider the choice of database type (SQL vs NoSQL) and justify your decision based on access patterns...
The storage architecture uses three tiers, each optimized for a different access pattern. The interesting design decision is not the schema but why the system needs three separate storage systems instead of one.
Index: logs-{service_id}-{YYYY.MM.DD}
Mapping:
service_id keyword (term filter, not analyzed)
timestamp date (range queries, sort key)
level keyword (term filter: ERROR/WARN/INFO/DEBUG)
message text (full-text search, analyzed)
host keyword (term filter)
trace_id keyword (term filter for distributed tracing)
metadata object (nested fields, selectively indexed)
The index naming pattern logs-{service_id}-{YYYY.MM.DD} is deliberate. Daily indexes enable fast deletion of expired data: when the retention policy says "delete logs older than 30 days," you drop the entire index for that date rather than running expensive delete-by-query operations. Service-based index prefixes allow per-service retention policies and prevent a single high-volume service from dominating the cluster.
The message field is the only analyzed (full-text searchable) field. All other fields are keyword type, meaning they are indexed as exact values for term filters. This is intentional: analyzing every field would multiply index size without adding useful search capability. Nobody searches for partial matches on service_id.
Path: s3://logs-archive/{service_id}/{YYYY}/{MM}/{DD}/
Format: Parquet (columnar, compressed)
Partitioning: service_id / year / month / day
When Elasticsearch indexes reach 30 days old, a retention manager job converts them to Parquet format and uploads to S3. Parquet's columnar compression reduces storage by 5-10x compared to raw JSON. The directory partitioning mirrors the common query pattern (filter by service and date range), enabling partition pruning when querying with Presto or Trino.
alert_rules
id UUID Primary Key
name VARCHAR
condition JSONB (rule definition)
services TEXT[] (array of service_ids)
notify_channels TEXT[] (notification targets)
created_at TIMESTAMP
updated_at TIMESTAMP
tenants
id UUID Primary Key
name VARCHAR
retention_days INTEGER (per-tenant retention policy)
quota_events BIGINT (max events/sec)
Alert rules and tenant configuration live in PostgreSQL because they are low-volume, relational data that benefits from ACID guarantees. When an alert rule is created or updated, a change event is published to Kafka so the stream processor picks up the new rule within seconds.
Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.
1. Field normalization: Different services use different log formats. Service A logs {"level": "error", "msg": "timeout"} while service B logs {"severity": "ERROR", "message": "connection timed out"}. The normalizer maps these to a canonical schema (level, message, service_id, timestamp) so downstream processing and search work uniformly.
2. Alert rule evaluation: Each normalized event is checked against active alert rules loaded in memory. Rules define conditions like "error_rate for payment-service exceeds 5% in a 1-minute tumbling window." Flink maintains windowed state (event counts per service per window) and fires alerts when thresholds are crossed. Alert evaluation happens before indexing so notifications fire even if Elasticsearch is temporarily behind.
3. Aggregation computation: The pipeline computes real-time metrics (error count by service per minute, log volume per host, p99 latency from structured log fields) and writes them to a metrics store for Grafana dashboards. These aggregations are computed incrementally, not by re-scanning stored data.
4. Bulk indexing to Elasticsearch: Processed events are buffered and flushed to Elasticsearch in bulk batches of 5,000-10,000 documents. Bulk indexing is critical: individual document indexing at 200K events/sec would require 200K HTTP requests/sec to Elasticsearch, overwhelming the cluster. Bulk batches amortize the HTTP and indexing overhead, reducing effective requests to roughly 20-40 bulk calls/sec.
One important nuance: Elasticsearch does not make documents searchable immediately upon indexing. Lucene stores data in immutable segments, and new documents only become visible after a segment refresh (default interval: 1 second). For log analysis this is acceptable because engineers debugging incidents need logs from the last 5-30 minutes, and a 1-second refresh delay is invisible at that timescale. The tradeoffs section discusses when a Cassandra hot layer in front of Elasticsearch would be warranted.
Log data is partitioned along two dimensions:
Time-based partitioning (Elasticsearch daily indexes): Each day gets a new index, enabling fast retention management (drop old indexes) and query optimization (skip irrelevant date ranges).
Service-based sharding: Within each daily index, Elasticsearch distributes data across shards. The routing key is service_id, so all logs for a given service land on the same shard. This optimizes queries that filter by service (the most common pattern) because the query only hits one shard instead of fanning out across all shards.
Naive service-based partitioning breaks when traffic is skewed. If one service (api-gateway) generates 50% of all log traffic, its Kafka partition receives 100K events/sec while other partitions sit idle. The Flink consumer assigned to that partition is overloaded. On the Elasticsearch side, all api-gateway documents route to the same shard, creating a write bottleneck and query hot spot.
The math makes this concrete. A high-traffic service producing 4 TB of logs per day generates roughly 170 GB per hour or 2.8 GB per minute. If Kafka partitions hold a single service's data and Elasticsearch shards are capped at the recommended 50 GB, one service fills a shard every 12 hours. Meanwhile, low-traffic services barely fill a few megabytes per day. Static partitioning wastes resources on one end and creates bottlenecks on the other.
Kafka-side mitigation: Instead of partitioning strictly by service_id, use a composite key of service_id + host_id. A service running on 200 hosts spreads its log traffic across 200 Kafka partitions, enabling parallel Flink consumption. The trade-off is losing strict per-service ordering, but log ordering within a single host is preserved, which is sufficient for debugging.
Elasticsearch-side mitigation: High-volume services get dedicated index templates with more primary shards. Before each daily index rollover, the index template for a service producing 4 TB per day is configured with 80 primary shards (50 GB each), while a service producing 1 GB per day shares a default index with 5 shards. Elasticsearch handles shard placement and rebalancing automatically once the template is defined, no custom shard router is needed. The key decision is choosing the right shard count per index template based on observed ingestion rates.
For a system with 10,000 services and 30 daily indexes, the total shard count can become large. The mitigation is index lifecycle management (ILM): after 7 days, daily indexes are force-merged into fewer segments and moved to warm nodes with cheaper storage.
Tiered storage lifecycle: logs move from Elasticsearch hot nodes (30 days) to S3 Parquet warm storage (12 months) to Glacier cold archive.
A background retention manager orchestrates the data lifecycle:
Hot tier (0-30 days): Logs in Elasticsearch on fast SSD nodes. Full-text search and aggregation queries complete in milliseconds.
Warm tier (30 days - 12 months): Elasticsearch indexes are converted to Parquet and uploaded to S3. Presto/Trino enables SQL queries over archived data. Query latency increases to seconds but storage cost drops 10-50x.
Cold tier (12+ months): Data moves to S3 Glacier for compliance retention. Retrieval takes minutes to hours. Only accessed for legal or forensic investigations.
Automatic deletion: The retention manager checks tenant-specific policies daily and deletes expired data. For hot tier, this means dropping Elasticsearch indexes. For warm/cold tiers, this means deleting S3 objects.