AWS
CloudWatch Insights
Log Aggregation
Data Visualization
Time Series Analysis

Aggregate logs by field value and plot as multiple series using AWS CloudWatch Insights

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Introduction

To plot multiple series in CloudWatch Logs Insights, your query needs two grouping dimensions: a time bucket and the field that should become the series split. If you only group by the field value, you get totals; if you group by time and field together, Insights can draw one line per field value.

The Basic Pattern

The usual query shape is:

text
fields @timestamp, myField
| stats count(*) by bin(5m), myField | sort bin(5m) asc ``` This does two important things: - '`bin(5m)` creates time buckets' - '`myField` creates separate series inside each time bucket' Without the time bucket, there is nothing to plot as a time series. ## Example: HTTP Status Codes as Separate Lines If your logs contain a parsed `statusCode` field, you can do: ```sql fields @timestamp, statusCode | stats count(*) as requests by bin(5m), statusCode | sort bin(5m) asc ``` In the visualization tab, choose a line or stacked area chart. CloudWatch Insights will plot one series for each distinct `statusCode`. That is the core answer to "aggregate by field value and plot as multiple series." ## Parse the Field First When Needed If the field is buried in raw message text, extract it first with `parse`. ```sql fields @timestamp, @message | parse @message /status=(?<statusCode>\d+)/ | stats count(*) as requests by bin(5m), statusCode | sort bin(5m) asc ``` Once the field exists in the query pipeline, the aggregation logic is the same. ## Example: Split by Service Name For application logs, a service or component name is often a better series key than status code: ```sql fields @timestamp, service | filter level = "ERROR" | stats count(*) as errors by bin(15m), service | sort bin(15m) asc ``` This produces one error-rate series per service across the selected time window. ## Keep Cardinality Under Control CloudWatch Insights visualizations become messy when the split field has too many distinct values. A field such as `requestId` is technically groupable, but it creates useless charts because every series has almost no repeated data. Good split fields are usually: - status code - service name - region - environment - endpoint group Bad split fields are usually high-cardinality identifiers such as unique IDs or user-specific tokens. ## Filter Before Aggregating Because Logs Insights charges by scanned data volume, filter early whenever possible. ```sql fields @timestamp, service, level | filter level = "ERROR" | stats count(*) as errors by bin(5m), service | sort bin(5m) asc ``` This is usually cheaper and faster than grouping the full dataset and filtering only later. ## Visualization Expectations CloudWatch Insights decides the chart from the result shape. To get multiple time-series lines, you generally want: - one numeric aggregate column - one binned timestamp column - one categorical split field If the result shape is wrong, the chart options will be limited or the graph will not look like you expect. ## Common Pitfalls - Grouping only by the field value and forgetting the time bucket. - Using a split field with huge cardinality, which makes the chart unreadable. - Forgetting to parse the field out of `@message` before aggregation. - Sorting by the category field instead of by the time bin. - Scanning too much data because filtering happens too late or not at all. ## Summary - To plot multiple series in Logs Insights, group by both `bin(...)` and the series field. - Use `stats ... by bin(5m), fieldName` as the standard pattern. - Parse the field first if it is embedded in raw log text. - Keep the split field low-cardinality so the chart stays useful. - Filter early to improve speed, reduce cost, and make the chart easier to read.

Course illustration
Course illustration

All Rights Reserved.