0%
AWS in Practice: Services, Limits, and Failure Modes
Account Foundations
Compute
Storage
Databases
Networking
Limits, Cost and Recovery
CloudWatch and X-Ray
CloudWatch is where most AWS operational data lands: metrics published by services, logs from functions and containers, alarms that page engineers, and dashboards that summarise health. X-Ray, and the OpenTelemetry based tooling around it, adds traces that follow a request across services. Used well, these tools answer what is broken, where and why within minutes. Used carelessly, they produce alarms that never fire or always fire, logs that cost more than the service they describe, and metrics that cannot answer the question asked during an incident. This lesson covers how each part works and where teams usually go wrong.
Namespaces, metrics and dimensions
A CloudWatch metric is identified by a namespace, a name and a set of dimensions. AWS/ApplicationELB with HTTPCode_Target_5XX_Count and the dimension LoadBalancer is one metric; adding TargetGroup as a dimension makes a different one. Every unique combination of dimension values is a separate metric that CloudWatch stores and bills. AWS services publish their metrics for free at basic resolution, while custom metrics that applications publish cost per metric per month, which makes the choice of dimensions a cost decision as much as a design one.
Dimensions are also how a metric is found later. CloudWatch does not let a query wildcard across dimension values the way a log search can, so an alarm or graph must name the exact combination it wants, or use a search expression. Teams that publish the same metric with dimensions in different orders, or with a dimension present on some services and missing on others, end up with dashboards that silently show part of the fleet. Agreeing on a short naming standard for namespaces, metric names and dimension keys, and checking it in code review, saves a great deal of confusion during the first serious incident.
Statistics and percentiles
When graphing or alarming, CloudWatch aggregates the data points in each period with a statistic: Average, Sum, Minimum, Maximum, SampleCount, or a percentile such as p99. Averages hide tail latency, because a service where 98 percent of requests take 50 milliseconds and 2 percent take five seconds has an average under 150 milliseconds, which looks healthy while one user in fifty waits five seconds. Percentiles show the experience of slow requests directly, and alarms on p99 or p99.9 latency catch the problems that users feel. Sum is the right statistic for counts, such as errors per minute.
Percentiles have one important limit: they cannot be averaged. The p99 of a fleet is not the average of each instance's p99, and a one hour p99 is not the average of sixty one minute values. CloudWatch computes percentiles correctly when it has the raw distribution, which it keeps for metrics published with values and counts or through EMF, but a percentile copied into another metric as a plain number loses that property. When a dashboard needs fleet wide latency, publish raw latency to one metric with a service dimension and let CloudWatch compute the percentile, rather than aggregating precomputed numbers.
Resolution and retention
Standard resolution metrics have one minute granularity; high resolution custom metrics can be published every second, at higher cost, and alarms on them can evaluate every ten seconds. CloudWatch keeps one minute data for fifteen days, five minute data for sixty-three days and one hour data for fifteen months, aggregating older data automatically. That means a precise graph of last week is available, but last quarter's detail is not, so teams that need long term high resolution data for capacity planning export metrics to another store through metric streams.
Period choice interacts with retention. An alarm or graph at one minute period over a range older than fifteen days quietly falls back to five minute data, so a short spike that was obvious at the time becomes a gentle bump. Engineers comparing this week's incident with one from two months ago should expect the older graph to look smoother, and should not conclude the earlier event was milder. For incidents worth comparing later, saving a snapshot of the graph or the raw query output into the review document preserves the detail that CloudWatch itself will eventually aggregate away.
Publishing custom metrics
Applications can publish custom metrics with the PutMetricData API, but calling it on every request is slow and expensive. The Embedded Metric Format is a better default: the application writes a structured JSON log line that includes metric definitions, and CloudWatch extracts the metrics from the log asynchronously. Lambda functions and containers can use EMF with no extra network calls, and the log line also keeps high cardinality context, such as the request ID, which would be far too expensive as a metric dimension but is useful when investigating.
Cardinality and cost
Because every dimension combination is a separate metric, a dimension with many values multiplies cost. A latency metric with dimensions for endpoint, status code and customer ID, across ten endpoints, five status codes and twenty thousand customers, creates up to a million metrics, which can cost far more than the service it measures. The rule is to use dimensions for the small, bounded sets that alarms and dashboards actually group by, such as service, endpoint and status class, and to put unbounded values like customer or request ID in logs and traces instead.
The cardinality problem appears gradually. A team adds a tenant dimension when there are thirty tenants, which is fine, and two years later the product has thirty thousand and the observability bill has quietly grown into one of the largest line items. Reviewing the number of custom metrics per namespace monthly, and alarming when it jumps, catches this before the invoice does. Where per tenant views are genuinely needed, Contributor Insights can rank the top contributors from logs without a metric per tenant, which answers the usual question of who is causing the load at far lower cost.
Metric math and derived signals
Metric math combines metrics into new series without publishing anything extra. An error rate is errors divided by requests; availability is one minus that; the share of throttled requests is throttles divided by invocations. Alarms can be set on these expressions, which matters because raw counts mislead: ten errors a minute is alarming at a hundred requests and irrelevant at a million. Search expressions can also aggregate across many metrics, such as the sum of errors for every function in a service, so dashboards stay correct as functions are added.
What services publish for free
Before adding custom metrics, it is worth knowing what services already publish. Load balancers report request counts, latency and status codes per target group. Lambda reports invocations, errors, throttles, duration and concurrent executions. DynamoDB reports consumed capacity and throttled requests per table and index. SQS reports queue depth and the age of the oldest message. These metrics cover much of what a service owner needs, and missing ones, such as business events or per endpoint latency inside an application, are the ones worth adding as custom metrics.
A practical starting set for a new service is short. Request count, error count and latency per endpoint and status class from the load balancer or API layer; queue depth and oldest message age for every queue it owns; throttles and concurrency for its functions; and two or three business metrics that show whether the service is doing its job, such as orders placed or messages delivered. That set, perhaps twenty or thirty metrics, supports almost every alarm and dashboard the team will need in its first year, and anything added later should answer a question this set cannot.
Use percentiles for latency and Sum for counts, publish custom metrics through the Embedded Metric Format, keep dimensions to small bounded sets, put customer and request IDs in logs and traces, and derive rates with metric math rather than alarming on raw counts.