Grafana
Kubernetes
Prometheus
Metrics
Monitoring

Show metrics in Grafana from the Kubernetes Pod that was scraped last by Prometheus

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design

Introduction

If you want Grafana to show the metric series from the pod that Prometheus scraped most recently, the key PromQL concept is sample timestamp. Prometheus stores a timestamp for each sample, and you can inspect that with timestamp(metric). The subtle part is deciding whether you truly mean "last scraped pod" or whether you actually mean "newest pod" or "currently active pod," because those are not the same thing.

Use timestamp to Inspect Sample Freshness

PromQL can expose the timestamp of the latest sample for each time series.

promql
timestamp(http_requests_total{namespace="prod"})

This returns a value per matching series representing when Prometheus last stored a sample for that series. If you want the most recently scraped series, topk is the natural next step.

promql
topk(1, timestamp(http_requests_total{namespace="prod"}))

That identifies the matching series with the greatest timestamp.

Filter the Real Metric to That Series

If you want the actual metric value, not just its timestamp, you can use the timestamp-selected series as a filter.

promql
http_requests_total{namespace="prod"}
and on(namespace, pod)
topk(1, timestamp(http_requests_total{namespace="prod"}))

The and keeps the left-hand metric series only for the pod selected by the right-hand side. The label match keys in on(...) should reflect the labels that uniquely identify the pod in your setup.

This is the general pattern for "show me the current metric value from the most recently scraped pod."

Be Careful: "Last Scraped" Changes Constantly

Prometheus scrapes targets repeatedly, so the most recently scraped pod can change every scrape cycle. That means the selected series may bounce between pods simply because of scrape timing, not because something meaningful changed in the workload.

If the real question is one of these instead:

  • newest pod by creation time
  • currently ready pod
  • leader pod
  • pod with freshest application heartbeat

then timestamp(metric) may not be the right selection rule. In many dashboards, what people really want is not scrape recency but some domain-specific "current" pod.

Use Kubernetes Metadata if You Mean Newest Pod

If your actual intent is "show metrics for the newest pod," use Kubernetes metadata metrics such as kube_pod_created from kube-state-metrics rather than scrape timestamps.

promql
topk(1, kube_pod_created{namespace="prod", pod=~"my-app-.*"})

Then join that with the workload metric if needed. This distinction matters because scrape order is an artifact of monitoring timing, while creation time is a property of the workload itself.

Grafana Panels Need Stable Label Matching

In Grafana, this PromQL works best when your metric carries labels such as namespace and pod consistently. If the metric has additional dimensions like instance, container, or job, you may need to aggregate or match more carefully.

For example, if one pod exposes multiple series for the same metric, you may want to aggregate first:

promql
sum by (namespace, pod) (rate(http_requests_total{namespace="prod"}[5m]))
and on(namespace, pod)
topk(1, max by (namespace, pod) (timestamp(http_requests_total{namespace="prod"})))

This prevents the selection logic from being distorted by per-container or per-path label explosion.

Debug the Labels Before Building the Dashboard

A good workflow is:

  1. query the raw metric
  2. inspect which labels identify the pod
  3. query timestamp(metric)
  4. apply topk
  5. join back to the original metric

Trying to jump directly to a complex Grafana panel without understanding the label set usually leads to confusing results.

Common Pitfalls

  • Using scrape timestamp when the real intent is selecting the newest or currently active pod.
  • Joining on the wrong labels and getting no matches or too many matches.
  • Forgetting that scrape recency changes constantly and can make the panel unstable.
  • Applying the logic to a metric with many extra labels without aggregating first.
  • Assuming Prometheus scrape order has application meaning when it may just reflect monitoring timing.

Summary

  • Use timestamp(metric) to inspect when Prometheus last stored samples for each series.
  • Use topk(1, timestamp(metric)) to identify the most recently scraped series.
  • Join that selection back to the real metric with and on(...) when you want the metric value.
  • Be sure that "last scraped pod" is actually the thing you want to visualize.
  • If you really mean newest pod or active pod, use Kubernetes metadata or domain-specific health signals instead.

Course
Beginner
27 lessons
10 hours
System Design Fundamentals

Build a strong foundation in designing scalable, reliable distributed systems.

View the course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design