Introduction

Prometheus configuration is YAML, but it is important not to assume that means you automatically get full shell-style variable substitution everywhere in the file. Prometheus does not behave like a generic template engine for the entire configuration file.

In practice, teams usually solve "variable substitution" in one of two ways: they preprocess the YAML before starting Prometheus, or they rely on the limited places where Prometheus itself supports environment-variable references. That distinction matters because many examples online imply broader native support than people actually get.

Know What Prometheus Does Natively

Prometheus configuration docs define the file format and also note that environment variable references can be replaced in certain contexts. One documented example is external_labels, where ${VAR} and $VAR references are expanded from the current environment.

For example:

1global:
2  external_labels:
3    cluster: ${CLUSTER_NAME}
4    region: $REGION

If the variables are present in the Prometheus process environment, Prometheus can expand them there. If they are missing, they are replaced with empty strings.

That is not the same as saying "every field in the YAML supports arbitrary variable substitution." The safe mindset is to treat native substitution as limited and field-specific unless the documentation says otherwise.

Preprocess the Config When You Need General Substitution

If you want broad variable substitution across a Prometheus configuration file, the common operational solution is to render the final YAML before Prometheus reads it. A shell-based example using envsubst looks like this:

export SCRAPE_TARGET="node-exporter:9100"
envsubst < prometheus.yml.tmpl > prometheus.yml

Template file:

1scrape_configs:
2  - job_name: "node"
3    static_configs:
4      - targets: ["${SCRAPE_TARGET}"]

Rendered output:

1scrape_configs:
2  - job_name: "node"
3    static_configs:
4      - targets: ["node-exporter:9100"]

This approach is simple, explicit, and works well in Docker entrypoints, Kubernetes init flows, and deployment pipelines.

A typical container entrypoint looks like this:

1#!/usr/bin/env sh
2set -eu
3
4envsubst < /etc/prometheus/prometheus.yml.tmpl > /etc/prometheus/prometheus.yml
5exec prometheus --config.file=/etc/prometheus/prometheus.yml

That makes the substitution step visible instead of relying on undocumented behavior.

Be Clear About Where Substitution Happens

When teams say "Prometheus variable substitution," they often mean one of three different things:

• environment expansion in a documented config field
• preprocessing the YAML before Prometheus starts
• PromQL dashboard variables in Grafana, which are a different feature entirely

Keeping those concepts separate prevents a lot of confusion. Prometheus server configuration, PromQL, and dashboard templating are related parts of the ecosystem, but they are not the same substitution mechanism.

Common Pitfalls

The biggest mistake is assuming Prometheus acts like a full YAML templating engine for every configuration field.

Another common issue is relying on shell-style placeholders in the config file without actually running a preprocessing step before startup.

It is also easy to confuse Grafana dashboard variables with Prometheus server config substitution. They solve different problems in different layers.

Finally, if you preprocess configuration, make sure the rendered file is what Prometheus actually loads. A template that looks correct is useless if the runtime still points at the unrendered source file.

It is also worth validating the rendered YAML before rollout so missing environment variables do not quietly turn into empty labels or broken target values.

Summary

• Prometheus YAML does not behave like a universal template engine.
• Some documented fields support environment-variable references, but native substitution is not a blanket rule for the entire file.
• For broad substitution, render the config before starting Prometheus.
• Keep server config substitution separate from Grafana-style dashboard variables.
• Always verify which file Prometheus actually loads at runtime.