Prometheus not scraping additional scrapes
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.
Introduction
When Prometheus is "not scraping additional scrapes," the problem is rarely the scrape loop itself. In most cases, the extra job was never loaded, the target was dropped by relabeling, the target is down, or a config reload failed and Prometheus kept using the old configuration.
Start By Checking What Prometheus Actually Loaded
Do not debug YAML in the abstract. First confirm whether Prometheus is using the scrape config you think it is using.
Useful places to check are:
- the
/targetspage - the
/configpage - Prometheus logs
- the result of
promtool check config
A minimal scrape job looks like this:
If your new job is missing from /config, the file was not reloaded or the wrong file was edited.
Reload Problems Are Common
Prometheus does not magically reread the configuration unless you restart it, send SIGHUP, or use the reload endpoint when lifecycle reload is enabled.
If the new config is invalid, Prometheus keeps the previous working configuration. That leads many teams to believe the "additional scrapes" were accepted when they were actually ignored.
Also verify that the process is using the file you edited. The active config path comes from the --config.file flag.
Target Discovery May Succeed But Relabeling May Drop It
Sometimes the target is discovered and then removed by relabeling rules.
That configuration drops app:8080 because it only keeps targets ending in 9090.
If a target never appears in /targets, inspect relabel_configs and metric_relabel_configs separately. relabel_configs can remove the scrape target itself; metric_relabel_configs only affect scraped samples after the target has already been reached.
Timeouts, Intervals, And Bad Endpoints
A scrape job can exist and still collect nothing if the endpoint is not reachable or takes too long.
Practical checks:
If the endpoint hangs, returns an HTML error page, or requires authentication that Prometheus does not send, the job will show failures even though the configuration is present.
Also remember that scrape_timeout cannot exceed scrape_interval. Invalid combinations will fail validation.
Extra Config Files Need To Be Referenced Correctly
Some setups split scrape jobs across multiple files. In that case, changing an included file is not enough unless the main config references it correctly and reload behavior covers referenced files.
For example:
Notice that rule_files and scrape_configs are different concepts. Prometheus does not load extra scrape jobs from rule_files. If you are using a deployment-specific pattern such as mounted fragments or operator-managed additionalScrapeConfigs, verify that the final generated Prometheus configuration contains the expected job.
Kubernetes And Operator Setups Add Another Layer
In Kubernetes, teams often edit a Secret or Helm value and assume Prometheus picked it up. In operator-based setups, the real questions are:
- did the Secret update correctly
- did the Prometheus resource reference it correctly
- did the operator regenerate configuration
- did Prometheus reload the generated config
So if you use a Prometheus Operator or chart wrapper, debug both the generated Prometheus config and the higher-level Kubernetes objects.
Common Pitfalls
The most common mistake is editing a config file and forgetting to reload Prometheus.
Another common problem is validating only YAML syntax, not the actual scrape behavior. A syntactically correct job can still point to a dead endpoint.
Teams also confuse relabeling with scraping and accidentally drop targets before they ever appear.
Finally, in Kubernetes environments, the source of truth may be a Secret, ConfigMap, or custom resource rather than the file you are inspecting on disk.
Summary
- Check
/configand/targetsbefore guessing. - Validate config with
promtooland reload Prometheus explicitly. - Inspect relabel rules if targets disappear.
- Test the metrics endpoint directly with
curl. - In Kubernetes, verify the generated config path, not only the higher-level manifest.
Related reading
- Prometheus Pods restart in grafana
- Promote secondary to primary from secondary node
- Pros and Cons of Amazon SageMaker VS. Amazon EMR, for deploying TensorFlow-based deep learning models?
- Pros and cons on utilizing Azure Service Fabric vs Custom Azure Cloud?
- Propagate Sleuth baggage on parallel streams
- Proper way of handling exception in task continuewith
- Provision multiple logical databases with Terraform on AWS RDS cluster instance
- Psycopg2 on Amazon Elastic Beanstalk

System Design Fundamentals
Build a strong foundation in designing scalable, reliable distributed systems.
View the courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.