Introduction

Prometheus outside a Kubernetes cluster cannot directly scrape metrics-server in most setups because metrics-server is designed primarily for Kubernetes APIs, not general Prometheus exposition. A practical solution is exposing Kubernetes resource metrics through a Prometheus-compatible endpoint using an adapter or exporter pattern. Security and network access design are as important as query correctness.

Understand Metrics Sources First

metrics-server feeds the Kubernetes metrics API used by autoscaling and kubectl top. It is not a long-term time-series store and does not always expose a scrape-friendly endpoint for external Prometheus.

Prometheus expects HTTP endpoints returning the Prometheus text format.

Recommended Architecture

Common external-scrape pattern:

1. inside cluster component fetches Kubernetes metrics API
2. component exposes Prometheus-formatted metrics
3. external Prometheus scrapes exposed endpoint

You can use kube-state-metrics and node-exporter for many resource metrics, and add custom exporters only for gaps.

Example Service Exposure

Expose exporter service through controlled endpoint.

1apiVersion: v1
2kind: Service
3metadata:
4  name: metrics-exporter
5  namespace: monitoring
6spec:
7  type: NodePort
8  selector:
9    app: metrics-exporter
10  ports:
11    - port: 8080
12      targetPort: 8080
13      nodePort: 30080

For production, prefer Ingress or private load balancer with TLS and auth rather than raw NodePort.

External Prometheus Scrape Config

1scrape_configs:
2  - job_name: kubernetes-metrics-exporter
3    metrics_path: /metrics
4    static_configs:
5      - targets:
6          - 10.0.0.25:30080

Validate target reachability and firewall rules from Prometheus host.

Authentication and RBAC

Exporter inside cluster often needs permissions to read metrics APIs. Grant minimal RBAC scope. Avoid cluster-admin defaults for convenience.

Also protect external endpoint with network policy, mTLS, and allow-listing where possible.

Troubleshooting Checklist

• verify exporter pod running
• verify service endpoint reachable
• verify Prometheus target status page
• inspect scrape errors and HTTP status
• confirm RBAC allows metrics reads

A staged checklist is faster than editing many components simultaneously.

Prefer Native Prometheus Stack Where Possible

For full Kubernetes monitoring, Prometheus typically runs inside cluster and remote-writes to external systems. External-only scrape designs can work, but they add network complexity and authentication overhead. Pick architecture based on operational constraints and compliance requirements.

Example Exporter Deployment Skeleton

A simple deployment can expose transformed metrics inside the cluster.

1apiVersion: apps/v1
2kind: Deployment
3metadata:
4  name: metrics-exporter
5  namespace: monitoring
6spec:
7  replicas: 1
8  selector:
9    matchLabels:
10      app: metrics-exporter
11  template:
12    metadata:
13      labels:
14        app: metrics-exporter
15    spec:
16      containers:
17        - name: exporter
18          image: example/metrics-exporter:latest
19          ports:
20            - containerPort: 8080

Keep image and RBAC policy aligned with your security baseline.

Network Topology Options

For external Prometheus, common options include private load balancer, VPN-linked service discovery, or pull-through gateway in DMZ networks. Choose topology that matches compliance requirements and avoids public exposure of internal metrics endpoints.

Alerting Validation

After scrape is successful, add a test alert rule on exporter up-metric to confirm end-to-end visibility from collection to alerting pipeline. Monitoring without alert validation can fail silently during incidents.

TLS and Certificate Management

If external Prometheus scrapes over HTTPS, automate certificate rotation and validate trust chains. Expired certificates are a frequent cause of sudden scrape failures in otherwise healthy monitoring stacks.

Capacity Planning

Exporter endpoints should be sized for scrape interval and target cardinality. Under-provisioned exporters can become bottlenecks and produce intermittent timeout errors.

Common Pitfalls

• Assuming metrics-server is directly scrape-ready for external Prometheus.
• Exposing metrics endpoints publicly without authentication.
• Skipping RBAC hardening for exporter service accounts.
• Using NodePort in production without network controls.
• Debugging scrape failures without checking Prometheus target status first.

Summary

• Metrics-server is not usually a direct external Prometheus scrape target.
• Use exporter or adapter layers to expose Prometheus-formatted metrics.
• Secure endpoint access with RBAC, network controls, and TLS.
• Validate reachability and scrape status step by step.
• Prefer architecture that balances observability, security, and operational simplicity.