Introduction
Finding pods running on nodes with a specific label is a common Kubernetes troubleshooting and operations task. You usually need this during maintenance windows, targeted rollouts, or incident triage. The key is to combine node label filtering with pod scheduling data.
Step 1: Identify Nodes with the Target Label
First list matching nodes. Assume the label key is nodepool and target value is gpu.
kubectl get nodes -l nodepool=gpu
For scripting, extract only names.
kubectl get nodes -l nodepool=gpu -o jsonpath='{.items[*].metadata.name}'
This gives the node set you care about.
Step 2: Query Pods by Scheduled Node Name
Pods record assigned node in spec.nodeName. For each node, list pods across namespaces.
1for node in $(kubectl get nodes -l nodepool=gpu -o jsonpath='{.items[*].metadata.name}'); do
2 echo "Node: $node"
3 kubectl get pods -A --field-selector spec.nodeName=$node
4 echo
5 done
This is direct and works without additional tools.
Single Command Using jq
If you want one consolidated command with machine-friendly output, use JSON plus jq.
kubectl get pods -A -o json | jq -r '
.items[]
| [.metadata.namespace, .metadata.name, .spec.nodeName] | @tsv ' ``` Combine with a node allow-list built from label selection if needed. ```bash nodes=$(kubectl get nodes -l nodepool=gpu -o json | jq -r '.items[].metadata.name') kubectl get pods -A -o json | jq -r --argjson arr "$(printf '%s ' "$nodes" | jq -R . | jq -s .)" ' .items[] | select(.spec.nodeName as $n | $arr | index($n)) | [.metadata.namespace, .metadata.name, .spec.nodeName] | @tsv ' ``` ## Include Pod Readiness and Owner Data For operations, raw pod names are not always enough. Add status and owner context. ```bash kubectl get pods -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,NODE:.spec.nodeName,PHASE:.status.phase,OWNER:.metadata.ownerReferences[0].kind' --sort-by=.spec.nodeName ``` Then filter results for selected nodes with shell tools. ```bash kubectl get pods -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,NODE:.spec.nodeName,PHASE:.status.phase' --no-headers | awk '$3 ~ /gpu-node-prefix/' ``` ## Production Use Cases Common real-world uses: * Verify workload isolation to specific hardware pools. * Audit whether critical services avoid tainted nodes. * Prepare drain plans by listing affected pods beforehand. * Confirm scheduler behavior after node label updates. Automating this check in runbooks reduces incident response time. ## Namespace and Workload Focused Queries In large clusters, cluster-wide listings can be noisy. Narrow by namespace or workload owner when running targeted checks. ```bash # only one namespace kubectl get pods -n payments --field-selector spec.nodeName=gpu-node-1 # include owner references for controller mapping kubectl get pods -n payments -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName,OWNER:.metadata.ownerReferences[0].name' ``` This helps when auditing only one application domain during incident response. ## Automated Script for Reuse You can wrap label to pod mapping into a reusable shell script. ```bash #!/usr/bin/env bash set -euo pipefail label_key=${1:-nodepool} label_value=${2:-gpu} nodes=$(kubectl get nodes -l "${label_key}=${label_value}" -o jsonpath='{.items[*].metadata.name}') for node in $nodes; do echo "=== ${node} ===" kubectl get pods -A --field-selector "spec.nodeName=${node}" --no-headers echo done ``` A script like this is useful in runbooks and maintenance playbooks. ## Scheduling and Label Lifecycle Notes Changing labels affects future scheduling, but existing pods usually stay where they are. If you need movement, trigger rollout restart or drain workflows explicitly. Knowing this prevents confusion when query output seems to ignore new labels right after updates. ## Common Pitfalls * Filtering nodes by label but forgetting pods can be pending with no `spec.nodeName`. * Querying one namespace only and missing cluster-wide workloads. * Assuming label changes move existing pods automatically. * Running manual filters that hide daemonset pods unintentionally. ## Summary * Start by selecting nodes with the target label. * Use `spec.nodeName` to map pods to those nodes. * Prefer scriptable output for operations and audits. * Include namespace and status fields for actionable results.
