Introduction

The log message saying system:serviceaccount:default:default cannot get services means the pod identity lacks read permission in the target namespace. This usually appears when a workload relies on service discovery, but the default service account has not been granted explicit rights. The fix is not in application code, it is in Kubernetes RBAC policy.

Why This Permission Error Appears

Kubernetes authenticates the request as a service account and then authorizes it through Role Based Access Control rules. If no matching rule allows the get verb on the services resource in the namespace, the API server denies the request. The default service account is intentionally limited, so this is expected behavior in secure clusters.

A common source of confusion is namespace scope. A Role is namespaced and only grants permissions inside one namespace. A ClusterRole can span the cluster, but it is broader and should be used carefully. For this case, a namespace local Role and RoleBinding are usually enough.

You can confirm the effective permission with an impersonation check. This avoids guesswork and tells you exactly what the API server sees.

kubectl auth can-i get services   --as=system:serviceaccount:default:default   -n default

If the command prints no, RBAC is the direct cause. If it prints yes, look for a different issue such as wrong namespace, network policy, or a malformed client request.

Implement A Least Privilege RBAC Fix

Create a dedicated service account for the workload instead of expanding rights for the namespace default account. Then bind only the verbs and resources the workload needs.

1apiVersion: v1
2kind: ServiceAccount
3metadata:
4  name: service-reader
5  namespace: default
6---
7apiVersion: rbac.authorization.k8s.io/v1
8kind: Role
9metadata:
10  name: service-read-role
11  namespace: default
12rules:
13  - apiGroups: [""]
14    resources: ["services"]
15    verbs: ["get", "list", "watch"]
16---
17apiVersion: rbac.authorization.k8s.io/v1
18kind: RoleBinding
19metadata:
20  name: service-read-binding
21  namespace: default
22subjects:
23  - kind: ServiceAccount
24    name: service-reader
25    namespace: default
26roleRef:
27  apiGroup: rbac.authorization.k8s.io
28  kind: Role
29  name: service-read-role

Update the pod or deployment to use service-reader.

1apiVersion: apps/v1
2kind: Deployment
3metadata:
4  name: api
5  namespace: default
6spec:
7  replicas: 1
8  selector:
9    matchLabels:
10      app: api
11  template:
12    metadata:
13      labels:
14        app: api
15    spec:
16      serviceAccountName: service-reader
17      containers:
18        - name: api
19          image: nginx:stable

Apply the manifests and validate again with kubectl auth can-i. This pattern keeps permissions explicit, reviewable, and easy to audit.

Troubleshooting Workflow In Production

When the error appears in logs, follow a short sequence.

1. Identify the exact identity in the error message.
2. Verify the namespace where the API call is made.
3. Run kubectl auth can-i for that identity and verb.
4. Inspect current bindings with kubectl get rolebinding -n default.
5. Apply the minimal Role and RoleBinding.

This workflow prevents over granting permissions under pressure. It also helps teams document why each permission exists, which is important during security reviews.

Verification And Hardening Steps

After the fix, verify from inside the running pod, not only from an admin workstation. Execute a simple API read with the pod identity and confirm expected access while unrelated resources stay denied. This validates both authentication token mounting and effective RBAC policy.

You can also automate this check in deployment pipelines. For example, run kubectl auth can-i for a small matrix of required verbs and resources before promoting manifests. If a permission is missing, fail fast with a clear message. If a permission is broader than intended, flag it for review.

Finally, document ownership for each Role and RoleBinding. Teams often inherit old permissions that no longer map to active workloads. Regular cleanup keeps the namespace policy understandable and reduces accidental privilege growth over time.

Common Pitfalls

• Binding permissions to the default service account for convenience. This often leaks rights to unrelated pods.
• Creating a Role in one namespace while the workload runs in another namespace.
• Granting only list but not get, then seeing partial behavior.
• Using ClusterRoleBinding for a namespace only need.
• Assuming restart is optional after changing service account configuration on a deployment.

Summary

• The error is an RBAC authorization failure, not an application logic bug.
• Use kubectl auth can-i with impersonation to verify permissions quickly.
• Prefer dedicated service accounts and least privilege namespace roles.
• Bind only required verbs for required resources.
• Keep a repeatable troubleshooting sequence for faster incident response.