Introduction

When the Cloud SQL Auth Proxy on GKE reports a "cannot fetch token" error, the proxy usually is not failing at database connectivity yet. It is failing earlier, at Google authentication. In practical terms, the proxy cannot obtain an access token from the credential source available to the pod. On modern GKE, that usually means a Workload Identity configuration problem, a missing IAM permission, or a credential source that the pod cannot actually reach.

What the Proxy Is Trying to Do

The Cloud SQL Auth Proxy needs Google credentials before it can open the database connection. On GKE, the common patterns are:

• Workload Identity Federation for GKE
• a mounted service account key file, though this is less preferred

If the proxy cannot turn those credentials into an OAuth access token, it stops before database traffic even starts.

That is why this error is fundamentally an auth-path problem, not a SQL problem.

The Preferred Setup: Workload Identity

On GKE, the recommended approach is usually Workload Identity. The basic chain is:

1. a Kubernetes service account is used by the pod
2. that service account is mapped to a Google service account
3. the Google service account has Cloud SQL access

The Google service account needs at least the Cloud SQL client role:

gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="serviceAccount:GSA_NAME@PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/cloudsql.client"

Then allow the Kubernetes service account to impersonate that Google service account:

1gcloud iam service-accounts add-iam-policy-binding \
2  GSA_NAME@PROJECT_ID.iam.gserviceaccount.com \
3  --role="roles/iam.workloadIdentityUser" \
4  --member="serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/app-ksa]"

And annotate the Kubernetes service account:

kubectl annotate serviceaccount app-ksa \
  --namespace NAMESPACE \
  iam.gke.io/gcp-service-account=GSA_NAME@PROJECT_ID.iam.gserviceaccount.com

If one of those links is missing, token retrieval fails.

Make Sure the Pod Actually Uses That Service Account

The workload identity binding is useless if the pod runs under the wrong Kubernetes service account.

Your workload spec must reference it:

1apiVersion: apps/v1
2kind: Deployment
3metadata:
4  name: app
5spec:
6  replicas: 1
7  selector:
8    matchLabels:
9      app: app
10  template:
11    metadata:
12      labels:
13        app: app
14    spec:
15      serviceAccountName: app-ksa
16      containers:
17        - name: cloud-sql-proxy
18          image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.18.3
19          args:
20            - "--structured-logs"
21            - "--port=5432"
22            - "PROJECT_ID:REGION:INSTANCE_NAME"

If serviceAccountName is missing, the pod may use the default Kubernetes service account instead, and the proxy will not get the expected credentials.

If You Use a Key File Instead

Some clusters still mount a Google service account JSON key into the pod. If that is your setup, verify all of these:

• the secret is mounted correctly
• 'GOOGLE_APPLICATION_CREDENTIALS points to the mounted file'
• the key belongs to a service account with Cloud SQL access

Example:

1env:
2  - name: GOOGLE_APPLICATION_CREDENTIALS
3    value: /secrets/gcp/key.json
4volumeMounts:
5  - name: gcp-key
6    mountPath: /secrets/gcp
7    readOnly: true

If the file path is wrong or the secret is not mounted, the proxy cannot fetch a token because it has no usable credentials source.

Network and Metadata Access Still Matter

Even with correct IAM, the pod still needs to reach the services involved in token exchange. Depending on your setup, problems can come from:

• blocked egress
• misconfigured private networking
• missing access to the metadata or token exchange path used by Workload Identity

This is why "cannot fetch token" sometimes appears even when IAM roles look correct on paper. The identity path must be both authorized and reachable.

If the cluster is heavily locked down, check whether the pod can access the required Google identity endpoints for its auth flow.

Read the Error Message Carefully

The exact wording often points to the broken link:

• no credentials found
• permission denied
• metadata server unavailable
• token exchange failure

Those are different failures. Do not jump straight to generic Cloud SQL troubleshooting before understanding whether the proxy lacks credentials, lacks permission, or lacks network reachability to obtain a token.

A Practical Troubleshooting Order

Use this order:

1. confirm the pod's serviceAccountName
2. confirm the Kubernetes service account annotation
3. confirm the IAM workloadIdentityUser binding
4. confirm the Google service account has roles/cloudsql.client
5. confirm the pod can reach the auth path needed to obtain a token

That sequence usually finds the problem faster than checking database settings first.

Common Pitfalls

The biggest mistake is granting roles/cloudsql.client and assuming that is enough. On Workload Identity, the impersonation link between the Kubernetes service account and the Google service account also has to exist.

Another issue is forgetting to set serviceAccountName on the pod, which causes the workload to run as the default Kubernetes service account instead of the intended one.

Developers also sometimes mount a key file but misconfigure GOOGLE_APPLICATION_CREDENTIALS, leaving the proxy with no readable credential source.

Finally, do not treat every token error as an IAM-only issue. Network restrictions and metadata access can also block token retrieval.

Summary

• "Cannot fetch token" usually means the Cloud SQL proxy cannot obtain Google credentials, not that the database itself rejected the connection.
• On GKE, Workload Identity is the preferred path and must be wired correctly end to end.
• Check the Kubernetes service account, its annotation, the IAM impersonation binding, and the Google service account role.
• If using a key file, verify the mount path and GOOGLE_APPLICATION_CREDENTIALS.
• Confirm that the pod can actually reach the identity path needed to exchange credentials for a token.