Introduction

If a container needs to run gsutil inside GKE, the clean solution is to let the pod inherit Google Cloud credentials through workload identity instead of baking service-account key files into the image. The mechanics are straightforward once you separate the pieces: Kubernetes service account, Google Cloud permissions, and a pod that uses the bound identity. The important point is that gsutil should discover credentials through application default credentials, not through manual gcloud auth login inside the container.

What gsutil Needs Inside the Container

gsutil only needs valid Google Cloud credentials with the right Storage permissions. In a GKE workload-identity setup, the pod obtains those credentials from the cluster and metadata path rather than from a JSON key mounted in the filesystem.

That gives you three practical benefits:

1. No long-lived key file in the image.
2. Permissions can be narrowed to one workload.
3. Rotating identity is handled by the platform rather than by secret replacement.

From the container's perspective, the goal is simply that this command works:

gsutil ls gs://my-example-bucket

Bind Kubernetes Identity to Google Cloud Permissions

The common pattern is to let a Kubernetes service account represent the pod and then allow it to act as a Google Cloud service account that has the needed Storage role.

Create the Google Cloud service account:

gcloud iam service-accounts create gsutil-runner \
  --project=my-project-id

Grant it the Storage access your workload actually needs:

gcloud projects add-iam-policy-binding my-project-id \
  --member="serviceAccount:[email protected]" \
  --role="roles/storage.objectViewer"

Then allow the Kubernetes service account to use that identity:

1gcloud iam service-accounts add-iam-policy-binding \
2  [email protected] \
3  --role="roles/iam.workloadIdentityUser" \
4  --member="serviceAccount:my-project-id.svc.id.goog[ml/gsutil-ksa]"

The exact project, namespace, and account names vary, but the pattern stays the same.

Create and Annotate the Kubernetes Service Account

Create the Kubernetes service account in the namespace where the pod will run:

1apiVersion: v1
2kind: ServiceAccount
3metadata:
4  name: gsutil-ksa
5  namespace: ml
6  annotations:
7    iam.gke.io/gcp-service-account: gsutil-runner@my-project-id.iam.gserviceaccount.com

Apply it with:

kubectl apply -f service-account.yaml

This tells GKE which Google Cloud identity should back the Kubernetes service account used by the pod.

Run the Container with That Service Account

The pod spec must explicitly use the Kubernetes service account. Otherwise the workload runs as the default account and the binding you created is never used.

1apiVersion: v1
2kind: Pod
3metadata:
4  name: gsutil-test
5  namespace: ml
6spec:
7  serviceAccountName: gsutil-ksa
8  containers:
9    - name: runner
10      image: google/cloud-sdk:slim
11      command: ["sh", "-c", "gsutil ls gs://my-example-bucket && sleep 3600"]

Once the pod starts, gsutil should resolve credentials through application default credentials automatically. No interactive login should be required.

Verify the Identity Before Blaming gsutil

If authentication fails, check the identity chain step by step instead of treating gsutil as the root problem.

Useful checks inside the pod:

gcloud auth list
gsutil ls gs://my-example-bucket

Useful checks outside the pod:

1. Confirm the pod is using the intended Kubernetes service account.
2. Confirm the Kubernetes service account annotation is correct.
3. Confirm the Google Cloud service account has the required Storage role.
4. Confirm the workload-identity binding uses the right namespace and service-account name.

Most failures come from a mismatch in one of those bindings, not from the gsutil binary itself.

Avoid Key Files in the Image

The anti-pattern for this setup is copying a service-account JSON key into the container or mounting it through a generic secret just to make gsutil work. That approach is operationally heavier and materially weaker from a security standpoint. Workload identity exists so the workload can prove who it is at runtime without carrying permanent credential files around.

If a container still depends on GOOGLE_APPLICATION_CREDENTIALS pointing to a key file, the workload identity migration is not actually complete.

Keep Permissions Narrow

The workload usually does not need broad Storage admin access. If it only downloads model artifacts, a read-only storage role is enough. If it uploads outputs, scope the write permissions only to what the job actually needs. This matters even more for ML workloads that may be cloned, retried, or run at scale across many pods.

Common Pitfalls

• Running the pod under the default Kubernetes service account instead of the workload-specific one you configured.
• Granting Storage permissions to the wrong Google Cloud service account and then debugging gsutil instead of IAM.
• Forgetting the Kubernetes-service-account annotation that links it to the Google Cloud identity.
• Falling back to JSON key files in the container image even though workload identity was the intended design.
• Treating gsutil as interactive tooling and trying to use manual login flows inside an automated container.

Summary

• 'gsutil in a GKE container should authenticate through application default credentials provided by workload identity.'
• Bind a Kubernetes service account to a Google Cloud service account with the needed Storage permissions.
• Run the pod with that Kubernetes service account explicitly.
• Verify the identity chain step by step when authentication fails.
• Avoid service-account key files in images or mounted secrets unless there is a hard migration blocker.