GitLab-CI
Kubernetes
CI/CD
Environment Variables
Troubleshooting

GitLab-CI Kubernetes Variables aren't set?

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design

Introduction

When GitLab CI/CD variables are not available in Kubernetes deployments, the most common causes are: the variable is not exported to the Kubernetes runner environment, envsubst or variable substitution is not applied to manifest files, or the variable scope is restricted to a specific environment or branch. GitLab CI variables are available as environment variables in the runner shell, but they do not automatically propagate into Kubernetes manifests. You must explicitly inject them using envsubst, Helm values, or Kubernetes secrets.

How GitLab CI Variables Work

yaml
1# .gitlab-ci.yml
2variables:
3  APP_VERSION: "1.2.3"
4  DATABASE_HOST: "db.example.com"
5
6deploy:
7  stage: deploy
8  script:
9    - echo $APP_VERSION          # Works — available in shell
10    - kubectl apply -f deploy.yaml  # Variables NOT substituted in YAML

GitLab CI variables are environment variables in the runner's shell. They are not automatically substituted inside files referenced by kubectl apply. The Kubernetes API receives the literal text $APP_VERSION, not the value.

Fix 1: Use envsubst

yaml
1# deploy.yaml (template with variable placeholders)
2apiVersion: apps/v1
3kind: Deployment
4metadata:
5  name: my-app
6spec:
7  template:
8    spec:
9      containers:
10        - name: app
11          image: registry.example.com/app:${APP_VERSION}
12          env:
13            - name: DATABASE_HOST
14              value: "${DATABASE_HOST}"
yaml
1# .gitlab-ci.yml
2deploy:
3  stage: deploy
4  script:
5    - envsubst < deploy.yaml | kubectl apply -f -

envsubst replaces ${VAR} placeholders with environment variable values before passing the manifest to kubectl.

Fix 2: Use sed for Targeted Replacement

yaml
1# .gitlab-ci.yml
2deploy:
3  stage: deploy
4  script:
5    - sed -i "s|__APP_VERSION__|${APP_VERSION}|g" deploy.yaml
6    - sed -i "s|__DATABASE_HOST__|${DATABASE_HOST}|g" deploy.yaml
7    - kubectl apply -f deploy.yaml

Fix 3: Use Helm Values

yaml
1# .gitlab-ci.yml
2deploy:
3  stage: deploy
4  script:
5    - helm upgrade --install my-app ./chart
6        --set image.tag=${APP_VERSION}
7        --set database.host=${DATABASE_HOST}
yaml
1# chart/values.yaml
2image:
3  repository: registry.example.com/app
4  tag: latest
5database:
6  host: localhost
yaml
1# chart/templates/deployment.yaml
2containers:
3  - name: app
4    image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
5    env:
6      - name: DATABASE_HOST
7        value: "{{ .Values.database.host }}"

Fix 4: Create Kubernetes Secrets from CI Variables

yaml
1# .gitlab-ci.yml
2deploy:
3  stage: deploy
4  script:
5    # Create or update a Kubernetes secret from CI variables
6    - kubectl create secret generic app-secrets
7        --from-literal=db-password="${DB_PASSWORD}"
8        --from-literal=api-key="${API_KEY}"
9        --dry-run=client -o yaml | kubectl apply -f -
10    - kubectl apply -f deploy.yaml
yaml
1# deploy.yaml
2spec:
3  containers:
4    - name: app
5      envFrom:
6        - secretRef:
7            name: app-secrets

Variable Scope Issues

yaml
1# .gitlab-ci.yml
2variables:
3  GLOBAL_VAR: "available everywhere"
4
5deploy-staging:
6  stage: deploy
7  variables:
8    STAGE_VAR: "only in this job"
9  environment:
10    name: staging
11  script:
12    - echo $GLOBAL_VAR   # Works
13    - echo $STAGE_VAR    # Works
14    - echo $PROD_SECRET  # Empty — scoped to production
15
16deploy-production:
17  stage: deploy
18  environment:
19    name: production
20  script:
21    - echo $PROD_SECRET  # Works — scoped to this environment

In GitLab Settings > CI/CD > Variables, each variable can be scoped to a specific environment (staging, production) or branch (main, develop). If a variable appears empty, check its scope settings.

Debugging Missing Variables

yaml
1# .gitlab-ci.yml
2debug-vars:
3  stage: test
4  script:
5    # List all available environment variables (mask sensitive ones)
6    - env | grep -E "^(CI_|APP_|DB_)" | sort
7
8    # Check if a specific variable is set
9    - |
10      if [ -z "$APP_VERSION" ]; then
11        echo "ERROR: APP_VERSION is not set"
12        exit 1
13      fi
14
15    # Verify envsubst works
16    - echo 'Image: ${APP_VERSION}' | envsubst
17
18    # Check kubectl context
19    - kubectl config current-context
20    - kubectl get namespace

Protected and Masked Variables

yaml
1# Variables marked as "protected" in GitLab UI are only available
2# on protected branches (main, release/*)
3# Variables marked as "masked" are hidden in job logs
4
5# Check if running on a protected branch
6deploy:
7  script:
8    - echo "Branch: $CI_COMMIT_BRANCH"
9    - echo "Protected: $CI_COMMIT_REF_PROTECTED"
10    # If CI_COMMIT_REF_PROTECTED is "false", protected variables are empty

Common Pitfalls

  • Expecting kubectl manifests to auto-substitute variables: kubectl apply -f deploy.yaml reads the file literally. ${APP_VERSION} in the YAML is treated as a string, not expanded. Use envsubst < deploy.yaml | kubectl apply -f - to substitute variables before applying.
  • Variable scoped to wrong environment or branch: GitLab CI/CD variables can be restricted to specific environments (production, staging) or protected branches. A variable set for "production" is empty in the staging pipeline. Check Settings > CI/CD > Variables and verify the scope matches your deployment target.
  • Protected variables empty on non-protected branches: Variables marked as "Protected" in GitLab are only injected on protected branches (typically main/master). Feature branch pipelines do not receive these variables. Either mark the branch as protected or create non-protected copies of the variables.
  • envsubst replacing unintended placeholders: envsubst replaces all $VAR and ${VAR} patterns in the file, including Kubernetes YAML that uses $(...) for container command substitution. Use envsubst '$APP_VERSION $DB_HOST' to limit substitution to specific variables only.
  • Not base64-encoding secrets in Kubernetes manifests: Kubernetes Secret resources expect base64-encoded values in the data field. Injecting a raw CI variable into data.password without encoding it causes a validation error. Use kubectl create secret --from-literal instead, which handles encoding automatically.

Summary

  • GitLab CI variables are shell environment variables — they do not auto-propagate into Kubernetes YAML files
  • Use envsubst, sed, or Helm --set to inject variables into manifests before applying
  • Check variable scope (environment, branch) and protection settings when variables appear empty
  • Use kubectl create secret --from-literal to safely create Kubernetes secrets from CI variables
  • Debug with env | grep to verify which variables are actually available in the runner

Related reading
Course
Beginner
27 lessons
10 hours
System Design Fundamentals

Build a strong foundation in designing scalable, reliable distributed systems.

View the course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design

All Rights Reserved.