Best practices for storing kubernetes configuration in source control
Master System Design with Codemia
Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.
Introduction
Kubernetes configuration belongs in source control, but how you store it determines whether deployments stay safe and repeatable. Good repository structure makes changes auditable and easy to promote across environments. Poor structure leads to drift, hidden secrets, and hard to review pull requests.
Organize Manifests by Reusable Base and Environment Overlay
A common pattern is one reusable base plus environment overlays for development, staging, and production. Tools such as Kustomize or Helm can render environment specific outputs without duplicating everything.
This keeps shared logic centralized while still allowing environment specific differences.
Keep Secrets Out of Plain Text
Do not commit unencrypted secrets in Kubernetes Secret manifests. Even private repositories are not a safe secret storage layer by themselves. Use encrypted workflows such as SOPS, sealed secrets, or external secret managers integrated with cluster controllers.
Only encrypted secret artifacts should be committed. Decryption keys and cloud credentials must be managed outside the repository with strict access controls.
Enforce Validation and Policy in CI
Source control alone does not prevent invalid or unsafe manifests. Add automated checks in CI for schema validation, policy, and security posture.
Useful checks include:
- Kubernetes schema validation on every pull request.
- Linting for Helm or Kustomize outputs.
- Policy rules for privileged containers, host networking, and image tags.
- Drift detection between desired and live cluster state.
These gates catch issues early and reduce deployment surprises.
Use GitOps for Change Delivery
GitOps controllers apply repository state to clusters continuously and provide a strong audit trail. Every change passes through pull requests, review, and merge history. Rollbacks become a normal Git operation instead of manual cluster editing.
Avoid direct kubectl edit in production except during incident response with clear follow up reconciliation. Manual live edits create drift and make incident reconstruction difficult.
Tag releases and map them to cluster environments so teams can answer two critical questions quickly: what is running now, and when did it change.
Repository governance is equally important. Define code owners for platform level folders, require approvals from security or SRE reviewers for RBAC and network policy changes, and keep pull requests small enough for meaningful review. High quality review discipline is one of the best controls against configuration mistakes reaching production. Keep commit messages descriptive so incident reviews can trace intent quickly. Include ticket references when policy exceptions are approved so auditors can follow context. This improves long term governance. It also speeds audits for teams.
Common Pitfalls
A common pitfall is storing all environments in one large manifest file with repeated blocks. This increases merge conflicts and review noise. Use overlays to isolate only the differences.
Another issue is committing generated output without tracking source templates. Generated files can be useful for debugging, but template sources should remain the primary editable artifacts.
Teams also mix application code and cluster platform config without ownership boundaries. Define clear directories and review owners to avoid accidental privilege changes.
Finally, using mutable image tags such as latest makes rollbacks unreliable. Pin immutable tags and promote deliberately through environments.
Summary
- Store Kubernetes config in version control with base plus overlay structure.
- Never commit plain text secrets; use encrypted secret workflows.
- Add schema, lint, and policy checks to CI pipelines.
- Use GitOps to enforce audited, declarative delivery.
- Pin immutable image versions and minimize manual cluster edits.
