Git with large files
Interview Questions practice on Codemia
Over 8,000 real interview questions from top companies, searchable by company and role.
Introduction
Git is excellent at tracking text history, but it performs poorly when repositories fill up with large binary files. Every revision of a large binary tends to be stored as a new blob, so clone size, fetch time, and repository maintenance all get worse unless you change how those files are handled.
Why Large Files Hurt Git
Git stores content as immutable objects. That works brilliantly for source code because text compresses well and delta storage is effective. Large binaries are different.
Typical problems include:
- repository size grows rapidly
- clone and fetch operations become slow
- CI pipelines spend extra time downloading history
- history rewriting becomes painful if large files were committed by mistake
This is why "Git with large files" is mostly a workflow question, not just a storage question.
Use Git LFS for Large Binary Assets
Git LFS, usually expanded as Git Large File Storage, is the standard answer when you truly need versioned large files inside a Git-based workflow. Git keeps small pointer files, while the actual large content is stored separately by the LFS backend.
A minimal setup looks like this:
This keeps normal Git history much lighter than storing each binary revision as a full Git blob.
Git LFS is a good fit for:
- design assets
- game assets
- trained models that must stay versioned with code
- release artifacts that are part of the development workflow
Do Not Use Git as a Generic File Dump
Sometimes the best solution is not Git LFS but simply not storing the file in Git at all. Large generated files, build outputs, datasets, and media often belong in object storage, package registries, or release assets instead.
Examples of better homes for large artifacts include:
- S3 or other object storage
- release attachments on your hosting platform
- package registries
- dataset or model artifact stores
If the file does not benefit from line-by-line history and is only needed at runtime or release time, Git is often the wrong place for it.
Clean Up History if Large Files Are Already There
If a large file was committed into normal Git history, deleting it from the latest commit is not enough. The old blob still exists in history and still inflates clones.
This is where history-rewrite tools matter. Modern Git cleanup is commonly done with git filter-repo, and older workflows often mention BFG Repo-Cleaner.
A conceptual example with git filter-repo is:
After rewriting, you usually need to force-push and coordinate with every collaborator because history changed.
That is a serious operation, but it is the correct fix when the repository is already bloated by past binary commits.
Keep Clones Efficient
Even with LFS, large repositories benefit from disciplined access patterns. Developers do not always need the full history or every asset immediately.
Helpful techniques include:
- shallow clones for CI where full history is unnecessary
- partial clone strategies when supported by the hosting platform
- separate repositories if assets and code have very different lifecycles
The principle is the same: keep the default developer workflow small and fast.
Make the Team Policy Explicit
Large-file problems usually become expensive because there is no repository policy until after the repository is already huge. Decide early:
- which file types go into Git LFS
- which file types are forbidden from Git entirely
- how large-file mistakes are detected in CI or pre-commit hooks
That policy prevents the recurring cycle of accidental giant commits followed by emergency cleanup.
Common Pitfalls
- Committing large binaries directly to Git and planning to "clean it up later" often leaves permanent repository bloat.
- Using Git LFS for files that should really live in object storage still creates unnecessary version-control overhead.
- Deleting a large file from the latest commit does not remove it from repository history.
- Rewriting history to remove large blobs without coordinating the team can break every collaborator's local clone.
- Ignoring generated files in
.gitignoreis a common way to let build artifacts and caches creep into the repository.
Summary
- Git handles source code well, but large binary files degrade repository performance quickly.
- Use Git LFS when large files truly need versioning alongside the codebase.
- Keep runtime artifacts, datasets, and generated outputs out of Git when possible.
- If large files already polluted history, remove them with a history-rewrite tool rather than just deleting the current file.
- Establish a clear team policy so large-file mistakes do not become a recurring repository problem.
Related reading
- Git workflow and rebase vs merge questions
- Git workflow and rebase vs merge questions
- git Your branch is ahead by X commits
- Github - unexpected disconnect while reading sideband packet
- GitHub Actions build outside vs inside container?
- GitHub Error Message - Permission denied (publickey)
- GitHub Error Message - Permission denied publickey
- GitHub fatal remote origin already exists
.png&w=3840&q=75)
Tackling System Design Interview Problems
A short course that equips you with the skills to approach system design interviews methodically.
Start the free courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
Interview Questions practice on Codemia
Over 8,000 real interview questions from top companies, searchable by company and role.