Cleaning Up Git

[KittyChiu]

Note

TL;DR Sharing field notes on Git maintenance, a preparation step for migration to Git

Git is a text-based file system for software configuration management, optimized for tracking changes and compressing text files efficiently.

Regular Git maintenance helps to:

• Improve Git performance
• Prepare git file system for migration to another hosting platform
• Reduce repository size and complexity

Tip

Sometimes I am amused how people use Git beyond text files, they store binary files like snapshots, media files, backups, libraries and pakages, etc. With that, we need to do Git maintenance to remediate the performance issue.

Below are practical methods for Git maintenance, ranging from simple cleanup to advanced repository surgery.

Simple Cleanup: Delete Stale Branches or Tags

Remove unused branches or tags from the remote repository:

git push origin --delete BRANCH-OR-TAG-NAME

Advanced Cleanup: Repository Surgery

Important

Always clone a fresh copy of your repository before performing advanced operations. These actions rewrite history and are irreversible after pushing.

To clone a mirror repository for safe experimentation:

git clone --mirror YOUR-REPO-GIT-URL
cd YOUR-REPO
git remote set-url origin YOUR-REPO-GIT-URL

Lever 1: Migrate Large Files to Git LFS

Use Git Large File Storage (LFS) to manage large files efficiently.

Step 1: Install Git LFS:

git lfs install

Step 2 (Optional): Configure custom LFS storage location:

git config lfs.url https://YOUR-ARTIFACTORY-INSTANCE/artifactory/api/lfs/YOUR-REPOSITORY-NAME

Step 3: Track specific file types with LFS:

git lfs track '*.FILE-EXTENSION'

Step 4: Migrate existing files to LFS:

git lfs migrate import --include='*.FILE-EXTENSION'

Step 5: Verify LFS-tracked files:

git lfs ls-files

Lever 2: Remove Unwanted Files by Rewriting History

Use git-filter-repo to permanently remove unwanted files from repository history.

Step 1: Install git-filter-repo.

Step 2: Analyze repository history:

git filter-repo --analyze

Step 3: Review large or unwanted files in .git/filter-repo/analysis/path-all-sizes.txt.

Step 4: Remove unwanted files using filters:

git filter-repo --invert-paths --path DIR-OR-FILE
git filter-repo --invert-paths --path-glob GLOB
git filter-repo --invert-paths --path-regex REGEX
git filter-repo --strip-blobs-bigger-than SIZE-EXAMPLE-10M
git filter-repo --strip-blobs-with-ids BLOB-ID-FILENAME

Step 5 (Optional): Backup commit-map for reference:

cp filter-repo/commit-map ./_filter_repo_commit_map_$(date +%s)

Lever 3: Drop Specific Commits

Remove unwanted commits from history using interactive rebase.

Step 1: Identify problematic commits:

With git-sizer

git-sizer --verbose

With Git commands (sort by size, list top 25)

git rev-list --objects --all | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' | sort -k3 -nr | head -n 25

Examine a specific commit

git ls-tree -r -t  --format='%(objectname) %(objecttype) %(path)' $COMMIT | git cat-file --batch-check='%(objectsize) %(objectname) %(rest)'

Step 2: Start interactive rebase from a commit before the unwanted commit:

git rebase -i HEAD~3

Step 3: Edit the Rebase Todo List. This will open an editor with a list of commits. The list might look something like this:

pick i7j8k9l Initial commit
pick e4f5g6h Fixed a bug
pick a1b2c3d Added new feature

This will begin an interactive rebase from a point before the commit you want to remove.

Step 3: In the editor, mark the unwanted commit with drop:

For example, if the commit e4f5g6h is the second commit to drop, you would update drop next to it.

pick i7j8k9l Initial commit
drop e4f5g6h Fixed a bug
pick a1b2c3d Added new feature

Step 4: Save and exit the editor to apply changes.

Git will then replay the commits, excluding the one you marked to drop.

Lever 4: Prune Reference Logs

Clean up old or unreachable references to reduce repository size.

Option 1: Remove unreachable references immediately:

git reflog expire --expire-unreachable=now --all

Option 2: Remove all references older than now:

git reflog expire --expire=now --all

Lever 5: Garbage Collection

Clean up dangling objects and optimize repository storage.

Step 1: Review unreachable and dangling objects:

git fsck --full
git fsck --unreachable

Step 2: Remove dangling objects:

git gc --prune=now --aggressive

Step 3: Verify integrity and connectivity of all objects:

git fsck --full

Final Step: Push Changes to Remote

After performing advanced cleanup, push to rewrite remote history:

Option 1: Rewrite all references (branches, tags, remotes):

git push origin --force --mirror

Option 2: Rewrite all branches only:

git push --all --force

Option 3: Rewrite specific branches:

git push origin --force 'refs/heads/*'

Option 4: Rewrite all tags only:

git push --tags --force

Option 5: Rewrite specific tags:

git push origin --force 'refs/tags/*'

Warning

Force-pushing rewrites remote history. Ensure all collaborators are informed and have synchronized their local repositories accordingly.

[Salgado2004]

Very interesting! Recently I've been experimenting some advanced git commands like filter-repo 😅 they are really powerful but also scary, so I'm going slow. This is a very useful article! Thank you very much 😁

[readytotest]

I was having similar issues with repo size due to Allure reports from running Playwright tests, which made the repo grow to 1GB in a few weeks. Anyway, I automated the cleanup with a GitHub Actions workflow that uses BFG Repo-Cleaner. I actually tried git filter-repo first but ran into issues later on in the workflow, so I switched to BFG to see how it'd work. If you're worried about messing things up, you could always create a temporary experimental repo to test it out first! Here’s the link to my workflow YAML. Feel free to give it a try or experiment with it!

[spenserblack]

Since we're talking about rebasing, here's a couple of options for git commit I love:

git commit --fixup <ref>
git commit --squash <ref>

Where <ref> refers to a commit, like HEAD or a commit hash. This autogenerates a header in the commit message that's an instruction that the commit you're making should be combined with another commit. --fixup and --squash are pretty similar, but you don't have to write a commit message when you use --fixup.

Then, git rebase -i --autosquash <ref>, where <ref> is the point where you want to start rewriting history. The --autosquash lets git rebase know that any "squash" or "fixup" commits should be applied to the git rebase to-do list as such! It's very helpful when you're making a git history that's kind of messy, like temp commits, and you plan on cleaning it up later. Of course, like said above, git rebase is dangerous because it rewrites your git history, so be careful!

[Ayokanmi-Adejola]

The explanation of using git filter-repo and the various filtering options is very well done. It's a powerful tool, but as you rightly point out, requires careful handling. The emphasis on cloning a mirror first is crucial.