Run git gc periodically on the crates.io index (#778)

[ohaddahan]

Forking git gc periodically from start_registry_watcher.

Closes #778.

[Nemo157]

This needs the path to the repo to clean. I think it'd make most sense to move this to a method on index::Index since that's what needs cleaning and has the path available. You can then access it via docbuilder.index to call.

[ohaddahan]

@Nemo157 inside what method?

How about RustwideBuilder::build_package ? Doesn't it run in the same location as the git repo of the crate?

[Nemo157]

A new method, probably just keep this name for it run_git_gc.

I have vague ideas of eventually removing the index to queue handling from docbuilder completely, it isn't really related to building the docs.

[ohaddahan]

@Nemo157 what I meant was, where in the Index flow to add it.

I checked now and the Index is called from RustwideBuilder which is called from the Queue.

So maybe add last_gc_run property on the Queue and call run_git_gc conditionally from inside build_package? Something along these lines?

[Nemo157]

I was thinking just keep calling it from here, this is the place that causes us to create new packfiles so should be responsible for knowing when to clean them up.

[ohaddahan]

@Nemo157 what is the location you refer to by calling it from here?

[jyn514]

I assume he meant daemon.rs. So fn run_git_gc gets moved to Index, but still gets called from start_registry_watcher.

[ohaddahan]

@jyn514 @Nemo157 updated, let me know if that what you meant 😊

[jyn514]

I assume he meant daemon.rs. So fn run_git_gc gets moved to Index, but still gets called from start_registry_watcher.

[Kixiron]

@ohaddahan Could you push your commits so we can see the changes?

[ohaddahan]

@Kixiron ops sorry, forgot to merge my fork branch 😊

[jyn514]

@ohaddahan Hi, your commits messages say 'rebase' but rebases do not have commits, they apply the original commits on top of the original branch. As it stands this has 143 commits and a 900 line diff. Can you please rebase using something like the following:

git checkout master
git fetch origin
git rebase -i origin/master

Since you have several merge commits there will be many conflicts, so you should drop all commits that are not related to this PR. Alternatively, it might be easier to start over on a new branch:

git checkout origin/master
git checkout --branch git-gc
git cherry-pick all the relevent commit hashes

However, as-is, this is very difficult to review.

[jyn514]

If you want help feel free to join the discord channel: https://discord.gg/f7mTXPW

[ohaddahan]

@jyn514 yeah my fork went down the drain, I'll clean it up.

[jyn514]

Closing in favor of #975