Skip uninteresting commits for blame

[cruessler]

This PR is a PoC. It is a first attempt at implementing custom graph traversal for blame::file. This allows us to skip commits that we know don’t contain changes to the file being blamed. The PR is rough around the edges and contains a lot of TODOs and todo!()s.

A first round of tests showed a significant reduction in the number of commits traversed for gix blame STABILITY.md:

# before
Statistics {
    commits_traversed: 9261,
    commits_to_tree: 10710,
    trees_decoded: 10710,
    trees_diffed: 37,
    blobs_diffed: 36,
}

# after
Statistics {
    commits_traversed: 3657,
    commits_to_tree: 3749,
    trees_decoded: 3749,
    trees_diffed: 37,
    blobs_diffed: 36,
}

Also, profiling gix blame STABILITY.md on my machine revealed that the command took about 20 % less time to run.

[cruessler]

I think (and hope :-)) that this PR is now ready for the first round of reviews. There’s still two or three todo!() left, but I’ve addressed all of the TODOs as far as I can see and I’m really satisfied with the numbers so far as there seems to be a considerable speedup.

Also, the traversal is now much closer to what git does, and I hope that it most of the time is even identical unless there are differences in diffing results that can make traversals diverge. I haven’t found ways to validate this claim at a large scale, though, so I want to be cautious at this point.

gitoxide on  main [!?⇣] is 📦 v0.40.0 via 🦀 v1.84.0 took 5s
❯ hyperfine "$HOME/github/Byron/gitoxide/target/release/gix blame STABILITY.md" "$HOME/worktrees/gitoxide/branch-1/target/release/gix blame STABILITY.md"
Benchmark 1: /home/christoph/github/Byron/gitoxide/target/release/gix blame STABILITY.md
  Time (mean ± σ):     194.6 ms ±   2.3 ms    [User: 174.0 ms, System: 19.8 ms]
  Range (min … max):   190.8 ms … 198.5 ms    15 runs

Benchmark 2: /home/christoph/worktrees/gitoxide/branch-1/target/release/gix blame STABILITY.md
  Time (mean ± σ):      99.2 ms ±   1.5 ms    [User: 87.3 ms, System: 11.3 ms]
  Range (min … max):    97.1 ms … 104.1 ms    29 runs

Summary
  '/home/christoph/worktrees/gitoxide/branch-1/target/release/gix blame STABILITY.md' ran
    1.96 ± 0.04 times faster than '/home/christoph/github/Byron/gitoxide/target/release/gix blame STABILITY.md'

[Byron]

That's a massive speedup, fantastic work! And sorry for the late (first) review, I am now totally behind on emails each, catching up only during the weekend.

When looking at the commits I saw that they are a bit all over the place, often touching multiple crates at the same time. This is OK when not using <prefix>: <subject> notation, but it's not otherwise.

Can we practice this by adjusting the <prefix>: <subject> commits so that they only touch a single crate? This can be followed by an adapt to changes in <crate> commit to make fixes that make it compile (yes, unfortunately not every commit would pass CI, but it's for getting proper changelogs).

In that moment, please feel free to squash other commits that might not be so valuable all by themselves.

I am also more than happy to do this as part of my review, just let me know.

[cruessler]

Sure, no problem, thanks a lot for the detailed explanation! (I wasn’t aware that this means accepting intermediate commits that don’t pass CI.)

I will rearrange the commits, but I will only get to it on Monday or Tuesday, so feel free to take over if you get to it earlier!

[cruessler]

I finally got to re-arranging this PR’s commits. It’s now 3 commits, one per crate touched. The commit messages might need to be adapted as I probably don’t know all the conventions yet. I tried to capture as much information as possible in the individual commit message, so you have something to work with. Let me know if that works for you!

[Byron]

Thanks a lot, this is wonderful!

Previously, using gix blame on the linux Kernel was impossible as the previous topo-search would take a long time doing some precomputation (probably figuring out lane-width without a limit). This is now gone, allowing for first performance comparisons:

❯ hyperfine -w1 -N  'git -c core.commitgraph=0 blame README' 'gix blame README'
Benchmark 1: git -c core.commitgraph=0 blame README
  Time (mean ± σ):     233.2 ms ±   0.8 ms    [User: 206.5 ms, System: 25.3 ms]
  Range (min … max):   232.3 ms … 234.4 ms    12 runs

Benchmark 2: gix blame README
  Time (mean ± σ):     273.6 ms ±   2.0 ms    [User: 234.7 ms, System: 40.2 ms]
  Range (min … max):   271.4 ms … 277.4 ms    10 runs

Summary
  git -c core.commitgraph=0 blame README ran
    1.17 ± 0.01 times faster than gix blame README

linux ( master) +408 -798 [!⇣14583] took 6s
❯ hyperfine -w1 -N  'git -c core.commitgraph=0 blame README' 'gix -c core.commitgraph=0 blame README'
Benchmark 1: git -c core.commitgraph=0 blame README
  Time (mean ± σ):     230.6 ms ±   0.7 ms    [User: 204.2 ms, System: 25.0 ms]
  Range (min … max):   229.2 ms … 231.4 ms    13 runs

Benchmark 2: gix -c core.commitgraph=0 blame README
  Time (mean ± σ):     383.5 ms ±   1.3 ms    [User: 341.3 ms, System: 43.5 ms]
  Range (min … max):   382.1 ms … 385.4 ms    10 runs

Summary
  git -c core.commitgraph=0 blame README ran
    1.66 ± 0.01 times faster than gix -c core.commitgraph=0 blame README

linux ( master) +408 -798 [!⇣14583] took 8s
❯ hyperfine -w1 -N  'git blame README' 'gix blame README'
Benchmark 1: git blame README
  Time (mean ± σ):     112.4 ms ±   1.7 ms    [User: 86.3 ms, System: 25.0 ms]
  Range (min … max):   110.8 ms … 118.0 ms    26 runs

Benchmark 2: gix blame README
  Time (mean ± σ):     269.1 ms ±   0.8 ms    [User: 230.7 ms, System: 39.8 ms]
  Range (min … max):   267.4 ms … 270.1 ms    11 runs

Summary
  git blame README ran
    2.39 ± 0.04 times faster than gix blame README

Turning off the commit-graph also prevents Git to use the bloom filter for paths, but it turns out that I didn't even have these included.

When calculating them, Git gets even faster.

❯ l .git/objects/info/commit-graph
.r--r--r-- 76Mi byron staff 25 Dec  2024 .git/objects/info/commit-graph

linux ( master) +408 -798 [!⇣14583]
❯ git commit-graph write --changed-paths
Finding commits for commit graph among packed objects: 100% (10972752/10972752), done.
Expanding reachable commits in commit graph: 1356480, done.
Computing commit changed paths Bloom filters: 100% (1323740/1323740), done.
Writing out commit graph in 7 passes: 100% (9266180/9266180), done.

linux ( master) +408 -798 [!⇣14583] took 1m16s
❯ l .git/objects/info/commit-graph
.r--r--r-- 94Mi byron staff  1 Feb 16:39 .git/objects/info/commit-graph

linux ( master) +408 -798 [!⇣14583]
❯ hyperfine -w1 -N  'git blame README' 'gix blame README'
Benchmark 1: git blame README
  Time (mean ± σ):      81.5 ms ±   1.6 ms    [User: 32.3 ms, System: 50.0 ms]
  Range (min … max):    76.3 ms …  85.5 ms    38 runs

Benchmark 2: gix blame README
  Time (mean ± σ):     291.3 ms ±   1.8 ms    [User: 229.5 ms, System: 63.3 ms]
  Range (min … max):   289.4 ms … 294.4 ms    10 runs

Summary
  git blame README ran
    3.57 ± 0.07 times faster than gix blame README

It's notable that if both aren't using the commitgraph, git is 1.66x faster, and even when gix gets to use the commitgraph while git doesn't, Git handily beats it at 1.17x 😅.

It feels we already have come a long way, yet there is so much more to be done here.
My feeling is that if the tree could be diffed with a pathspec (or even, a single-path filter), it would save huge amounts of time.