Replace submodules with subtrees

[oli-obk]

cc @rust-lang/clippy @RalfJung

cc @rust-lang/compiler @rust-lang/infra we gate on clippy soon, so if you break clippy you need to fix it immediately in the same PR. This is trivial now though, since we are basically in a monorepo state, so you just edit stuff in src/tools/clippy/ until everything works again.

Synchronizations of the remote repo and the local subtree is explained in CONTRIBUTING.md

The first port (clippy) is being done in #70655

[RalfJung]

Cc @rust-lang/miri because we have that now :D

[eddyb]

Hmm, I was led to believe that git subrepo was nesting git repositories, in a way which git natively handles. But it turns out it's just a monorepo sync tool, same as manually making large commits. I don't think losing commit history is acceptable.

At a glance, git subtree looks like it might do the right thing, i.e. keep the repository nesting of a submodule, but I haven't tried it yet to see how it actually behaves in practice.
EDIT: ah, no, git subtree just seems to create synthetic commit histories. Still, could be better than a single sync mega-commit.
EDIT2: somehow it still managed to preserve commit history?! Compare:
https://github.com/rust-lang/rust-clippy/commits/master
https://github.com/rust-lang/rust/commits/c211cea3e99e04c2980a853b6637de22881b72eb

[oli-obk]

I moved to git subtree.

[eddyb]

In case it gets lost on the PR, I said: #70655 (comment)

Whoa, clippy commit hashes were kept intact, this makes me really happy!11

[RalfJung]

I moved to git subtree.

Would be good to get the pro's and con's of that decision documented somewhere. git submodule advertises itself as a better alternative to git subtree; why is subtree still the better choice for us?

(Also I have no idea in which thread to post now, there's at least this issue and 2 PRs. It's very confusing.^^)

[oli-obk]

I don't know why git subrepo is better than git subtree other than what the documentation of git subrepo states at https://github.com/ingydotnet/git-subrepo#benefits

Fixes known rebase failures with git-subtree.

git subtree is better than git subrepo because it preserves all commits in history instead of squishing them in the name of the person who synchronizes it.

[eddyb]

git subtree is upstream, which is nice, and it also supports preserving the commits better.
I believe the "rebase failures" that git subrepo talks about are actually an anti-pattern: rebasing a merge commit, it's not meant to work at all (the rebase will include both branches and lose the merge commit - this can already happens if you accidentally rebase the wrong thing in rust-lang/rust).

git subrepo seems to want to just throw all the changes into single large commits so you "don't have to deal with it", whereas git subtree is more history-preserving-oriented, which I prefer.

[RalfJung]

git subtree is better than git subrepo because it preserves all commits in history instead of squishing them in the name of the person who synchronizes it.

Is this true in both directions (subtree pull/push)?

Does git subtree also have the problem that a "subtree push" needs a commit on the rustc side?