Bump minimum LLVM version to LLVM 6

[gnzlbg]

It's that time of the year again - almost 6 months since we bumped the minimum supported system LLVM to version 5 (#51899). Now that emscripten has been updated to LVM 6.0.1 (#55626 - thank you @nikic ) I wanted to bump the minimum supported version to 6.

However, there has been some controversy about the minimum supported LLVM version lagging too much behind the bundled one, resulting in a lot of frustration for both contributors, users, and packagers (https://internals.rust-lang.org/t/rust-vs-llvm/8604).

Supporting older versions is particularly problematic because:

• we "take our time" upstreaming our patches, which are often required for things to work properly
• rustc at best just bugs! on older LLVM versions when using some language features, at worst it silently compiles with incorrect codegen - distro packagers don't see tests failing because we/them silence them.

I don't know how aware are distro packagers of these issues. I know that @cuviper and @infinity0 are aware of them, e.g., when packaging for a distro often the testsuite has to "pass" but it only passes with system LLVMs because we basically silence all tests that do not pass.

That is, currently, even though we let Rust build silently with LLVM 5, many parts of the language / compiler are just unusable (SIMD, wasm, ...). This is currently also the case for LLVM 6, and even LLVM 7!

So I think it is worth to reconsider our stance on this. I personally think that we should come up with an enforceable policy about the minimum system LLVM version that we support, and that the policy has to be better than "the last 3 LLVM releases" to add any value. If we were to continuously support the last released LLVM version (version 7 right now), we could, slowly:

• try to get most of the tests working properly on the supported system LLVM,
• try to upstream most / all of our patches, provide workarounds in rustc for those that cannot be upstreamed.

Currently, because we support the last 4 LLVM releases, an upstreamed patch lets us remove code in rustc in ~2 - 2.5 years. If we were to only support the last LLVM version, that would be reduced to 6 months, making it significantly more profitable to upstream our patches. It might also make sense to keep workarounds around to keep things going if we know these only have to live for 6 months.

Also, it is unclear to me what's the value of shipping new Rust versions on distros using a 2 year old LLVM. Couldn't these distros just ship an older Rust version that's known to work with the distros LLVM version ? (by known I mean, where most tests pass instead of being silenced).

So what I'd like to know is:

• is it possible at all for stakeholders to move to LLVM 7 now ? If not, why not?
• what are the pros / cons of moving to LLVM now ?
• do you think that a policy of supporting the last LLVM version (7 right now) is feasible?

I think that moving to something like that would be first step. And once everything is working release-wise for everybody, we can consider strengthening this policy to try to keep all / most tests running for one full LLVM release, and to at least encourage workarounds in rustc for the last LLVM version.

So what do you think? In particular, what do distro packagers thing? I know that @cuviper mentioned that Fedora 28 still only has LLVM6 and that it is supported till June 2019, but by ~February 2019 Rust will be targeting LLVM9 already which is released in August 2019.

---

cc @infinity0 @cuviper @Keruspe @glandium @alexcrichton

[gnzlbg]

cc @djc

[nikic]

I think there is a bit of a misapprehension around how the rust LLVM fork works. The fork basically only contains two types of patches:

• Build fixes for CI, because it uses very old glibc. This is not relevant for distros (or they already have their own patches anyway).
• Backports of upstream commits. The baseline requirement is that all patches going into the fork must be reviewed and committed upstream first.

There are only two commits that do not fall into this category:

• rust-lang/llvm@deaf37e -- this is a patch for some relatively unimportant optional functionality, but should definitely be upstreamed.
• rust-lang/llvm@2ed1478 -- this is an optional improvement for the optimizer. It can't be upstreamed in this form. Upstreaming this would require introducing an interface in LLVM to register custom allocators, which would be quite a bit more work.

I'm writing this to clarify that generally the flow of patches is always from LLVM upstream to the rust fork, not the other way around. I think that the issue that you have in mind is really a different one, namely that we don't really keep track of which upstream patches are needed for what over time, and also don't request necessary upstream patches to be merged into previous LLVM point releases.

(Definitely +1 on dropping old LLVM support though, as long as distros are fine with it.)

[petrochenkov]

LLVM 5.0 is barely a year old, and you are suggesting to abandon it already.
This is a requirement for people supporting LLVM forks (e.g. custom targets) to move at a quite speedy fashion.

One reasonable LLVM update policy is to migrate from X.0.1 to X+1.0.1 for reliability, rather than from X.0.0 to X+1.0.0.
With this policy update to LLVM 6 could start only since July 2018, that's only 4 months old.

[gnzlbg]

LLVM 5.0 is barely a year old, and you are suggesting to abandon it already.

To put things in perspective with respect to the release train: LLVM 5.0 was released in August 2017. If we decide to bump the minimum version, until the PR lands on nightly we are talking end of November 2018, 6 weeks later it lands in beta (mid January 2019) and 6 weeks afterwards in stable ~March 2019. By March 2019 LLVM 5.0 will be ~1.5 years old., and Rust master will be using LLVM 9@trunk.

This is a requirement for people supporting LLVM forks (e.g. custom targets) to move at a quite speedy fashion.

cc @kripken (emscripten) and @dylanmckay @shepmaster (avr-rust) - it would be great if you could share your thoughts on these issues.

[shepmaster]

avr-rust

We recently upgraded to LLVM 8 (based on 3480ac2) thanks to @TimNN, so I don't believe we are a concern here.

• The baseline requirement is that all patches going into the fork must be reviewed and committed upstream first.

This has been the AVR fork's intention as well — we try to only have LLVM patches that are already in LLVM master, or at least open for review to be merged.

[cuviper]

• is it possible at all for stakeholders to move to LLVM 7 now ? If not, why not?

As you mentioned, Fedora 28 is still on LLVM 6, and it's non-trivial to upgrade the main LLVM package for the distro. And now that 29 is out, we'll be trying even harder not to rock the boat on 28, as many people use release-1 as a more conservative (stable) Fedora choice.

If hard-pressed, I could add an llvm7 compat package, leaving the main one alone -- but that's not usually done for newer versions, and would present new difficulties for system upgrades. I could also build with the bundled LLVM if really necessary.

• do you think that a policy of supporting the last LLVM version (7 right now) is feasible?

Latest 2 LLVM versions would be better. For Fedora, this lines up nicely since we also release every 6 months and support the latest 2. (Although we actually support 3 for a window of about a month -- e.g. Fedora 27 isn't quite EOL yet.)

It would be even better if you wait for the stable patch release, giving more time for new issues to get worked out. We haven't even seen 7.0.1 yet, but that's in RC now. Ideally, this time until the patch release is a good time for new Rust backports on LLVM X.0 to also be suggested for the official LLVM X.0.1 release. I do try to work with @tstellar already to make that happen.

but by ~February 2019 Rust will be targeting LLVM9 already which is released in August 2019.

By March 2019 LLVM 5.0 will be ~1.5 years old., and Rust master will be using LLVM 9@trunk.

I don't think Rust should make a habit of jumping onto the latest trunk. I understand this was done currently for some WASM fixes, but this should be an exceptional case. Even right now, what we're calling "LLVM 8" is just a snapshot that's actually closer to 7 -- I count ~1200 commits past release_70, but trunk is now more than ~2800 commits beyond rust-llvm! I plan to at least rebase that when the new LLVM monorepo is officially ready for use.

It's well and good to make a best effort to support building with trunk, so we're ready for new releases sooner, but in general I think we should wait for the actual release before we move.

[gnzlbg]

I could also build with the bundled LLVM if really necessary.

@cuviper Could you mention why is it worth it to avoid the bundled LLVM ? Other distro packagers mentioned in the internal threads that they have decided to always use the bundled LLVM anyways.

I don't think Rust should make a habit of jumping onto the latest trunk.

IIRC LLVM recommends projects that depend on it but are hosted outside their main repo to follow trunk as closely as possible because its C++ API does not guarantee any kind of backwards compatibility and this allows new issues to be catched with enough time to fix them before each release.

It's well and good to make a best effort to support building with trunk, so we're ready for new releases sooner, but in general I think we should wait for the actual release before we move.

If we decide to go this route, it might be worth it to add a build bot that attempts to build rustc with LLVM trunk on some tier 1 targets (e.g. x86_64-apple-darwin and x86_64-unknown-linux-gnu).

[cuviper]

@cuviper Could you mention why is it worth it to avoid the bundled LLVM ?

It's a general policy: https://fedoraproject.org/wiki/Bundled_Libraries

And just as a practical matter, it really helps build times to not have to build LLVM too. We don't cache any artifacts between builds like Rust's CI does, and armv7hl in particular is already painfully long, over 5 hours for build and check.

Other distro packagers mentioned in the internal threads that they have decided to always use the bundled LLVM anyways.

I read it that they've resigned to using the bundled LLVM, not that this would be preferred.

IIRC LLVM recommends projects that depend on it but are hosted outside their main repo to follow trunk as closely as possible because its C++ API does not guarantee any kind of backwards compatibility and this allows new issues to be catched with enough time to fix them before each release.

It's a mixed bag, because this also means that fewer people would be testing the release. @tstellar had a talk at LLVM-Dev 2017 advocating more people use stable versions. Even still, we're not actually following trunk closely, as I pointed out how far behind we are right now.

We can still give that upstream benefit of testing pre-releases of LLVM before it gets out the door, but I don't think we should be releasing Rust based on unreleased LLVM.

[gnzlbg]

Thank you, all of that makes sense!

We can still give that upstream benefit of testing pre-releases of LLVM before it gets out the door, but I don't think we should be releasing Rust based on unreleased LLVM.

To implement this in a useful way we would have to stop making rustc require pre-release LLVM versions to work properly.

[djc]

So, as a Gentoo packager, here's my take:

• In terms of support window, it would be nice to be able to support previous major version in addition to current major version, just so that we can provide a stable version to our users. System packages like LLVM can take a while to bake in our stable tree, so longer support would be nice. We definitely don't have a need for LLVM 5, though.

• On the other hand, I'm actually more interested in specific and explicit information on what a given Rust release was actually tested with in terms of LLVM backend. Ideally this would take the form of something like LLVM 7, but with patch/commit X for miscompilation bug P, patch Y for bug Q, patch Z for bug R. In the end, I do believe "what hasn't been tested doesn't work", which I think is also the point @glandium was trying to make in the irlo thread.

• We might be able to carry some LLVM patches in our packages, especially if they have been committed upstream. However, something like the register allocator fix sounds like a potentially hairy thing to include for all of our users.

• Why we like to avoid bundled LLVM: allows us to centralized QA on LLVM-based packages in our tree, plus for Gentoo the added compile time is really a pain. Since Rust releases relatively frequently, this means that every Gentoo users building the default Rust packages spends a lot of time rebuilding the bundled LLVM right now. I resigned to using the bundled LLVM because I don't want the burden of dealing with miscompilation bugs from using a subtly-different standard LLVM release, but this is definitely suboptimal and frowned upon by other Gentoo developers, too.

It's well and good to make a best effort to support building with trunk, so we're ready for new releases sooner, but in general I think we should wait for the actual release before we move.

I very much support this line of thinking.

[gnzlbg]

In terms of support window, it would be nice to be able to support previous major version in addition to current major version, just so that we can provide a stable version to our users.

That is, if currently Rust is using LLVM8@trunk, would LLVM7 be enough? Or would you rather have LLVM6 ?

On the other hand, I'm actually more interested in specific and explicit information on what a given Rust release was actually tested with in terms of LLVM backend.

Would it help if, e.g., the release notes would contain the LLVM version + commit of the bundled LLVM used for stable Rust releases? (maybe with a link to the branch uses by rust's bundled LLVM fork?)

I resigned to using the bundled LLVM because I don't want the burden of dealing with miscompilation bugs from using a subtly-different standard LLVM release, but this is definitely suboptimal and frowned upon by other Gentoo developers, too.

@cuviper had the same request and this is a pretty hard constraint. To expand on this:

• some Rust features require newer LLVM features - these features could not be implemented until the LLVM features hit a stable release, this is bad because newer LLVM features are often buggy, and feedback from users is often able to catch bugs before a release - the LLVM RC period is not enough for this.
• some Rust features use experimental LLVM APIs - these features could never be stabilized because these experimental APIs are not guaranteed to exist across releases, or be remotely similar (they might change semantics so that using a different LLVM version "does something different") - we'll need a policy that enforces that none of these can land on stable.
• some bugs could not be fixed easily - LLVM has bugs, and for Rust master, the easiest way to solve a bug is often to just require the LLVM commit that has the fix or fixes - sometimes these are easy to backport, sometimes they aren't.

Any solution would have to make things work for distros, without making anything worse for those using the bundled LLVM.

A "compromise" might be to use LLVM@trunk on nightly, and to also test beta thoroughly with the minimum supported LLVM version (on all targets). This gives us 6 weeks to error properly on everything that requires the bundled LLVM or to try to provide a backwards compatible workarounds. But 6 weeks is not a long time and the older the minimum released LLVM version is, the higher the chances that Rust users on stable that are not using the bundled LLVM will just see an "error: your system's llvm version is too old"-type of error message.

[glandium]

Would it help if, e.g., the release notes would contain the LLVM version + commit of the bundled LLVM used for stable Rust releases? (maybe with a link to the branch uses by rust's bundled LLVM fork?)

This reminds me... rustc --version --verbose prints the plain LLVM version. That is, current nightly says: 8.0. That is essentially useless information, since this is not 8.0 at all.

[cuviper]

FYI, #55835 is updating to a newer trunk snapshot.

[gnzlbg]

FYI, to put that in context, there are people collaborating to specify 128-bit SIMD for WASM, implement that in LLVM, and exposing that from Rust. This is iterative work that requires keeping the spec, llvm, and Rust, in sync. And this is not the only feature that requires this kind of synchronization.

So rust-lang/master following LLVM trunk is pretty much a hard requirement for progress in some fronts right now.

As long as these features do not percolate into stable, this does not imply that Rust stable releases have to require LLVM trunk as well. I believe we can find a solution that works for all parties involved.