Cross compilation fails on Rust nightly when using proc macro crates due to GCC not understanding `-fuse-ld=lld`

[VorpalBlade]

Code

I tried this code:

❯ cargo new hello_world
    Creating binary (application) `hello_world` package
cnote: see more `Cargo.toml` keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
❯ cd hello_world/
❯ cross +nightly build --target aarch64-unknown-linux-gnu
[cross] warning: using newer rustc `1.80.0-nightly (d84b90375 2024-05-19)` for the target. Current active rustc on the host is `rustc 1.78.0 (9b00956e5 2024-04-29)`.
 > Update with `rustup update`
   Compiling hello_world v0.1.0 (/project)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.82s
❯ cargo add syn
    Updating crates.io index
      Adding syn v2.0.65 to dependencies
             Features:
             + clone-impls
             + derive
             + parsing
             + printing
             + proc-macro
             - extra-traits
             - fold
             - full
             - test
             - visit
             - visit-mut
    Updating crates.io index
❯ cross +nightly build --target aarch64-unknown-linux-gnu
[cross] warning: using newer rustc `1.80.0-nightly (d84b90375 2024-05-19)` for the target. Current active rustc on the host is `rustc 1.78.0 (9b00956e5 2024-04-29)`.
 > Update with `rustup update`
   Compiling proc-macro2 v1.0.83
   Compiling unicode-ident v1.0.12
error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/rust/lib/rustlib/x86_64-unknown-linux-gnu/bin:/rust/lib/rustlib/x86_64-unknown-linux-gnu/bin/self-contained:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/rust/bin" VSLANG="1033" "cc" "-m64" "/tmp/rustcAyUfnb/symbols.o" "/target/debug/build/proc-macro2-b1e0569744dc7f5d/build_script_build-b1e0569744dc7f5d.build_script_build.48684d15272f5bdb-cgu.0.rcgu.o" "/target/debug/build/proc-macro2-b1e0569744dc7f5d/build_script_build-b1e0569744dc7f5d.build_script_build.48684d15272f5bdb-cgu.1.rcgu.o" "/target/debug/build/proc-macro2-b1e0569744dc7f5d/build_script_build-b1e0569744dc7f5d.crzselpkxut2yhaw0phvg3mu9.rcgu.o" "-Wl,--as-needed" "-L" "/target/debug/deps" "-L" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-de48b8168d6cf4fa.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-9f3477fb95a0bba7.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-210d920812faea91.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-f3d3451767410a17.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-1a79dd36d08251de.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-886230e7120831b2.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-2caea079085a58a2.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-a9d7d97cdc65a449.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-5727477b0a78105a.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-7e555563aa211118.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-c8f13465f1a795b2.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-ee5b5774583426df.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-14df174c91007922.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-e8bfe52be756260a.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-6275035a459b3ada.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-b6892f3c52c68f01.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-f72b956e24d1de70.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-632ae0f28c5e55ff.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-e8b7e96e438f08f6.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-B/rust/lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld" "-fuse-ld=lld" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-L" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/self-contained" "-o" "/target/debug/build/proc-macro2-b1e0569744dc7f5d/build_script_build-b1e0569744dc7f5d" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-nodefaultlibs"
  = note: cc: error: unrecognized command line option '-fuse-ld=lld'
          

error: could not compile `proc-macro2` (build script) due to 1 previous error

I expected to see this happen: Cross compilation should work fine

Instead, this happened: Cross compilation fails due to gcc not accepting -fuse-ld=lld

Version it worked on

It most recently worked on: Nightly as of about a week ago, but it also works fine on stable.

Version with regression

rustc --version --verbose:

❯ rustc +nightly --version
rustc 1.80.0-nightly (d84b90375 2024-05-19)

Backtrace

Backtrace

<backtrace>

@rustbot modify labels: +regression-from-stable-to-nightly -regression-untriaged

[VorpalBlade]

Also reported as cross-rs/cross#1496 but it seems to me that it is a rust regression that rust tries to use a gcc option when the installed GCC version is too old to support said option.

[saethlin]

I agree that this is a regression. Our tier 1 targets support GLIBC 2.17+ which dates back to 2012 but -fuse-ld=lld in gcc was merged in 2018. While we do not explicitly declare support for any gcc or distro version, GLIBC 2.17+ is generally interpreted as RHEL/CentOS 7.

Unfortunately, the platforms that the toolchain can run on often constrains what platforms can be targeted. We currently do not provide a mechanism to target GLIBC versions older than the host.

So I do not think it is not a big stretch to argue that passing -fuse-ld=lld without a check for a compatible gcc breaches the target tier support that we declare.

[VorpalBlade]

So I do not think it is not a big stretch to argue that passing

@saethlin The double negation is hurting my brain, what do you actually mean here?

[saethlin]

I mean that the new behavior violates of the spirit of the target tier policy, but not the very precise guarantees we document.

[lqd]

I had expected us to have a gcc version check already, especially since our dist i686/x64 linux CI builders are based on CentOS 7, but it seems the original -Zgcc-ld=lld implementation doesn't have such a version check, and that CI had gcc 9.5 available because it actually builds it from source using the GCC 4.8.5 package available.

I'll look into adding the version check.

[lqd]

Hmm, it seems

• version detection was deliberately avoided in the implementation, not forgotten
• and that this issue of old GCCs was discussed in How to find LLD with old versions of gcc? #97402 where consensus at the time (amongst participants, not a wide-reaching discussion to be clear) seemed to be "there was no need to support GCCs this old". Which also could explain why we don't have GCC 4 when running tests on CI.

Maybe we need to reassess both of these and discuss more broadly, and/or document more precisely what we intend to rely on in the target support documentation, and/or more easily offer feature detection for users where these cases are important.

The latter could be a workflow like the following, where users want to dynamically opt-out of using lld:

• until stabilization, check if using a nightly compiler -- and use -Z/-Zunstable-options flags that won't be needed in the future
• running rustc --print linker-features returns whether lld is supported (we'd need to add support for linker-features in rustc --print first of course). We could also check if rust-lld exists in the sysroot here, instead of only as a fallback during compilation like rust-lld: fallback to rustc's sysroot if there's no path to the linker in the target sysroot #125263.
• if that succeeds (handwaving details), then users wanting to opt-out could do so with -Z/-Clinker-features=-lld. Of course, here they could also add their own dynamic opt-in/opt-out checks, like a GCC version check.
• otherwise, the default linker would not be lld in the first place, and nothing would need to be done

That would allow for users supporting multiple versions of the compiler and multiple versions of GCC if they need to.

cc @petrochenkov what do you think?

[VorpalBlade]

My use case is building in CI, both for testing and for distributable binaries. I don't run old distros myself (I run Arch, which is rolling release). I just rely on cross selecting a suitable docker image such that it can cross compile binaries that are portable to any relevant glibc based Linux distros. And it seems that works okay (even on nightly), until it tries to build a proc macro that should execute on the host in the docker container.

Now I would say that "building portable binaries for distribution" is an important use case for Rust. Any binary crate that is not targeted at rust developers might want to do this. I don't expect my users to have a rust toolchain (or any toolchain at all) installed. So asking them to use cargo install does not make much sense.

Because of the unfortunate situation on Linux with glibc backward compatibility, the only reasonable way to do this is to build on an old distro (since glibc makes sure the sure the binary is forwards compatible, but not the reverse).

I believe this is a use case that needs to work. I don't particularly care about how it is made to work. Maybe rust can just bypass cc and call lld directly for example.