Linker binaries (`ld`, `lld`, `rust-lld`) fail to find system-wide libraries, while using compiler binaries (`gcc`, `clang`) works fine

[vadorovsky]

I'm trying to build the integration-test subproject from https://github.com/aya-rs/aya on Alpine Linux, for x86_64-unknown-linux-musl target.

Using the default linker (without specifying, so it defaults to gcc) works fine:

77d2cb0e3483:~/aya# cargo build --target x86_64-unknown-linux-musl --package integration-test
[...]
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 17.70s

(full log)

Same when specifying gcc as a linker:

77d2cb0e3483:~/aya# cargo build --target x86_64-unknown-linux-musl --config "target.x86_64-unknown-linux-musl.linker = \"gcc\"" --package integration-test
[...]
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 18.06s

(full log)

Using clang works too:

77d2cb0e3483:~/aya# cargo build --target x86_64-unknown-linux-musl --config "target.x86_64-unknown-linux-musl.linker = \"clang\"" --package integration-test
[...]
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 17.93s

(full log)

However, using linker binaries directly doesn't work, regardless of which linker I'm trying to use.

GNU ld:

77d2cb0e3483:~/aya# cargo build --target x86_64-unknown-linux-musl --config "target.x86_64-unknown-linux-musl.linker = \"ld\"" --package integration-test
[...]
   Compiling memoffset v0.6.5
error: linking with `ld` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/bin:/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/bin:/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/bin:/usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" VSLANG="1033" "ld" "--version-script=/tmp/rustccmphOr/list" "--no-undefined-version" "/tmp/rustccmphOr/symbols.o" "/root/aya/target/debug/deps/indoc-4e3fe2b7cb1d37d2.indoc.42d2e69cb3cf60e2-cgu.0.rcgu.o" "/root/aya/target/debug/deps/indoc-4e3fe2b7cb1d37d2.indoc.42d2e69cb3cf60e2-cgu.1.rcgu.o" "/root/aya/target/debug/deps/indoc-4e3fe2b7cb1d37d2.indoc.42d2e69cb3cf60e2-cgu.2.rcgu.o" "/root/aya/target/debug/deps/indoc-4e3fe2b7cb1d37d2.indoc.42d2e69cb3cf60e2-cgu.3.rcgu.o" "/root/aya/target/debug/deps/indoc-4e3fe2b7cb1d37d2.0b2ljoonqdxfdg051zdpul0wc.rcgu.rmeta" "/root/aya/target/debug/deps/indoc-4e3fe2b7cb1d37d2.1hsylygegiy7qnlli60pr13rf.rcgu.o" "--as-needed" "-L" "/root/aya/target/debug/deps" "-L" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Bstatic" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libproc_macro-d2aa9b2c4004c068.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-1b60b176e45a0318.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-57b45c66d2e62c49.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libobject-0123438d987de009.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libmemchr-05f3c42b49d78619.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libaddr2line-45c700404e36c3de.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libgimli-cf5f9b37b545ed6e.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_demangle-44abbf672f21a10b.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd_detect-9b6d712a985955e4.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libhashbrown-e4ec61de4b8ef447.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_alloc-c72777eb88217fe3.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libminiz_oxide-b0a8429443e0fed6.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libadler-820055e0e9cc55a4.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libunwind-2ceb129e8b1d5c77.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcfg_if-8cd2ce184499d012.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/liblibc-ada011f709b467ef.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc-67a0423d44ab989a.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_core-15fc71c4e52ac90f.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcore-273f69939f389095.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-c8f82c72237d07e8.rlib" "-Bdynamic" "-lgcc_s" "-lc" "--eh-frame-hdr" "-z" "noexecstack" "-L" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib" "-o" "/root/aya/target/debug/deps/libindoc-4e3fe2b7cb1d37d2.so" "--gc-sections" "-shared" "-soname=libindoc-4e3fe2b7cb1d37d2.so" "-z" "relro" "-z" "now"
  = note: ld: cannot find -lgcc: No such file or directory

(full log)

lld:

77d2cb0e3483:~/aya# cargo build --target x86_64-unknown-linux-musl --config "target.x86_64-unknown-linux-musl.linker = \"lld\"" --package integration-test
[...]
error: could not compile `indoc` (lib) due to 1 previous error
warning: build failed, waiting for other jobs to finish...
error: linking with `lld` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/bin:/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/bin:/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/bin:/usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" VSLANG="1033" "lld" "-flavor" "gnu" "--version-script=/tmp/rustc2C6Ud8/list" "--no-undefined-version" "/tmp/rustc2C6Ud8/symbols.o" "/root/aya/target/debug/deps/rustversion-a51ee28261058709.rustversion.4145b995296d1182-cgu.0.rcgu.o" "/root/aya/target/debug/deps/rustversion-a51ee28261058709.rustversion.4145b995296d1182-cgu.1.rcgu.o" "/root/aya/target/debug/deps/rustversion-a51ee28261058709.rustversion.4145b995296d1182-cgu.2.rcgu.o" "/root/aya/target/debug/deps/rustversion-a51ee28261058709.rustversion.4145b995296d1182-cgu.3.rcgu.o" "/root/aya/target/debug/deps/rustversion-a51ee28261058709.rustversion.4145b995296d1182-cgu.4.rcgu.o" "/root/aya/target/debug/deps/rustversion-a51ee28261058709.rustversion.4145b995296d1182-cgu.5.rcgu.o" "/root/aya/target/debug/deps/rustversion-a51ee28261058709.rustversion.4145b995296d1182-cgu.6.rcgu.o" "/root/aya/target/debug/deps/rustversion-a51ee28261058709.rustversion.4145b995296d1182-cgu.7.rcgu.o" "/root/aya/target/debug/deps/rustversion-a51ee28261058709.rustversion.4145b995296d1182-cgu.8.rcgu.o" "/root/aya/target/debug/deps/rustversion-a51ee28261058709.f01o1e0hk9d1mcr47j33gnmo6.rcgu.rmeta" "/root/aya/target/debug/deps/rustversion-a51ee28261058709.8219qqgkv2oj66wv6vzyrwhri.rcgu.o" "--as-needed" "-L" "/root/aya/target/debug/deps" "-L" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Bstatic" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libproc_macro-d2aa9b2c4004c068.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-1b60b176e45a0318.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-57b45c66d2e62c49.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libobject-0123438d987de009.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libmemchr-05f3c42b49d78619.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libaddr2line-45c700404e36c3de.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libgimli-cf5f9b37b545ed6e.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_demangle-44abbf672f21a10b.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd_detect-9b6d712a985955e4.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libhashbrown-e4ec61de4b8ef447.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_alloc-c72777eb88217fe3.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libminiz_oxide-b0a8429443e0fed6.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libadler-820055e0e9cc55a4.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libunwind-2ceb129e8b1d5c77.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcfg_if-8cd2ce184499d012.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/liblibc-ada011f709b467ef.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc-67a0423d44ab989a.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_core-15fc71c4e52ac90f.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcore-273f69939f389095.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-c8f82c72237d07e8.rlib" "-Bdynamic" "-lgcc_s" "-lc" "--eh-frame-hdr" "-z" "noexecstack" "-L" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib" "-o" "/root/aya/target/debug/deps/librustversion-a51ee28261058709.so" "--gc-sections" "-shared" "-soname=librustversion-a51ee28261058709.so" "-z" "relro" "-z" "now"
  = note: lld: error: unable to find library -lgcc_s
          lld: error: unable to find library -lc

(full log)

rust-lld:

77d2cb0e3483:~/aya# cargo build --target x86_64-unknown-linux-musl --config "target.x86_64-unknown-linux-musl.linker = \"rust-lld\"" --package integration-test
[...]
   Compiling bytes v1.7.1
error: linking with `rust-lld` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/bin:/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/bin:/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/bin:/usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" VSLANG="1033" "rust-lld" "-flavor" "gnu" "--version-script=/tmp/rustcGf3OGU/list" "--no-undefined-version" "/tmp/rustcGf3OGU/symbols.o" "/root/aya/target/debug/deps/indoc-4e3fe2b7cb1d37d2.indoc.42d2e69cb3cf60e2-cgu.0.rcgu.o" "/root/aya/target/debug/deps/indoc-4e3fe2b7cb1d37d2.indoc.42d2e69cb3cf60e2-cgu.1.rcgu.o" "/root/aya/target/debug/deps/indoc-4e3fe2b7cb1d37d2.indoc.42d2e69cb3cf60e2-cgu.2.rcgu.o" "/root/aya/target/debug/deps/indoc-4e3fe2b7cb1d37d2.indoc.42d2e69cb3cf60e2-cgu.3.rcgu.o" "/root/aya/target/debug/deps/indoc-4e3fe2b7cb1d37d2.0b2ljoonqdxfdg051zdpul0wc.rcgu.rmeta" "/root/aya/target/debug/deps/indoc-4e3fe2b7cb1d37d2.1hsylygegiy7qnlli60pr13rf.rcgu.o" "--as-needed" "-L" "/root/aya/target/debug/deps" "-L" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Bstatic" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libproc_macro-d2aa9b2c4004c068.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-1b60b176e45a0318.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-57b45c66d2e62c49.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libobject-0123438d987de009.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libmemchr-05f3c42b49d78619.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libaddr2line-45c700404e36c3de.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libgimli-cf5f9b37b545ed6e.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_demangle-44abbf672f21a10b.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd_detect-9b6d712a985955e4.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libhashbrown-e4ec61de4b8ef447.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_alloc-c72777eb88217fe3.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libminiz_oxide-b0a8429443e0fed6.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libadler-820055e0e9cc55a4.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libunwind-2ceb129e8b1d5c77.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcfg_if-8cd2ce184499d012.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/liblibc-ada011f709b467ef.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc-67a0423d44ab989a.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_core-15fc71c4e52ac90f.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcore-273f69939f389095.rlib" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-c8f82c72237d07e8.rlib" "-Bdynamic" "-lgcc_s" "-lc" "--eh-frame-hdr" "-z" "noexecstack" "-L" "/usr/local/rustup/toolchains/1.81.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib" "-o" "/root/aya/target/debug/deps/libindoc-4e3fe2b7cb1d37d2.so" "--gc-sections" "-shared" "-soname=libindoc-4e3fe2b7cb1d37d2.so" "-z" "relro" "-z" "now"
  = note: rust-lld: error: unable to find library -lgcc_s
          rust-lld: error: unable to find library -lc

(full log)

My preference would be to be able to use rust-lld, which is currently broken.

I will try to dig in and fix it myself, but I would be thankful for any information where I could look. For now my plan is to check what -L arguments are being passed for each build and see whether they are constructed differently when using linker binaries.

[Noratrieb]

i think this is expected, you're supposed to use the C compiler driver as the linker, as it knows where to find these libraries. use -Clink-arg=-fuse-ld=lld to then make the C compiler invoke lld as the actual linker.

[vadorovsky]

Nightly on x86_64-unknown-linux-gnu recently switched to rust-lld as the default linker, so I think that "you're supposed to use the C compiler as the linker" isn't true and rust-lld is already being more and more adopted?

I was hoping to be able to use rust-lld, because it makes linking faster and makes cross-compilation easier, without necessity to install LLVM toolchain on the system.

[bjorn3]

On nightly x86_64 Linux we tell the C compiler to invoke rust-lld instead of the system linker. The C compiler still handles passing the right linker args for things like finding system libraries.

[vadorovsky]

Gotcha, thanks for clarification. I also found your another explanation here