ELF binary built linked against fake loader

[slondr]

hi, I'm compiling a Rust program on my ARM AArch64 machine which is running linux with a musl libc. Rust wants to compile the binary with the interpreter set as /lib/ld-linux-aarch64.so.1 which does not exist on my system. I do have /lib/ld-musl-aarch64.so.1 however.

As a result when I try to execute the binary built by rust in this case I just get ENOENT. In fact when I strace ./binary there is only a single syscall printed, the execve on the binary itself, which returns ENOENT. This makes me sad...

Manually loading the binary (ie running exec /lib/ld-musl-aarch64.so.1 $PWD/binary) gets me a lot farther in the execution of my program which then crashes for other reasons, presumably my fault :) But that's out of scope for this issue.

If it helps I installed Rust via rustup.

[compiler-errors]

See #82519 I think.

[bjorn3]

What linker did you tell rustc to use for AArch64? You need to use a gcc or clang for cross-compiling to aarch64+musl, not one for aarch64+glibc.

[slondr]

@bjorn3 I am not cross-compiling, I'm just running cargo build

[bjorn3]

My bad. What does rustc -vV show?

[slondr]

@bjorn3 Here you go:

rustc 1.64.0 (a55dd71d5 2022-09-19)
binary: rustc
commit-hash: a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52
commit-date: 2022-09-19
host: aarch64-unknown-linux-musl
release: 1.64.0
LLVM version: 14.0.6

[jyn514]

The issue tracker is not a support forum. For future questions please ask in one of these places:

https://discord.gg/rust-lang/
https://discord.gg/rust-lang-community/
https://users.rust-lang.org/
https://rust-lang.zulipchat.com/

I suspect your current issue is a problem with your default linker, not the compiler itself.

[slondr]

For the record, I only posted this issue here after discussing it in the rust matrix chat, where I was told (after some group investigation) that it looked like a bug in cargo or rustc.

[jyn514]

Ok. You can use cargo build -v to show the invocations, and RUSTFLAGS=--print=link-args to see what's being passed to the linker. I'd still suggest trying to build a normal C program and make sure that works ok before building a rust program.

[jbg]

I just hit this as well. To be clear, no cross-compiling is involved. Both host and target are aarch64-unknown-linux-musl.

It seems to be triggered by external C library deps that cause the binary to be dynamically linked (rather than static as would be the default for -musl targets). The resulting binary always has the wrong loader, and only works if the libc6-compat package is installed.

Here's an example of reproducing it by linking curl. In a clean alpine:3.17 container, reproduce as follows:

apk add build-base curl curl-dev pkgconf
curl https://sh.rustup.rs -sSf | sh -s -- -y
. ~/.cargo/env
cargo init --bin tester
cd tester
cargo add curl
echo 'use curl::easy::Easy; fn main() { Easy::new(); }' > src/main.rs
cargo build

Running the resulting binary fails:

# target/debug/tester
/bin/sh: target/debug/tester: not found

The loader is wrong:

# ldd target/debug/tester
	/lib/ld-linux-aarch64.so.1 (0xffffa0b61000)
	libcurl.so.4 => /usr/lib/libcurl.so.4 (0xffffa0ace000)
	libnghttp2.so.14 => /usr/lib/libnghttp2.so.14 (0xffffa0a8d000)
	libssl.so.3 => /lib/libssl.so.3 (0xffffa09e9000)
	libcrypto.so.3 => /lib/libcrypto.so.3 (0xffffa0663000)
	libbrotlidec.so.1 => /usr/lib/libbrotlidec.so.1 (0xffffa0642000)
	libz.so.1 => /lib/libz.so.1 (0xffffa0611000)
	libc.musl-aarch64.so.1 => /lib/ld-linux-aarch64.so.1 (0xffffa0b61000)
	libbrotlicommon.so.1 => /usr/lib/libbrotlicommon.so.1 (0xffffa05e0000)

# ls /lib/ld-linux-aarch64.so.1
ls: /lib/ld-linux-aarch64.so.1: No such file or directory

[jyn514]

@jbg do you have the same issue in a standalone C problem that links against curl? If so, this isn't a bug in the compiler itself. If not, you may want to talk to the owners of the curl crate on crates.io, there are various flags build scripts can pass that affect how the final program is linked. You can use cargo build -vvv curl to see build script output.

[bjorn3]

It seems to be triggered by external C library deps that cause the binary to be dynamically linked (rather than static as would be the default for -musl targets). The resulting binary always has the wrong loader, and only works if the libc6-compat package is installed.

Does setting RUSTFLAGS="-Ctarget-feature=-crt-static" fix this issue?

[jbg]

No, linking isn't broken in Alpine in general, it's only Rust that has this issue.

rust-builder-test:~$ cat > curl.c
#include <curl/curl.h>

int main(void) {
  CURL *curl = curl_easy_init();
  return 0;
}
rust-builder-test:~$ gcc curl.c -lcurl -o curl
rust-builder-test:~$ ldd curl
	/lib/ld-musl-aarch64.so.1 (0xffffbef04000)
	libcurl.so.4 => /usr/lib/libcurl.so.4 (0xffffbee50000)
	libc.musl-aarch64.so.1 => /lib/ld-musl-aarch64.so.1 (0xffffbef04000)
	libnghttp2.so.14 => /usr/lib/libnghttp2.so.14 (0xffffbee0f000)
	libssl.so.3 => /lib/libssl.so.3 (0xffffbed6b000)
	libcrypto.so.3 => /lib/libcrypto.so.3 (0xffffbe9e5000)
	libbrotlidec.so.1 => /usr/lib/libbrotlidec.so.1 (0xffffbe9c4000)
	libz.so.1 => /lib/libz.so.1 (0xffffbe993000)
	libbrotlicommon.so.1 => /usr/lib/libbrotlicommon.so.1 (0xffffbe962000)

The curl crate was just a convenient example. All Rust crates that dynamically link a C library seem to be affected. curl-sys seems to just use the pkg-config crate in its build.rs which outputs the usual rustc-link-lib lines.

[jbg]

@bjorn3 yes, setting RUSTFLAGS="-Ctarget-feature=-crt-static" resolves it.

$ RUSTFLAGS="-Ctarget-feature=-crt-static" cargo build
[...]
    Finished dev [unoptimized + debuginfo] target(s) in 10.89s
$ ldd target/debug/tester
	/lib/ld-musl-aarch64.so.1 (0xffff928d0000)
	libcurl.so.4 => /usr/lib/libcurl.so.4 (0xffff927cc000)
	libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0xffff9279b000)
	libc.musl-aarch64.so.1 => /lib/ld-musl-aarch64.so.1 (0xffff928d0000)
	libnghttp2.so.14 => /usr/lib/libnghttp2.so.14 (0xffff9275a000)
	libssl.so.3 => /lib/libssl.so.3 (0xffff926b6000)
	libcrypto.so.3 => /lib/libcrypto.so.3 (0xffff92330000)
	libbrotlidec.so.1 => /usr/lib/libbrotlidec.so.1 (0xffff9230f000)
	libz.so.1 => /lib/libz.so.1 (0xffff922de000)
	libbrotlicommon.so.1 => /usr/lib/libbrotlicommon.so.1 (0xffff922ad000)