Build and link issues on Linux hosts with newer Android NDKs

[b-spencer]

This issue was originally filed as rust-lang/cargo#7611, but @alexcrichton helped me understand that it is more likely an issue with the cc crate itself.

Problem

Using fresh installations of Rust installed on x86_64-unknown-linux-gnu (Debian 9) via the rustup method, compiling packages for Android with newer versions of the NDK does not work without overriding a number of tool paths manually.

It seems like the Android targets expect older NDK toolchain binary name patterns. It also seems like rustc expects the host linker to be able to link target binaries, which seems wrong.

Steps

1. sh rustup-init.sh (fresh install for the local user)

2. rustup target add aarch64-linux-android

3. Create the hello world example from the documentation:

$ cat Cargo.toml
[package]

name = "hello_world"
version = "0.0.1"
authors = [ "Your name <you@example.com>" ]

$ cat src/main.rs
fn main() {
    println!("Hello, world!");
}

4. Compile for the host (x86_64 Linux), and all is well:

$ cargo build
   Compiling hello_world v0.0.1 (/home/bspencer/sandbox/hello)
    Finished dev [unoptimized + debuginfo] target(s) in 0.33s
$ ./target/debug/hello_world
Hello, world!

5. Compile for aarch64-linux-android and it fails to link because it tries to use the host linker instead of the Android NDK linker:

$ cargo build --target aarch64-linux-android
   Compiling hello_world v0.0.1 (/home/bspencer/sandbox/hello)
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-Wl,--allow-multiple-definition" "-L" "/home/bspencer/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-linux-android/lib" "/home/bspencer/sandbox/hello/target/aarch64-linux-android/debug/deps/hello_world-2ab8e5cae937849a.2epep7ugdczvg2yq.rcgu.o" "/home/bspencer/sandbox/hello/target/aarch64-linux-android/debug/deps/hello_world-2ab8e5cae937849a.2w74efdyklh5hkqb.rcgu.o" "/home/bspencer/sandbox/hello/target/aarch64-linux-android/debug/deps/hello_world-2ab8e5cae937849a.442x5o9sb1humiwd.rcgu.o" "/home/bspencer/sandbox/hello/target/aarch64-linux-android/debug/deps/hello_world-2ab8e5cae937849a.4tjohv7q6sq14oh5.rcgu.o" "/home/bspencer/sandbox/hello/target/aarch64-linux-android/debug/deps/hello_world-2ab8e5cae937849a.4xzbcgkopcdgl5qw.rcgu.o" "/home/bspencer/sandbox/hello/target/aarch64-linux-android/debug/deps/hello_world-2ab8e5cae937849a.5d0w6t79x6lz02zn.rcgu.o" "-o" "/home/bspencer/sandbox/hello/target/aarch64-linux-android/debug/deps/hello_world-2ab8e5cae937849a" "/home/bspencer/sandbox/hello/target/aarch64-linux-android/debug/deps/hello_world-2ab8e5cae937849a.291xa8pa54xhw5ny.rcgu.o" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs" "-L" "/home/bspencer/sandbox/hello/target/aarch64-linux-android/debug/deps" "-L" "/home/bspencer/sandbox/hello/target/debug/deps" "-L" "/home/bspencer/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-linux-android/lib" "-Wl,--start-group" "-Wl,-Bstatic" "/home/bspencer/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-linux-android/lib/libstd-141ed0a5e0e1c2ba.rlib" "/home/bspencer/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-linux-android/lib/libpanic_unwind-a72070139220275e.rlib" "/home/bspencer/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-linux-android/lib/libhashbrown-093434daf7d99801.rlib" "/home/bspencer/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-linux-android/lib/librustc_std_workspace_alloc-24daf38551b7a03b.rlib" "/home/bspencer/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-linux-android/lib/libbacktrace-36d70d9746402ce9.rlib" "/home/bspencer/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-linux-android/lib/libbacktrace_sys-7acfc843240167a8.rlib" "/home/bspencer/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-linux-android/lib/librustc_demangle-eb2e0f5fe057b8b3.rlib" "/home/bspencer/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-linux-android/lib/libunwind-75e9ddd83715a368.rlib" "/home/bspencer/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-linux-android/lib/libcfg_if-af51e7c6fd7d1248.rlib" "/home/bspencer/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-linux-android/lib/liblibc-27f2a77b2995d98c.rlib" "/home/bspencer/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-linux-android/lib/liballoc-ad10152c26711a1e.rlib" "/home/bspencer/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-linux-android/lib/librustc_std_workspace_core-291bd2456cb6c9fe.rlib" "/home/bspencer/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-linux-android/lib/libcore-fc6e9071307a3016.rlib" "-Wl,--end-group" "/home/bspencer/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-linux-android/lib/libcompiler_builtins-ebe4001ded7f33e7.rlib" "-Wl,-Bdynamic" "-ldl" "-llog" "-lgcc" "-lc" "-lm"
  = note: /usr/bin/ld: /home/bspencer/sandbox/hello/target/aarch64-linux-android/debug/deps/hello_world-2ab8e5cae937849a.2epep7ugdczvg2yq.rcgu.o: Relocations in generic ELF (EM: 183)
          /usr/bin/ld: /home/bspencer/sandbox/hello/target/aarch64-linux-android/debug/deps/hello_world-2ab8e5cae937849a.2epep7ugdczvg2yq.rcgu.o: Relocations in generic ELF (EM: 183)
          /usr/bin/ld: /home/bspencer/sandbox/hello/target/aarch64-linux-android/debug/deps/hello_world-2ab8e5cae937849a.2epep7ugdczvg2yq.rcgu.o: Relocations in generic ELF (EM: 183)
          /usr/bin/ld: /home/bspencer/sandbox/hello/target/aarch64-linux-android/debug/deps/hello_world-2ab8e5cae937849a.2epep7ugdczvg2yq.rcgu.o: Relocations in generic ELF (EM: 183)
          /usr/bin/ld: /home/bspencer/sandbox/hello/target/aarch64-linux-android/debug/deps/hello_world-2ab8e5cae937849a.2epep7ugdczvg2yq.rcgu.o: Relocations in generic ELF (EM: 183)
          /usr/bin/ld: /home/bspencer/sandbox/hello/target/aarch64-linux-android/debug/deps/hello_world-2ab8e5cae937849a.2epep7ugdczvg2yq.rcgu.o: Relocations in generic ELF (EM: 183)
          /home/bspencer/sandbox/hello/target/aarch64-linux-android/debug/deps/hello_world-2ab8e5cae937849a.2epep7ugdczvg2yq.rcgu.o: error adding symbols: File in wrong format
          collect2: error: ld returned 1 exit status


error: aborting due to previous error

error: could not compile `hello_world`.

To learn more, run the command again with --verbose.

By specifying the location of the (r19c) NDK linker, we can avoid this problem:

$ CARGO_TARGET_AARCH64_LINUX_ANDROID_LINKER=$ANDROID_NDK_HOME/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android24-clang cargo build --target aarch64-linux-android
   Compiling hello_world v0.0.1 (/home/bspencer/sandbox/hello)
    Finished dev [unoptimized + debuginfo] target(s) in 0.34s
$ file target/aarch64-linux-android/debug/hello_world
target/aarch64-linux-android/debug/hello_world: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /system/bin/linker64, not stripped

Telling cargo and rustc where the Android NDK tools are seems reasonable, but it's odd that they default to using the system linker which isn't going to work. Using PATH isn't enough because the Android NDK doesn't call its linker (or compiler) driver cc.

6. However, if you try to use JNI support (which is necessary for many Android projects), even specifying the linker isn't enough.

Consider this trivial JNI project (which is from the docs with some parts deleted):

$ cat Cargo.toml
[package]
name = "jni"
version = "0.1.0"
authors = ["Your name <you@example.com>"]
edition = "2018"

[dependencies]
jni = "0.14.0"

[lib]
crate_type = ["cdylib"]

$ cat src/lib.rs
use jni::JNIEnv;
use jni::objects::{JClass};
#[no_mangle]
pub extern "system" fn Java_HelloWorld_hello(_env: JNIEnv,
                                             _class: JClass)
{}

It builds fine for the host, but fails when targeting Android, even with the environment variable from above:

$ CARGO_TARGET_AARCH64_LINUX_ANDROID_LINKER=$ANDROID_NDK_HOME/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android24-clang cargo build --target aarch64-linux-android
   Compiling libc v0.2.65
   Compiling cc v1.0.47
   Compiling cfg-if v0.1.10
   Compiling byteorder v1.3.2
   Compiling version_check v0.1.5
   Compiling memchr v2.2.1
   Compiling same-file v1.0.5
   Compiling rustc-demangle v0.1.16
   Compiling void v1.0.2
   Compiling log v0.4.8
   Compiling either v1.5.3
   Compiling ascii v0.9.3
   Compiling jni-sys v0.3.0
   Compiling cesu8 v1.1.0
   Compiling unreachable v1.0.0
   Compiling walkdir v2.2.9
   Compiling error-chain v0.12.1
   Compiling jni v0.14.0
   Compiling combine v3.8.1
   Compiling backtrace-sys v0.1.32
error: failed to run custom build command for `backtrace-sys v0.1.32`

Caused by:
  process didn't exit successfully: `/home/bspencer/sandbox/jni/target/debug/build/backtrace-sys-fa8161a373e78eda/build-script-build` (exit code: 1)
--- stdout
OPT_LEVEL = Some("0")
TARGET = Some("aarch64-linux-android")
HOST = Some("x86_64-unknown-linux-gnu")
CC_aarch64-linux-android = None
CC_aarch64_linux_android = None
TARGET_CC = None
CC = None
CFLAGS_aarch64-linux-android = None
CFLAGS_aarch64_linux_android = None
TARGET_CFLAGS = None
CFLAGS = None
CRATE_CC_NO_DEFAULTS = None
DEBUG = Some("true")
running: "aarch64-linux-android-clang" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "--target=aarch64-linux-android" "-Wall" "-Wextra" "-E" "src/android-api.c"

--- stderr


error occurred: Failed to find tool. Is `aarch64-linux-android-clang` installed?

It seems that cargo is now looking for the actual Android NDK C compiler. Using PATH isn't enough because even though the newer NDKs come with "already standalone" toolchains (and in fact the scripts that one used to use to copy the toolchains warn you not to, basically), the names of the compiler and linker drivers include the triplet and the API level integer.

So the next iteration is to tell cargo via a different set of environment variables where the Android NDK C compiler is:

CC_aarch64_linux_android=$ANDROID_NDK_HOME/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android24-clang \
CARGO_TARGET_AARCH64_LINUX_ANDROID_LINKER=$ANDROID_NDK_HOME/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android24-clang \
cargo build --lib --release --target aarch64-linux-android

This gets further, but then it chokes on aarch64-linux-android-ar instead:

running: "aarch64-linux-android-ar" "crs" "/home/bspencer/sandbox/jni/target/aarch64-linux-android/release/build/backtrace-sys-d902dc0510ff98a2/out/libbacktrace.a" "/home/bspencer/sandbox/jni/target/aarch64-linux-android/release/build/backtrace-sys-d902dc0510ff98a2/out/src/libbacktrace/alloc.o" "/home/bspencer/sandbox/jni/target/aarch64-linux-android/release/build/backtrace-sys-d902dc0510ff98a2/out/src/libbacktrace/dwarf.o" "/home/bspencer/sandbox/jni/target/aarch64-linux-android/release/build/backtrace-sys-d902dc0510ff98a2/out/src/libbacktrace/fileline.o" "/home/bspencer/sandbox/jni/target/aarch64-linux-android/release/build/backtrace-sys-d902dc0510ff98a2/out/src/libbacktrace/posix.o" "/home/bspencer/sandbox/jni/target/aarch64-linux-android/release/build/backtrace-sys-d902dc0510ff98a2/out/src/libbacktrace/read.o" "/home/bspencer/sandbox/jni/target/aarch64-linux-android/release/build/backtrace-sys-d902dc0510ff98a2/out/src/libbacktrace/sort.o" "/home/bspencer/sandbox/jni/target/aarch64-linux-android/release/build/backtrace-sys-d902dc0510ff98a2/out/src/libbacktrace/state.o" "/home/bspencer/sandbox/jni/target/aarch64-linux-android/release/build/backtrace-sys-d902dc0510ff98a2/out/src/libbacktrace/elf.o"

--- stderr


error occurred: Failed to find tool. Is `aarch64-linux-android-ar` installed?

This time, the PATH will help because at least cargo is trying to invoke the tool with the right name:

PATH=$ANDROID_NDK_HOME/toolchains/llvm/prebuilt/linux-x86_64/bin/:$PATH \
CC_aarch64_linux_android=$ANDROID_NDK_HOME/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android24-clang \
CARGO_TARGET_AARCH64_LINUX_ANDROID_LINKER=$ANDROID_NDK_HOME/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android24-clang \
cargo build --lib --release --target aarch64-linux-android

That's pretty awkward.

Questions

1. Have I misunderstood how to configure for Android cross targets?

2. Is there some unified way of specifying where the build tools should find the appropriate CC, AR, LD, for the target instead of using three different kinds of environment variables? Note that I purposely didn't use ~/.cargo/config or similar approaches because they didn't work for all of the required tools required by such builds, and those files can't use a variably-located Android NDK. Note how I am using $ANDROID_NDK_HOME to find the NDK in all instances.

3. Should the host linker really be used by default for cross target linking?

4. Have the Android targets been updated to be aware of the newer Android NDK toolchain naming conventions?

Thanks!

Notes

Output of cargo version:

$ cargo --version
cargo 1.39.0 (1c6ec66d5 2019-09-30)
$ rustc --version
rustc 1.39.0 (4560ea788 2019-11-04)
$ cat $ANDROID_NDK_HOME/source.properties
Pkg.Desc = Android NDK
Pkg.Revision = 19.2.5345600

[Zazck]

same issue encountered when trying to compile subparse on windows to x86_64-linux-android which depends on failure -> backtrace -> backtrace-sys -> cc
I'm using ~/.cargo/config but it seems no help at all.
after setting envvar, it compiles ok.

[togetherwithasteria]

Hiii! I now get this when trying the workaround from @b-spencer.

  = note: ld: error: unable to find library -lgcc
          clang-12: error: linker command failed with exit code 1 (use -v to see invocation)

Seems someone wants to add libgcc, on a Clang setup? Not sure if this is cc-rs's fault since there's no mention of that in the codebase.

But here is the erroring command!
https://gist.github.com/lovelyyfiaaa/fdcf25749fad5253546d5e83d266d55f

[togetherwithasteria]

It seems my problem is related to rust-mobile/ndk#189?

[plusls]

Any progress?

I found that it is a bug in is_flag_supported, when use aarch64-linux-android33-clang.cmd in windows.

https://github.com/rust-lang/rust/blob/1536ab1b383f21b38f8d49230a2aecc51daffa3d/library/unwind/build.rs#L19

If I change it to let has_unwind = true;, it will compile success and without link lgcc.