no_std + liballoc can not compile with os based toolchain caused by eh_personality required

[gngshn]

Update at 2023-01-20

The previous test that met the expected conditions was actually just a case of not testing enough to get the linker's GC optimizations out of the way. The problem is now updated to the fact that in the case of default_alloc_error_handler merged, the OS-based toolchain cannot compile the no_std + liballoc crate without using the unstable build-std feature, and the error shows up as a missing rust_eh_ personality dependency. Using OS-based toolchain to compile no_std + liballoc is intended to compile small binaries for tiny embedded linux.

---

I tried this code:

#![no_std]
#![no_main]

extern crate alloc;

use alloc::{boxed::Box, vec};
use libc_alloc::LibcAlloc;
use libc_print::std_name::println;

#[global_allocator]
static GLOBAL_ALLOC: LibcAlloc = LibcAlloc;

#[panic_handler]
fn panic(_: &core::panic::PanicInfo) -> ! {
    unsafe {
        libc::abort();
    }
}

#[no_mangle]
pub extern "C" fn main() -> i32 {
    println!("hello world!");
    let test1 = vec![1, 2, 3];
    for v in test1 {
        println!("{v}")
    }
    let test2 = Box::new(10);
    println!("{test2}");
    0
}

I expected to see this happen: This code build and run perfectly with panic=abort and non nightly features in nightly rust, and when rust 1.68 is comming, we can build this in stable rust.
We use no_std + liballoc to build small linux app in embedded linux, in production, we need stable rust.

Instead, this happened: The latest rust nightly can not build this code again(nightly-2022-12-29 is okay), it failed with this link error:

"_rust_eh_personality", referenced from:
                core::panicking::panic_nounwind_fmt::hdc7b70c1a43a8935 in libcore-d0a8c087921eb265.rlib(core-d0a8c087921eb265.core.5d748034-cgu.0.rcgu.o)

this seem caused by #106045

Meta

rustc --version --verbose:

rustc 1.68.0-nightly (0b90256ad 2023-01-13)

Backtrace

$ RUST_BACKTRACE=1 cargo build
   Compiling libc v0.2.139
   Compiling libc_alloc v1.0.4
   Compiling libc-print v0.1.20
   Compiling core_alloc v0.1.0 (/Users/gngshn/develop/rust/core_alloc)
error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/Users/gngshn/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/bin:/usr/local/opt/coreutils/libexec/gnubin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/gngshn/.cargo/bin:/usr/local/sbin:/Users/gngshn/.local/bin:/Users/gngshn/.local/go/bin:/usr/local/opt/fzf/bin" VSLANG="1033" ZERO_AR_DATE="1" "cc" "-arch" "x86_64" "-m64" "/var/folders/q6/qjg_84cs7m33457bzpwyj89c0000gn/T/rustc8tgW18/symbols.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.1e17zqd3jy504d2a.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.1eijle777i9sq0pk.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.1fe54lcf0ayzfo9i.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.1gkto627j4tsu01c.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.1krvpfd8926axeg1.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.1regfi20mw4f4nr0.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.1ymh0vbbid03fxd7.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.20xrjwb4wnpodfwk.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.236beanv78eknhkk.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.2604lu66786wcvm8.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.2e5d7pgn1zhe8xd1.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.2jkx96ov7p0k0svd.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.2mziia6eks3r7iri.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.2qq6ma07ep3fkrie.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.3bavlu7c0u97zdy1.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.3gw933f5fvrbyrdu.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.3nynhhrqjy01lq2w.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.3qhxpnb40mit3m4e.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.3u938251jwm9ry7f.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.3ze778jtnc4j14s2.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.4415dzizekl18aax.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.45ocggxfjsa6x3p.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.4ddbu8w29d6lkeu5.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.4ijt1knocvq1kql3.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.4rlcr2h5yovnam8y.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.4z4w04hvu0o9i9c6.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.5517fkyk2vfortv2.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.jfxexxow8wjy8dn.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.qqacfrqsjnmg87k.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.wk4sxdt5nszlnfu.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.zgclkfsy9t3hh6v.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.16he8t3edhiz7wic.rcgu.o" "-L" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps" "-L" "/Users/gngshn/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/liblibc_print-f9cd8051d3145ecd.rlib" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/liblibc-064686b452655b71.rlib" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/liblibc_alloc-0c85c4bf859b14f8.rlib" "/Users/gngshn/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/liballoc-c6fc9abf4e398cbb.rlib" "/Users/gngshn/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_core-1bf2f523c1bee03d.rlib" "/Users/gngshn/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libcore-d0a8c087921eb265.rlib" "/Users/gngshn/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libcompiler_builtins-19bcb300462f8eea.rlib" "-lc" "-lm" "-liconv" "-L" "/Users/gngshn/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib" "-o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4" "-Wl,-dead_strip" "-nodefaultlibs"
  = note: Undefined symbols for architecture x86_64:
            "_rust_eh_personality", referenced from:
                core::panicking::panic_nounwind_fmt::hdc7b70c1a43a8935 in libcore-d0a8c087921eb265.rlib(core-d0a8c087921eb265.core.5d748034-cgu.0.rcgu.o)
          ld: symbol(s) not found for architecture x86_64
          clang: error: linker command failed with exit code 1 (use -v to see invocation)


error: could not compile `core_alloc` due to previous error

[gngshn]

Hi @nbdd0121
I have filed a new issue

cc @RalfJung @Amanieu

[nbdd0121]

Here's one potential solution for this:

• When a panic handler is defined, in addition to rust_begin_unwind, generate a shim function rust_panic_handler_nounwind, which is a mere wrapper of the panic handler, but is defined as #[rustc_nounwind]. This function is generated in the same crate as the panic handler, not libcore.
• panic_nounwind_fmt then is changed to call this function instead of rust_begin_unwind (i.e. panic_impl lang item).

This solves the issue here because:

• panic_nounwind_fmt now does not call into any unwindable function, so it needs no personality.
• rust_panic_handler_nounwind is generated in the same crate as the panic handler, which will be -Cpanic=abort, so it also does not require personality.

This approach however does add some complexity to the codegen part.

Any thoughts, @Amanieu?

[CAD97]

Alternatively (but likely of similar complexity), would it be possible to have panic_nounwind_fmt/#[rustc_nounwind] "just" not introduce a dependency on _rust_eh_personality? Since #[rustc_nounwind] wants to catch and immediately abort on any unwind, it doesn't seem strictly necessary for it to know the Rust unwind personality. It's perfectly fine (and even desirable) for it to just catch/suspend foreign unwinds the same way as Rust unwinds (that is, abort without cleaning up the unwind) rather than inspecting personality to determine if it's a foreign unwind (e.g. to instead print the "cannot catch foreign unwind" rtabort).

I forget the exact unwinding ABI(s) atm, though, so I don't recall if it's necessary for an unwind terminating frame is still required to specify a personality even if it wants to catch all unwinds including foreign.

(It'd be good to verify the behavior of #[rustc_nounwind] when hit by a C++ (or other foreign) unwind. Not to say it's necessarily broken, but just to ensure that it isn't.)

[bjorn3]

Since #[rustc_nounwind] wants to catch and immediately abort on any unwind, it doesn't seem strictly necessary for it to know the Rust unwind personality.

The personality function is necessary to catch and abort. Any frame without a personality function is skipped entirely by the unwinder. The abort information is either in a landingpad, which execution is redirected to by the rust personality function or possibly in the future in the LSDA (language specific data area) which is interpreted by the personality function and is an opaque blob to the unwinder.

It's perfectly fine (and even desirable) for it to just catch/suspend foreign unwinds the same way as Rust unwinds (that is, abort without cleaning up the unwind)

For rust panics we never call into the unwinder in case of -Cpanic=abort. We literally swap out the function to initiate the rust panic with one which immediately aborts through the panic_abort crate. We can't replace the function to start unwinding for foreign exceptions.

[CAD97]

So yes, it's not possible to catch/terminate an unwind without specifying a personality. I wasn't certain and thought a catch-all could be defined without, but apparently I misremembered. (It's also possible that's true in some unwind ABIs but not always in general e.g. as exposed by LLVM.)

aside clarification, off-topic to OP

And to be clear, I wasn't meaning to say to change anything about how unwinds are started, just how #[rustc_nounwind] terminates unwinds (to prevent the function from unwinding). IIRC extern "C-unwind" immediately catches and aborts on a foreign unwind entering Rust when -Cpanic=abort¹, so a foreign unwind hitting a -Cpanic=abort-mode #[rustc_nounwind] is an impossibility (or at least UB); the aside was off-topic for -Cpanic=unwind² when foreign unwinds can reach it.

Footnotes

1. ... Since you say catching unwinds always requires a personality, I assume this means extern "C-unwind" still requires _rust_eh_personality even with -Cpanic=abort -Zbuild-std? That seems at least reasonable (you're explicitly asking to interface with unwinding) but still somewhat unfortunate, since no Rust unwinds will be triggered (on this side of FFI). ↩

2. Catching foreign unwinds when -Cpanic=unwind is still of course necessary but easy to do when we've got access to the full unwinding runtime. ↩

[bjorn3]

I assume this means extern "C-unwind" still requires _rust_eh_personality even with -Cpanic=abort -Zbuild-std?

For C-unwind -> Rust, yes. For Rust -> C-unwind I believe we skip aborting as no panic/exception can reach it. Or at least we can do so in the future without too much effort I think.

[RalfJung]

Ah, that's unfortunate. :/ Sorry for the breakage.

We can also revert that PR for now. I won't have the time to do follow-on work here.

[gngshn]

It seems that there is no easy solution to this problem, Since #106045 does not introduce new features or fix any bugs, just behavioral optimization. Can we just revert it first as @RalfJung said, and discuss the consistency of no_std and std panic behavior later when @RalfJung has time to follow on it?

[bjorn3]

Only if #102318 is reverted too. Otherwise default_alloc_error_handler will get stabilized in a way that allows unwinding from the alloc error handler, which we wouldn't be able to revert back to aborting by landing #106045 again.

[gngshn]

#106045 only change the behavor in no_std environment. And it seems that almost all the no_std crates set panic=abort. Just only revert #106045 seems not bad?

[bjorn3]

I guess

[gngshn]

If revert #106045 alone is not an acceptable option, then it is better to continue to wait for someone to fix this problem, and keep these two merged PRs at least allow bare metal development to use the alloc in rust 1.68 stable :)

[RalfJung]

later when @RalfJung has time to follow on it?

FWIW I don't think I would do the follow-on in the future either. This was meant to be a "quick fix". I hope someone else can take over. :)