cargo 1.56 beta hang when run inside Gentoo's sandbox

[12101111]

Not sure if this is considered a bug/regression of rust. If not, just close this issue.

#81825 break cargo run inside Gentoo's sandbox.

How to Reproduce

1. try to run sandbox cargo build on any rust project that has few dependence ( eg. rustc, Firefox or https://github.com/ogham/exa )
2. cargo hang after rustc compiling some crates.
3. Some zombie rustc or cargo processes don't exit

 16.7m   0.0   0.1   0:01.05 S  `- /home/han/.rustup/toolchains/nightly-x86_64-unknown-linux-musl/bin/cargo build --offline
  0.0m   0.0   0.0   0:00.00 Z      `- [cargo] <defunct>
  7.2m   0.0   0.0   0:00.00 S      `- /home/han/.rustup/toolchains/nightly-x86_64-unknown-linux-musl/bin/cargo build --offline
  0.0m   0.0   0.0   0:00.00 Z      `- [cargo] <defunct>
  7.3m   0.0   0.0   0:00.00 S      `- /home/han/.rustup/toolchains/nightly-x86_64-unknown-linux-musl/bin/cargo build --offline
  7.3m   0.0   0.0   0:00.00 S      `- /home/han/.rustup/toolchains/nightly-x86_64-unknown-linux-musl/bin/cargo build --offline

The bisect process:

1. Use cargo bisect-rustc with add timeout functionality for bisecting hangs  cargo-bisect-rustc#135
2. test.sh:

#!/bin/sh
export LIBGIT2_SYS_USE_PKG_CONFIG=1
export PKG_CONFIG_ALLOW_CROSS=1
sandbox cargo build --offline

3. cargo bisect-rustc --start=2021-07-30 --end=2021-09-30 --script=./test.sh -t 120
4. Result: Regression in 4e21ef2

Meta

rustc --version --verbose:

rustc 1.57.0-nightly (c02371c44 2021-10-01)
binary: rustc
commit-hash: c02371c442f811878ab3a0f5a813402b6dfd45d2
commit-date: 2021-10-01
host: x86_64-unknown-linux-musl
release: 1.57.0-nightly
LLVM version: 13.0.0

and

rustc 1.56.0-nightly (gentoo)
binary: rustc
commit-hash: unknown
commit-date: unknown
host: x86_64-unknown-linux-musl
release: 1.56.0-nightly
LLVM version: 13.0.0

(my custom build of rustc 1.56 beta3)

Backtrace of cargo

* thread #1, name = 'cargo', stop reason = signal SIGSTOP
  * frame #0: 0x00007f133a7fc005 ld-musl-x86_64.so.1`__cp_end
    frame #1: 0x00007f133a7f7d4f ld-musl-x86_64.so.1`__syscall_cp_c(nr=202, u=<unavailable>, v=<unavailable>, w=<unavailable>, x=<unavailable>, y=<unavailable>, z=<unavailable>) at pthread_cancel.c:33:6
    frame #2: 0x00007f133a7f6d9d ld-musl-x86_64.so.1`__timedwait_cp [inlined] __futex4_cp(addr=0x00007ffc7fd9fe34, op=<unavailable>, val=2, to=0x00007ffc7fd9fdb0) at __timedwait.c:24:6
    frame #3: 0x00007f133a7f6d73 ld-musl-x86_64.so.1`__timedwait_cp(addr=<unavailable>, val=2, clk=<unavailable>, at=<unavailable>, priv=<unavailable>) at __timedwait.c:52:7
    frame #4: 0x00007f133a7f8364 ld-musl-x86_64.so.1`__pthread_cond_timedwait(c=<unavailable>, m=<unavailable>, ts=0x00007ffc7fd9fef8) at pthread_cond_timedwait.c:100:9
    frame #5: 0x000055e1fdfe0f69 cargo`std::sys::unix::condvar::Condvar::wait_timeout::h23e2f7508abac320(self=0x00007f1339e367f0, mutex=0x00007f1339e01fe0, dur=<unavailable>) at condvar.rs:114:17
    frame #6: 0x000055e1fdab6ddd cargo`cargo::core::compiler::job_queue::DrainState::drain_the_queue::h8617a92b08e36d7f [inlined] std::sys_common::condvar::Condvar::wait_timeout::h66918be3196c044a(self=0x00007f1339e39930, mutex=0x00007f1339e39900, dur=<unavailable>) at condvar.rs:56:9
    frame #7: 0x000055e1fdab6db3 cargo`cargo::core::compiler::job_queue::DrainState::drain_the_queue::h8617a92b08e36d7f at condvar.rs:383:27
    frame #8: 0x000055e1fdab6db3 cargo`cargo::core::compiler::job_queue::DrainState::drain_the_queue::h8617a92b08e36d7f at condvar.rs:460:21
    frame #9: 0x000055e1fdab6d1b cargo`cargo::core::compiler::job_queue::DrainState::drain_the_queue::h8617a92b08e36d7f [inlined] cargo::util::queue::Queue$LT$T$GT$::pop::h2ea0ee746f980829(self=0x00007f1339e39900, timeout=(secs = 0, nanos = 500000000)) at queue.rs:53:35
    frame #10: 0x000055e1fdab6cff cargo`cargo::core::compiler::job_queue::DrainState::drain_the_queue::h8617a92b08e36d7f at job_queue.rs:753:23
    frame #11: 0x000055e1fdab6196 cargo`cargo::core::compiler::job_queue::DrainState::drain_the_queue::h8617a92b08e36d7f(self=DrainState @ 0x00007ffc7fda1860, cx=<unavailable>, plan=0x00007ffc7fda06e0, scope=0x00007ffc7fda0538, jobserver_helper=0x00007ffc7fda1828) at job_queue.rs:815:26
    frame #12: 0x000055e1fda72921 cargo`cargo::core::compiler::context::Context::compile::h9f54fb813dbf7813 [inlined] cargo::core::compiler::job_queue::JobQueue::execute::_$u7b$$u7b$closure$u7d$$u7d$::hf90730b55ba59c90(scope=0x00007ffc7fda0538) at job_queue.rs:523:19
    frame #13: 0x000055e1fda728ce cargo`cargo::core::compiler::context::Context::compile::h9f54fb813dbf7813 [inlined] crossbeam_utils::thread::scope::_$u7b$$u7b$closure$u7d$$u7d$::hf13fb80fde516676 at thread.rs:160:65
    frame #14: 0x000055e1fda728a4 cargo`cargo::core::compiler::context::Context::compile::h9f54fb813dbf7813 [inlined] _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h88df4760e96f8590(self=AssertUnwindSafe<crossbeam_utils::thread::scope::{closure#0}> @ 0x00007f555674c930) at unwind_safe.rs:271:9
    frame #15: 0x000055e1fda728a4 cargo`cargo::core::compiler::context::Context::compile::h9f54fb813dbf7813 at panicking.rs:403:40
    frame #16: 0x000055e1fda728a4 cargo`cargo::core::compiler::context::Context::compile::h9f54fb813dbf7813 at panicking.rs:367:19
    frame #17: 0x000055e1fda728a4 cargo`cargo::core::compiler::context::Context::compile::h9f54fb813dbf7813 [inlined] std::panic::catch_unwind::h0b5c675c73e34c87(f=AssertUnwindSafe<crossbeam_utils::thread::scope::{closure#0}> @ 0x00007f555674beb0) at panic.rs:129:14
    frame #18: 0x000055e1fda728a4 cargo`cargo::core::compiler::context::Context::compile::h9f54fb813dbf7813 at thread.rs:160:18
    frame #19: 0x000055e1fda727b4 cargo`cargo::core::compiler::context::Context::compile::h9f54fb813dbf7813 at job_queue.rs:522:9
    frame #20: 0x000055e1fda7246c cargo`cargo::core::compiler::context::Context::compile::h9f54fb813dbf7813(self=Context @ 0x00007ffc7fda1ff8, exec=0x00007ffc7fda3338) at mod.rs:172:9
    frame #21: 0x000055e1fdb8c003 cargo`cargo::ops::cargo_compile::compile_ws::h6ab50db5e4205ff8(ws=<unavailable>, options=<unavailable>, exec=0x00007ffc7fda3338) at cargo_compile.rs:289:5
    frame #22: 0x000055e1fdb8bd78 cargo`cargo::ops::cargo_compile::compile::hb9e83d2d2ee350b1 [inlined] cargo::ops::cargo_compile::compile_with_exec::hdf76cca5b8825788(ws=0x00007ffc7fda3880, options=0x00007ffc7fda3398, exec=0x00007ffc7fda3338) at cargo_compile.rs:273:5
    frame #23: 0x000055e1fdb8bd49 cargo`cargo::ops::cargo_compile::compile::hb9e83d2d2ee350b1(ws=0x00007ffc7fda3880, options=0x00007ffc7fda3398) at cargo_compile.rs:262:5
    frame #24: 0x000055e1fd7b9821 cargo`cargo::commands::build::exec::hdb65e5bc1890c5dd(config=0x00007ffc7fda43c0, args=0x00007f133a8096f8) at build.rs:70:5
    frame #25: 0x000055e1fd7a7b4b cargo`cargo::cli::main::h020e3208cf5c699b [inlined] cargo::cli::execute_subcommand::hafd2d73dcffb661a(config=0x00007ffc7fda43c0, cmd=(data_ptr = "build", length = 5), subcommand_args=0x00007f133a8096f8) at cli.rs:289:16
    frame #26: 0x000055e1fd7a7b2c cargo`cargo::cli::main::h020e3208cf5c699b(config=<unavailable>) at cli.rs:158:5
    frame #27: 0x000055e1fd7cc721 cargo`cargo::main::he0c7f80e60940d9a at main.rs:39:13
    frame #28: 0x000055e1fd7625b3 cargo`std::sys_common::backtrace::__rust_begin_short_backtrace::hc64d3bba25859aa2 [inlined] core::ops::function::FnOnce::call_once::ha569b56495e86f01((null)=<unavailable>) at function.rs:227:5
    frame #29: 0x000055e1fd7625b1 cargo`std::sys_common::backtrace::__rust_begin_short_backtrace::hc64d3bba25859aa2(f=<unavailable>) at backtrace.rs:125:18
    frame #30: 0x000055e1fd7642f9 cargo`std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::hb8b16518e98ed065 at rt.rs:63:18
    frame #31: 0x000055e1fdfdf36d cargo`std::rt::lang_start_internal::hd97130f1945ced1b [inlined] core::ops::function::impls::_$LT$impl$u20$core..ops..function..FnOnce$LT$A$GT$$u20$for$u20$$RF$F$GT$::call_once::h908ffaeb6451df95(self=<unavailable>) at function.rs:259:13
    frame #32: 0x000055e1fdfdf36a cargo`std::rt::lang_start_internal::hd97130f1945ced1b at panicking.rs:403:40
    frame #33: 0x000055e1fdfdf36a cargo`std::rt::lang_start_internal::hd97130f1945ced1b at panicking.rs:367:19
    frame #34: 0x000055e1fdfdf36a cargo`std::rt::lang_start_internal::hd97130f1945ced1b [inlined] std::panic::catch_unwind::h5113c0636a0e0138(f=<unavailable>) at panic.rs:129:14
    frame #35: 0x000055e1fdfdf36a cargo`std::rt::lang_start_internal::hd97130f1945ced1b [inlined] std::rt::lang_start_internal::_$u7b$$u7b$closure$u7d$$u7d$::h562d3655e61ef3f4 at rt.rs:45:48
    frame #36: 0x000055e1fdfdf36a cargo`std::rt::lang_start_internal::hd97130f1945ced1b at panicking.rs:403:40
    frame #37: 0x000055e1fdfdf36a cargo`std::rt::lang_start_internal::hd97130f1945ced1b at panicking.rs:367:19
    frame #38: 0x000055e1fdfdf36a cargo`std::rt::lang_start_internal::hd97130f1945ced1b [inlined] std::panic::catch_unwind::h94128c5cd6630768(f=(main = &(dyn core::ops::function::Fn<> @ 0x00007f555927e3d0)) at panic.rs:129:14
    frame #39: 0x000055e1fdfdf36a cargo`std::rt::lang_start_internal::hd97130f1945ced1b(main=&(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe) @ 0x00007f555927e3d0, argc=3, argv=0x00007ffc7fda52f8) at rt.rs:45:20
    frame #40: 0x000055e1fd7d3a3b cargo`main + 43
    frame #41: 0x00007f133a79f589 ld-musl-x86_64.so.1`libc_start_main_stage2(main=(cargo`main), argc=<unavailable>, argv=0x00007ffc7fda52f8) at __libc_start_main.c:94:7
    frame #42: 0x000055e1fd761396 cargo`_start + 22
  thread #2, name = 'cargo', stop reason = signal SIGSTOP
    frame #0: 0x00007f133a7fc005 ld-musl-x86_64.so.1`__cp_end
    frame #1: 0x00007f133a7f7d4f ld-musl-x86_64.so.1`__syscall_cp_c(nr=202, u=<unavailable>, v=<unavailable>, w=<unavailable>, x=<unavailable>, y=<unavailable>, z=<unavailable>) at pthread_cancel.c:33:6
    frame #2: 0x00007f133a7f6d9d ld-musl-x86_64.so.1`__timedwait_cp [inlined] __futex4_cp(addr=0x00007f1339ddc6f4, op=<unavailable>, val=2, to=0x0000000000000000) at __timedwait.c:24:6
    frame #3: 0x00007f133a7f6d73 ld-musl-x86_64.so.1`__timedwait_cp(addr=<unavailable>, val=2, clk=<unavailable>, at=<unavailable>, priv=<unavailable>) at __timedwait.c:52:7
    frame #4: 0x00007f133a7f8364 ld-musl-x86_64.so.1`__pthread_cond_timedwait(c=<unavailable>, m=<unavailable>, ts=0x0000000000000000) at pthread_cond_timedwait.c:100:9
    frame #5: 0x000055e1fdf9d558 cargo`std::sys_common::backtrace::__rust_begin_short_backtrace::heca2c6c8146f2d88 [inlined] std::sys::unix::condvar::Condvar::wait::h97cb58c028df68c7(self=0x00007f1339e368f0, mutex=<unavailable>) at condvar.rs:82:17
    frame #6: 0x000055e1fdf9d54c cargo`std::sys_common::backtrace::__rust_begin_short_backtrace::heca2c6c8146f2d88 [inlined] std::sys_common::condvar::Condvar::wait::hace72b24a7515a0c(self=0x00007f1339e3ac00, mutex=0x00007f1339e3abe0) at condvar.rs:44:9
    frame #7: 0x000055e1fdf9d532 cargo`std::sys_common::backtrace::__rust_begin_short_backtrace::heca2c6c8146f2d88 at condvar.rs:187:13
    frame #8: 0x000055e1fdf9d532 cargo`std::sys_common::backtrace::__rust_begin_short_backtrace::heca2c6c8146f2d88 at lib.rs:473:24
    frame #9: 0x000055e1fdf9d3c2 cargo`std::sys_common::backtrace::__rust_begin_short_backtrace::heca2c6c8146f2d88 [inlined] jobserver::imp::spawn_helper::_$u7b$$u7b$closure$u7d$$u7d$::h3ced2b0a48026e49 at unix.rs:240:9
    frame #10: 0x000055e1fdf9d3c2 cargo`std::sys_common::backtrace::__rust_begin_short_backtrace::heca2c6c8146f2d88(f=<unavailable>) at backtrace.rs:125:18
    frame #11: 0x000055e1fdf9da19 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h4d887db3b5e0c4e4 [inlined] std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h84be27402cbd26b3 at mod.rs:481:17
    frame #12: 0x000055e1fdf9da00 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h4d887db3b5e0c4e4 [inlined] _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h1755315c5044ae49 at unwind_safe.rs:271:9
    frame #13: 0x000055e1fdf9da00 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h4d887db3b5e0c4e4 [inlined] std::panicking::try::do_call::h278e56399fe643c7 at panicking.rs:403:40
    frame #14: 0x000055e1fdf9da00 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h4d887db3b5e0c4e4 [inlined] std::panicking::try::hca1de3f42269fb3b at panicking.rs:367:19
    frame #15: 0x000055e1fdf9da00 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h4d887db3b5e0c4e4 [inlined] std::panic::catch_unwind::h746387e2e744592d at panic.rs:129:14
    frame #16: 0x000055e1fdf9da00 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h4d887db3b5e0c4e4 [inlined] std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::hac2facaf9a6e368c at mod.rs:480:30
    frame #17: 0x000055e1fdf9d9b0 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h4d887db3b5e0c4e4((null)=0x00007f1339e369b0, (null)=<unavailable>) at function.rs:227:5
    frame #18: 0x000055e1fdfe7cb3 cargo`std::sys::unix::thread::Thread::new::thread_start::he0eb13837439c3a8 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h8ffa971d6f77ff65(self=Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> @ 0x00007f5559234cf0) at boxed.rs:1636:9
    frame #19: 0x000055e1fdfe7cad cargo`std::sys::unix::thread::Thread::new::thread_start::he0eb13837439c3a8 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::hf48067d9dbade9e9(self=0x00007f1339e02d50) at boxed.rs:1636:9
    frame #20: 0x000055e1fdfe7ca6 cargo`std::sys::unix::thread::Thread::new::thread_start::he0eb13837439c3a8(main=0x00007f1339e02d50) at thread.rs:106:17
    frame #21: 0x00007f133a7f955c ld-musl-x86_64.so.1`start(p=0x00007f1339ddc900) at pthread_create.c:203:17
    frame #22: 0x00007f133a7fbfcb ld-musl-x86_64.so.1`__clone + 47
  thread #3, name = 'cargo', stop reason = signal SIGSTOP
    frame #0: 0x00007f133a7fc005 ld-musl-x86_64.so.1`__cp_end
    frame #1: 0x00007f133a7f7d4f ld-musl-x86_64.so.1`__syscall_cp_c(nr=7, u=<unavailable>, v=<unavailable>, w=<unavailable>, x=<unavailable>, y=<unavailable>, z=<unavailable>) at pthread_cancel.c:33:6
    frame #2: 0x00007f133a7e36a5 ld-musl-x86_64.so.1`poll(fds=<unavailable>, n=<unavailable>, timeout=<unavailable>) at poll.c:9:9
    frame #3: 0x000055e1fdf94203 cargo`cargo_util::read2::imp::read2::h405e7286ee17620f(out_pipe=<unavailable>, err_pipe=<unavailable>, data=&mut dyn core::ops::function::FnMut<(bool, &mut alloc::vec::Vec<u8, alloc::alloc::Global>, bool), Output=()> @ 0x00007f55590e8610) at read2.rs:36:30
    frame #4: 0x000055e1fdf92284 cargo`cargo_util::process_builder::ProcessBuilder::exec_with_streaming::h35d634369e7c6306 at process_builder.rs:252:13
    frame #5: 0x000055e1fdf92188 cargo`cargo_util::process_builder::ProcessBuilder::exec_with_streaming::h35d634369e7c6306(self=<unavailable>, on_stdout_line=&mut dyn core::ops::function::FnMut<(&str), Output=core::result::Result<(), anyhow::Error>> @ 0x00007f13399be140, on_stderr_line=&mut dyn core::ops::function::FnMut<(&str), Output=core::result::Result<(), anyhow::Error>> @ 0x00007f13399be150, capture_output=<unavailable>) at process_builder.rs:248:22
    frame #6: 0x000055e1fdad7fed cargo`_$LT$cargo..core..compiler..DefaultExecutor$u20$as$u20$cargo..core..compiler..Executor$GT$::exec::hc2b9e38618027b47(self=<unavailable>, cmd=<unavailable>, _id=<unavailable>, _target=<unavailable>, _mode=<unavailable>, on_stdout_line=&mut dyn core::ops::function::FnMut<(&str), Output=core::result::Result<(), anyhow::Error>> @ 0x00007f55590ea490, on_stderr_line=&mut dyn core::ops::function::FnMut<(&str), Output=core::result::Result<(), anyhow::Error>> @ 0x00007f55590ea590) at mod.rs:133:9
    frame #7: 0x000055e1fdadd7c2 cargo`cargo::core::compiler::rustc::_$u7b$$u7b$closure$u7d$$u7d$::hd08ce3b824e17e69(state=<unavailable>) at mod.rs:318:13
    frame #8: 0x000055e1fd83759c cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h909d17ba2dd3a07c(self=Box<(dyn core::ops::function::FnOnce<(&cargo::core::compiler::job_queue::JobState), Output=core::result::Result<(), anyhow::Error>> + core::marker::Send), alloc::alloc::Global> @ 0x00007f5558fe79e0, args=(&cargo::core::compiler::job_queue::JobState) @ r15) at boxed.rs:1636:9
    frame #9: 0x000055e1fd837596 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] cargo::core::compiler::job::Work::call::he5314677cd002415(self=(inner = alloc::boxed::Box<(dyn core::ops::function::FnOnce<>, alloc::alloc::Global> @ 0x00007f5558fe7a60), tx=0x00007f13399be7c8) at job.rs:31:9
    frame #10: 0x000055e1fd837596 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] cargo::core::compiler::job::Work::then::_$u7b$$u7b$closure$u7d$$u7d$::hee839d6bf751337f(state=0x00007f13399be7c8) at job.rs:36:13
    frame #11: 0x000055e1fd83758f cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f((null)=0x00007f1339e253e0, (null)=(&cargo::core::compiler::job_queue::JobState) @ r15) at function.rs:227:5
    frame #12: 0x000055e1fd83759c cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h909d17ba2dd3a07c(self=Box<(dyn core::ops::function::FnOnce<(&cargo::core::compiler::job_queue::JobState), Output=core::result::Result<(), anyhow::Error>> + core::marker::Send), alloc::alloc::Global> @ 0x00007f5558fe7a20, args=(&cargo::core::compiler::job_queue::JobState) @ r15) at boxed.rs:1636:9
    frame #13: 0x000055e1fd837596 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] cargo::core::compiler::job::Work::call::he5314677cd002415(self=(inner = alloc::boxed::Box<(dyn core::ops::function::FnOnce<>, alloc::alloc::Global> @ 0x00007f5558fe7720), tx=0x00007f13399be7c8) at job.rs:31:9
    frame #14: 0x000055e1fd837596 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] cargo::core::compiler::job::Work::then::_$u7b$$u7b$closure$u7d$$u7d$::hee839d6bf751337f(state=0x00007f13399be7c8) at job.rs:36:13
    frame #15: 0x000055e1fd83758f cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f((null)=0x00007f1339e25410, (null)=(&cargo::core::compiler::job_queue::JobState) @ r15) at function.rs:227:5
    frame #16: 0x000055e1fdabb099 cargo`cargo::core::compiler::job_queue::DrainState::run::_$u7b$$u7b$closure$u7d$$u7d$::he62fa79974c9987b [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h909d17ba2dd3a07c(self=Box<(dyn core::ops::function::FnOnce<(&cargo::core::compiler::job_queue::JobState), Output=core::result::Result<(), anyhow::Error>> + core::marker::Send), alloc::alloc::Global> @ 0x00007f5558fe8330, args=(&cargo::core::compiler::job_queue::JobState) @ r15) at boxed.rs:1636:9
    frame #17: 0x000055e1fdabb093 cargo`cargo::core::compiler::job_queue::DrainState::run::_$u7b$$u7b$closure$u7d$$u7d$::he62fa79974c9987b [inlined] cargo::core::compiler::job::Work::call::he5314677cd002415(self=(inner = alloc::boxed::Box<(dyn core::ops::function::FnOnce<>, alloc::alloc::Global> @ 0x00007f5558fe83b0), tx=0x00007f13399be7c8) at job.rs:31:9
    frame #18: 0x000055e1fdabb093 cargo`cargo::core::compiler::job_queue::DrainState::run::_$u7b$$u7b$closure$u7d$$u7d$::he62fa79974c9987b [inlined] cargo::core::compiler::job::Job::run::heb80c1616b1f170f(state=0x00007f13399be7c8) at job.rs:62:9
    frame #19: 0x000055e1fdabb093 cargo`cargo::core::compiler::job_queue::DrainState::run::_$u7b$$u7b$closure$u7d$$u7d$::he62fa79974c9987b(state=JobState @ 0x00007f13399be7c8) at job_queue.rs:1040:34
    frame #20: 0x000055e1fd83904f cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hda7669d323da0eaf [inlined] cargo::core::compiler::job_queue::DrainState::run::_$u7b$$u7b$closure$u7d$$u7d$::hfe68ac7b45ac03ab at job_queue.rs:1098:21
    frame #21: 0x000055e1fd838ffd cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hda7669d323da0eaf at thread.rs:437:31
    frame #22: 0x000055e1fd838fd0 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hda7669d323da0eaf((null)=0x00007f1339e3a630, (null)=<unavailable>) at function.rs:227:5
    frame #23: 0x000055e1fd81863e cargo`std::sys_common::backtrace::__rust_begin_short_backtrace::h24259d003811a78b [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::ha48699e8e8187cbd(self=Box<(dyn core::ops::function::FnOnce<(), Output=()> + core::marker::Send), alloc::alloc::Global> @ 0x00007f5558fe8b30) at boxed.rs:1636:9
    frame #24: 0x000055e1fd81863b cargo`std::sys_common::backtrace::__rust_begin_short_backtrace::h24259d003811a78b [inlined] crossbeam_utils::thread::ScopedThreadBuilder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::hbf07fd2ee35affe3 at thread.rs:449:44
    frame #25: 0x000055e1fd81863b cargo`std::sys_common::backtrace::__rust_begin_short_backtrace::h24259d003811a78b(f=(closure = alloc::boxed::Box<(dyn core::ops::function::FnOnce<>, alloc::alloc::Global> @ 0x00007f5558fe8df0)) at backtrace.rs:125:18
    frame #26: 0x000055e1fd8393ab cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 [inlined] std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h77f48923b3b16330 at mod.rs:481:17
    frame #27: 0x000055e1fd8393a6 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 [inlined] _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::hcf85bf374106e3c6(self=<unavailable>) at unwind_safe.rs:271:9
    frame #28: 0x000055e1fd8393a6 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 at panicking.rs:403:40
    frame #29: 0x000055e1fd8393a6 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 at panicking.rs:367:19
    frame #30: 0x000055e1fd8393a6 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 [inlined] std::panic::catch_unwind::hfac1c3b1661d60a6(f=<unavailable>) at panic.rs:129:14
    frame #31: 0x000055e1fd8393a6 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 [inlined] std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::hdfc489316504bc38 at mod.rs:480:30
    frame #32: 0x000055e1fd839350 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3((null)=0x00007f1339e024f0, (null)=<unavailable>) at function.rs:227:5
    frame #33: 0x000055e1fdfe7cb3 cargo`std::sys::unix::thread::Thread::new::thread_start::he0eb13837439c3a8 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h8ffa971d6f77ff65(self=Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> @ 0x00007f5558fe8af0) at boxed.rs:1636:9
    frame #34: 0x000055e1fdfe7cad cargo`std::sys::unix::thread::Thread::new::thread_start::he0eb13837439c3a8 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::hf48067d9dbade9e9(self=0x00007f1339e02df0) at boxed.rs:1636:9
    frame #35: 0x000055e1fdfe7ca6 cargo`std::sys::unix::thread::Thread::new::thread_start::he0eb13837439c3a8(main=0x00007f1339e02df0) at thread.rs:106:17
    frame #36: 0x00007f133a7f955c ld-musl-x86_64.so.1`start(p=0x00007f13399be900) at pthread_create.c:203:17
    frame #37: 0x00007f133a7fbfcb ld-musl-x86_64.so.1`__clone + 47
  thread #4, name = 'cargo', stop reason = signal SIGSTOP
    frame #0: 0x00007f133a7fc005 ld-musl-x86_64.so.1`__cp_end
    frame #1: 0x00007f133a7f7d4f ld-musl-x86_64.so.1`__syscall_cp_c(nr=7, u=<unavailable>, v=<unavailable>, w=<unavailable>, x=<unavailable>, y=<unavailable>, z=<unavailable>) at pthread_cancel.c:33:6
    frame #2: 0x00007f133a7e36a5 ld-musl-x86_64.so.1`poll(fds=<unavailable>, n=<unavailable>, timeout=<unavailable>) at poll.c:9:9
    frame #3: 0x000055e1fdf94203 cargo`cargo_util::read2::imp::read2::h405e7286ee17620f(out_pipe=<unavailable>, err_pipe=<unavailable>, data=&mut dyn core::ops::function::FnMut<(bool, &mut alloc::vec::Vec<u8, alloc::alloc::Global>, bool), Output=()> @ 0x00007f5558fe8e50) at read2.rs:36:30
    frame #4: 0x000055e1fdf92284 cargo`cargo_util::process_builder::ProcessBuilder::exec_with_streaming::h35d634369e7c6306 at process_builder.rs:252:13
    frame #5: 0x000055e1fdf92188 cargo`cargo_util::process_builder::ProcessBuilder::exec_with_streaming::h35d634369e7c6306(self=<unavailable>, on_stdout_line=&mut dyn core::ops::function::FnMut<(&str), Output=core::result::Result<(), anyhow::Error>> @ 0x00007f13397bb140, on_stderr_line=&mut dyn core::ops::function::FnMut<(&str), Output=core::result::Result<(), anyhow::Error>> @ 0x00007f13397bb150, capture_output=<unavailable>) at process_builder.rs:248:22
    frame #6: 0x000055e1fdad7fed cargo`_$LT$cargo..core..compiler..DefaultExecutor$u20$as$u20$cargo..core..compiler..Executor$GT$::exec::hc2b9e38618027b47(self=<unavailable>, cmd=<unavailable>, _id=<unavailable>, _target=<unavailable>, _mode=<unavailable>, on_stdout_line=&mut dyn core::ops::function::FnMut<(&str), Output=core::result::Result<(), anyhow::Error>> @ 0x00007f5558fe8d10, on_stderr_line=&mut dyn core::ops::function::FnMut<(&str), Output=core::result::Result<(), anyhow::Error>> @ 0x00007f5558fe8db0) at mod.rs:133:9
    frame #7: 0x000055e1fdadd7c2 cargo`cargo::core::compiler::rustc::_$u7b$$u7b$closure$u7d$$u7d$::hd08ce3b824e17e69(state=<unavailable>) at mod.rs:318:13
    frame #8: 0x000055e1fd83759c cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h909d17ba2dd3a07c(self=Box<(dyn core::ops::function::FnOnce<(&cargo::core::compiler::job_queue::JobState), Output=core::result::Result<(), anyhow::Error>> + core::marker::Send), alloc::alloc::Global> @ 0x00007f5558fe8e70, args=(&cargo::core::compiler::job_queue::JobState) @ r15) at boxed.rs:1636:9
    frame #9: 0x000055e1fd837596 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] cargo::core::compiler::job::Work::call::he5314677cd002415(self=(inner = alloc::boxed::Box<(dyn core::ops::function::FnOnce<>, alloc::alloc::Global> @ 0x00007f5558fe8d10), tx=0x00007f13397bb7c8) at job.rs:31:9
    frame #10: 0x000055e1fd837596 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] cargo::core::compiler::job::Work::then::_$u7b$$u7b$closure$u7d$$u7d$::hee839d6bf751337f(state=0x00007f13397bb7c8) at job.rs:36:13
    frame #11: 0x000055e1fd83758f cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f((null)=0x00007f1339dfece0, (null)=(&cargo::core::compiler::job_queue::JobState) @ r15) at function.rs:227:5
    frame #12: 0x000055e1fd83759c cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h909d17ba2dd3a07c(self=Box<(dyn core::ops::function::FnOnce<(&cargo::core::compiler::job_queue::JobState), Output=core::result::Result<(), anyhow::Error>> + core::marker::Send), alloc::alloc::Global> @ 0x00007f5558fe8e50, args=(&cargo::core::compiler::job_queue::JobState) @ r15) at boxed.rs:1636:9
    frame #13: 0x000055e1fd837596 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] cargo::core::compiler::job::Work::call::he5314677cd002415(self=(inner = alloc::boxed::Box<(dyn core::ops::function::FnOnce<>, alloc::alloc::Global> @ 0x00007f5558fe8b10), tx=0x00007f13397bb7c8) at job.rs:31:9
    frame #14: 0x000055e1fd837596 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] cargo::core::compiler::job::Work::then::_$u7b$$u7b$closure$u7d$$u7d$::hee839d6bf751337f(state=0x00007f13397bb7c8) at job.rs:36:13
    frame #15: 0x000055e1fd83758f cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f((null)=0x00007f1339dfed10, (null)=(&cargo::core::compiler::job_queue::JobState) @ r15) at function.rs:227:5
    frame #16: 0x000055e1fdabb099 cargo`cargo::core::compiler::job_queue::DrainState::run::_$u7b$$u7b$closure$u7d$$u7d$::he62fa79974c9987b [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h909d17ba2dd3a07c(self=Box<(dyn core::ops::function::FnOnce<(&cargo::core::compiler::job_queue::JobState), Output=core::result::Result<(), anyhow::Error>> + core::marker::Send), alloc::alloc::Global> @ 0x00007f5558fe8e30, args=(&cargo::core::compiler::job_queue::JobState) @ r15) at boxed.rs:1636:9
    frame #17: 0x000055e1fdabb093 cargo`cargo::core::compiler::job_queue::DrainState::run::_$u7b$$u7b$closure$u7d$$u7d$::he62fa79974c9987b [inlined] cargo::core::compiler::job::Work::call::he5314677cd002415(self=(inner = alloc::boxed::Box<(dyn core::ops::function::FnOnce<>, alloc::alloc::Global> @ 0x00007f5558fe8af0), tx=0x00007f13397bb7c8) at job.rs:31:9
    frame #18: 0x000055e1fdabb093 cargo`cargo::core::compiler::job_queue::DrainState::run::_$u7b$$u7b$closure$u7d$$u7d$::he62fa79974c9987b [inlined] cargo::core::compiler::job::Job::run::heb80c1616b1f170f(state=0x00007f13397bb7c8) at job.rs:62:9
    frame #19: 0x000055e1fdabb093 cargo`cargo::core::compiler::job_queue::DrainState::run::_$u7b$$u7b$closure$u7d$$u7d$::he62fa79974c9987b(state=JobState @ 0x00007f13397bb7c8) at job_queue.rs:1040:34
    frame #20: 0x000055e1fd83904f cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hda7669d323da0eaf [inlined] cargo::core::compiler::job_queue::DrainState::run::_$u7b$$u7b$closure$u7d$$u7d$::hfe68ac7b45ac03ab at job_queue.rs:1098:21
    frame #21: 0x000055e1fd838ffd cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hda7669d323da0eaf at thread.rs:437:31
    frame #22: 0x000055e1fd838fd0 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hda7669d323da0eaf((null)=0x00007f1339e3a7c0, (null)=<unavailable>) at function.rs:227:5
    frame #23: 0x000055e1fd81863e cargo`std::sys_common::backtrace::__rust_begin_short_backtrace::h24259d003811a78b [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::ha48699e8e8187cbd(self=Box<(dyn core::ops::function::FnOnce<(), Output=()> + core::marker::Send), alloc::alloc::Global> @ 0x00007f5558fe8e30) at boxed.rs:1636:9
    frame #24: 0x000055e1fd81863b cargo`std::sys_common::backtrace::__rust_begin_short_backtrace::h24259d003811a78b [inlined] crossbeam_utils::thread::ScopedThreadBuilder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::hbf07fd2ee35affe3 at thread.rs:449:44
    frame #25: 0x000055e1fd81863b cargo`std::sys_common::backtrace::__rust_begin_short_backtrace::h24259d003811a78b(f=(closure = alloc::boxed::Box<(dyn core::ops::function::FnOnce<>, alloc::alloc::Global> @ 0x00007f5558fe8af0)) at backtrace.rs:125:18
    frame #26: 0x000055e1fd8393ab cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 [inlined] std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h77f48923b3b16330 at mod.rs:481:17
    frame #27: 0x000055e1fd8393a6 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 [inlined] _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::hcf85bf374106e3c6(self=<unavailable>) at unwind_safe.rs:271:9
    frame #28: 0x000055e1fd8393a6 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 at panicking.rs:403:40
    frame #29: 0x000055e1fd8393a6 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 at panicking.rs:367:19
    frame #30: 0x000055e1fd8393a6 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 [inlined] std::panic::catch_unwind::hfac1c3b1661d60a6(f=<unavailable>) at panic.rs:129:14
    frame #31: 0x000055e1fd8393a6 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 [inlined] std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::hdfc489316504bc38 at mod.rs:480:30
    frame #32: 0x000055e1fd839350 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3((null)=0x00007f1339e02430, (null)=<unavailable>) at function.rs:227:5
    frame #33: 0x000055e1fdfe7cb3 cargo`std::sys::unix::thread::Thread::new::thread_start::he0eb13837439c3a8 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h8ffa971d6f77ff65(self=Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> @ 0x00007f5558fe8e30) at boxed.rs:1636:9
    frame #34: 0x000055e1fdfe7cad cargo`std::sys::unix::thread::Thread::new::thread_start::he0eb13837439c3a8 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::hf48067d9dbade9e9(self=0x00007f1339e02ad0) at boxed.rs:1636:9
    frame #35: 0x000055e1fdfe7ca6 cargo`std::sys::unix::thread::Thread::new::thread_start::he0eb13837439c3a8(main=0x00007f1339e02ad0) at thread.rs:106:17
    frame #36: 0x00007f133a7f955c ld-musl-x86_64.so.1`start(p=0x00007f13397bb900) at pthread_create.c:203:17
    frame #37: 0x00007f133a7fbfcb ld-musl-x86_64.so.1`__clone + 47
  thread #5, name = 'cargo', stop reason = signal SIGSTOP
    frame #0: 0x00007f133a7fc005 ld-musl-x86_64.so.1`__cp_end
    frame #1: 0x00007f133a7f7d4f ld-musl-x86_64.so.1`__syscall_cp_c(nr=0, u=<unavailable>, v=<unavailable>, w=<unavailable>, x=<unavailable>, y=<unavailable>, z=<unavailable>) at pthread_cancel.c:33:6
    frame #2: 0x00007f133a800dff ld-musl-x86_64.so.1`read(fd=<unavailable>, buf=<unavailable>, count=<unavailable>) at read.c:6:9
    frame #3: 0x000055e1fdfe883c cargo`std::sys::unix::process::process_inner::_$LT$impl$u20$std..sys..unix..process..process_common..Command$GT$::spawn::h1d6f77b3e0d6cc67 [inlined] std::sys::unix::fd::FileDesc::read::ha60ab583a50f0955(buf=(data_ptr = "", length = 8)) at fd.rs:65:13
    frame #4: 0x000055e1fdfe882a cargo`std::sys::unix::process::process_inner::_$LT$impl$u20$std..sys..unix..process..process_common..Command$GT$::spawn::h1d6f77b3e0d6cc67 [inlined] std::sys::unix::pipe::AnonPipe::read::he8533efcd2d11395(buf=(data_ptr = "", length = 8)) at pipe.rs:49:9
    frame #5: 0x000055e1fdfe882a cargo`std::sys::unix::process::process_inner::_$LT$impl$u20$std..sys..unix..process..process_common..Command$GT$::spawn::h1d6f77b3e0d6cc67(self=0x00007f13395a4180, default=Stdio @ 0x00007f13395a3c60, needs_stdin=<unavailable>) at process_unix.rs:107:19
    frame #6: 0x000055e1fdfd988c cargo`std::process::Command::spawn::ha9bec88ec3740324(self=<unavailable>) at process.rs:879:9
    frame #7: 0x000055e1fdf9218e cargo`cargo_util::process_builder::ProcessBuilder::exec_with_streaming::h35d634369e7c6306 [inlined] cargo_util::process_builder::ProcessBuilder::exec_with_streaming::_$u7b$$u7b$closure$u7d$$u7d$::hee5747160a64ba2c at process_builder.rs:249:29
    frame #8: 0x000055e1fdf92188 cargo`cargo_util::process_builder::ProcessBuilder::exec_with_streaming::h35d634369e7c6306(self=<unavailable>, on_stdout_line=&mut dyn core::ops::function::FnMut<(&str), Output=core::result::Result<(), anyhow::Error>> @ 0x00007f13395a4140, on_stderr_line=&mut dyn core::ops::function::FnMut<(&str), Output=core::result::Result<(), anyhow::Error>> @ 0x00007f13395a4150, capture_output=<unavailable>) at process_builder.rs:248:22
    frame #9: 0x000055e1fdad7fed cargo`_$LT$cargo..core..compiler..DefaultExecutor$u20$as$u20$cargo..core..compiler..Executor$GT$::exec::hc2b9e38618027b47(self=<unavailable>, cmd=<unavailable>, _id=<unavailable>, _target=<unavailable>, _mode=<unavailable>, on_stdout_line=&mut dyn core::ops::function::FnMut<(&str), Output=core::result::Result<(), anyhow::Error>> @ 0x00007f5558e536f0, on_stderr_line=&mut dyn core::ops::function::FnMut<(&str), Output=core::result::Result<(), anyhow::Error>> @ 0x00007f5558e536f0) at mod.rs:133:9
    frame #10: 0x000055e1fdadd7c2 cargo`cargo::core::compiler::rustc::_$u7b$$u7b$closure$u7d$$u7d$::hd08ce3b824e17e69(state=<unavailable>) at mod.rs:318:13
    frame #11: 0x000055e1fd83759c cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h909d17ba2dd3a07c(self=Box<(dyn core::ops::function::FnOnce<(&cargo::core::compiler::job_queue::JobState), Output=core::result::Result<(), anyhow::Error>> + core::marker::Send), alloc::alloc::Global> @ 0x00007f5558e536f0, args=(&cargo::core::compiler::job_queue::JobState) @ r15) at boxed.rs:1636:9
    frame #12: 0x000055e1fd837596 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] cargo::core::compiler::job::Work::call::he5314677cd002415(self=(inner = alloc::boxed::Box<(dyn core::ops::function::FnOnce<>, alloc::alloc::Global> @ 0x00007f5558e53650), tx=0x00007f13395a47c8) at job.rs:31:9
    frame #13: 0x000055e1fd837596 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] cargo::core::compiler::job::Work::then::_$u7b$$u7b$closure$u7d$$u7d$::hee839d6bf751337f(state=0x00007f13395a47c8) at job.rs:36:13
    frame #14: 0x000055e1fd83758f cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f((null)=0x00007f1339dfc1f0, (null)=(&cargo::core::compiler::job_queue::JobState) @ r15) at function.rs:227:5
    frame #15: 0x000055e1fd83759c cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h909d17ba2dd3a07c(self=Box<(dyn core::ops::function::FnOnce<(&cargo::core::compiler::job_queue::JobState), Output=core::result::Result<(), anyhow::Error>> + core::marker::Send), alloc::alloc::Global> @ 0x00007f5558e53650, args=(&cargo::core::compiler::job_queue::JobState) @ r15) at boxed.rs:1636:9
    frame #16: 0x000055e1fd837596 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] cargo::core::compiler::job::Work::call::he5314677cd002415(self=(inner = alloc::boxed::Box<(dyn core::ops::function::FnOnce<>, alloc::alloc::Global> @ 0x00007f5558e53650), tx=0x00007f13395a47c8) at job.rs:31:9
    frame #17: 0x000055e1fd837596 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] cargo::core::compiler::job::Work::then::_$u7b$$u7b$closure$u7d$$u7d$::hee839d6bf751337f(state=0x00007f13395a47c8) at job.rs:36:13
    frame #18: 0x000055e1fd83758f cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f((null)=0x00007f1339dfb110, (null)=(&cargo::core::compiler::job_queue::JobState) @ r15) at function.rs:227:5
    frame #19: 0x000055e1fdabb099 cargo`cargo::core::compiler::job_queue::DrainState::run::_$u7b$$u7b$closure$u7d$$u7d$::he62fa79974c9987b [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h909d17ba2dd3a07c(self=Box<(dyn core::ops::function::FnOnce<(&cargo::core::compiler::job_queue::JobState), Output=core::result::Result<(), anyhow::Error>> + core::marker::Send), alloc::alloc::Global> @ 0x00007f5558e53650, args=(&cargo::core::compiler::job_queue::JobState) @ r15) at boxed.rs:1636:9
    frame #20: 0x000055e1fdabb093 cargo`cargo::core::compiler::job_queue::DrainState::run::_$u7b$$u7b$closure$u7d$$u7d$::he62fa79974c9987b [inlined] cargo::core::compiler::job::Work::call::he5314677cd002415(self=(inner = alloc::boxed::Box<(dyn core::ops::function::FnOnce<>, alloc::alloc::Global> @ 0x00007f5558e53650), tx=0x00007f13395a47c8) at job.rs:31:9
    frame #21: 0x000055e1fdabb093 cargo`cargo::core::compiler::job_queue::DrainState::run::_$u7b$$u7b$closure$u7d$$u7d$::he62fa79974c9987b [inlined] cargo::core::compiler::job::Job::run::heb80c1616b1f170f(state=0x00007f13395a47c8) at job.rs:62:9
    frame #22: 0x000055e1fdabb093 cargo`cargo::core::compiler::job_queue::DrainState::run::_$u7b$$u7b$closure$u7d$$u7d$::he62fa79974c9987b(state=JobState @ 0x00007f13395a47c8) at job_queue.rs:1040:34
    frame #23: 0x000055e1fd83904f cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hda7669d323da0eaf [inlined] cargo::core::compiler::job_queue::DrainState::run::_$u7b$$u7b$closure$u7d$$u7d$::hfe68ac7b45ac03ab at job_queue.rs:1098:21
    frame #24: 0x000055e1fd838ffd cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hda7669d323da0eaf at thread.rs:437:31
    frame #25: 0x000055e1fd838fd0 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hda7669d323da0eaf((null)=0x00007f1339e3ab80, (null)=<unavailable>) at function.rs:227:5
    frame #26: 0x000055e1fd81863e cargo`std::sys_common::backtrace::__rust_begin_short_backtrace::h24259d003811a78b [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::ha48699e8e8187cbd(self=Box<(dyn core::ops::function::FnOnce<(), Output=()> + core::marker::Send), alloc::alloc::Global> @ 0x00007f5558e543d0) at boxed.rs:1636:9
    frame #27: 0x000055e1fd81863b cargo`std::sys_common::backtrace::__rust_begin_short_backtrace::h24259d003811a78b [inlined] crossbeam_utils::thread::ScopedThreadBuilder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::hbf07fd2ee35affe3 at thread.rs:449:44
    frame #28: 0x000055e1fd81863b cargo`std::sys_common::backtrace::__rust_begin_short_backtrace::h24259d003811a78b(f=(closure = alloc::boxed::Box<(dyn core::ops::function::FnOnce<>, alloc::alloc::Global> @ 0x00007f5558e53bf0)) at backtrace.rs:125:18
    frame #29: 0x000055e1fd8393ab cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 [inlined] std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h77f48923b3b16330 at mod.rs:481:17
    frame #30: 0x000055e1fd8393a6 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 [inlined] _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::hcf85bf374106e3c6(self=<unavailable>) at unwind_safe.rs:271:9
    frame #31: 0x000055e1fd8393a6 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 at panicking.rs:403:40
    frame #32: 0x000055e1fd8393a6 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 at panicking.rs:367:19
    frame #33: 0x000055e1fd8393a6 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 [inlined] std::panic::catch_unwind::hfac1c3b1661d60a6(f=<unavailable>) at panic.rs:129:14
    frame #34: 0x000055e1fd8393a6 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 [inlined] std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::hdfc489316504bc38 at mod.rs:480:30
    frame #35: 0x000055e1fd839350 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3((null)=0x00007f13395a8110, (null)=<unavailable>) at function.rs:227:5
    frame #36: 0x000055e1fdfe7cb3 cargo`std::sys::unix::thread::Thread::new::thread_start::he0eb13837439c3a8 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h8ffa971d6f77ff65(self=Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> @ 0x00007f5558e54bb0) at boxed.rs:1636:9
    frame #37: 0x000055e1fdfe7cad cargo`std::sys::unix::thread::Thread::new::thread_start::he0eb13837439c3a8 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::hf48067d9dbade9e9(self=0x00007f1339e02b50) at boxed.rs:1636:9
    frame #38: 0x000055e1fdfe7ca6 cargo`std::sys::unix::thread::Thread::new::thread_start::he0eb13837439c3a8(main=0x00007f1339e02b50) at thread.rs:106:17
    frame #39: 0x00007f133a7f955c ld-musl-x86_64.so.1`start(p=0x00007f13395a4900) at pthread_create.c:203:17
    frame #40: 0x00007f133a7fbfcb ld-musl-x86_64.so.1`__clone + 47
  thread #6, name = 'cargo', stop reason = signal SIGSTOP
    frame #0: 0x00007f133a7fc005 ld-musl-x86_64.so.1`__cp_end
    frame #1: 0x00007f133a7f7d4f ld-musl-x86_64.so.1`__syscall_cp_c(nr=0, u=<unavailable>, v=<unavailable>, w=<unavailable>, x=<unavailable>, y=<unavailable>, z=<unavailable>) at pthread_cancel.c:33:6
    frame #2: 0x00007f133a800dff ld-musl-x86_64.so.1`read(fd=<unavailable>, buf=<unavailable>, count=<unavailable>) at read.c:6:9
    frame #3: 0x000055e1fdfe883c cargo`std::sys::unix::process::process_inner::_$LT$impl$u20$std..sys..unix..process..process_common..Command$GT$::spawn::h1d6f77b3e0d6cc67 [inlined] std::sys::unix::fd::FileDesc::read::ha60ab583a50f0955(buf=(data_ptr = "", length = 8)) at fd.rs:65:13
    frame #4: 0x000055e1fdfe882a cargo`std::sys::unix::process::process_inner::_$LT$impl$u20$std..sys..unix..process..process_common..Command$GT$::spawn::h1d6f77b3e0d6cc67 [inlined] std::sys::unix::pipe::AnonPipe::read::he8533efcd2d11395(buf=(data_ptr = "", length = 8)) at pipe.rs:49:9
    frame #5: 0x000055e1fdfe882a cargo`std::sys::unix::process::process_inner::_$LT$impl$u20$std..sys..unix..process..process_common..Command$GT$::spawn::h1d6f77b3e0d6cc67(self=0x00007f133939f180, default=Stdio @ 0x00007f133939ec60, needs_stdin=<unavailable>) at process_unix.rs:107:19
    frame #6: 0x000055e1fdfd988c cargo`std::process::Command::spawn::ha9bec88ec3740324(self=<unavailable>) at process.rs:879:9
    frame #7: 0x000055e1fdf9218e cargo`cargo_util::process_builder::ProcessBuilder::exec_with_streaming::h35d634369e7c6306 [inlined] cargo_util::process_builder::ProcessBuilder::exec_with_streaming::_$u7b$$u7b$closure$u7d$$u7d$::hee5747160a64ba2c at process_builder.rs:249:29
    frame #8: 0x000055e1fdf92188 cargo`cargo_util::process_builder::ProcessBuilder::exec_with_streaming::h35d634369e7c6306(self=<unavailable>, on_stdout_line=&mut dyn core::ops::function::FnMut<(&str), Output=core::result::Result<(), anyhow::Error>> @ 0x00007f133939f140, on_stderr_line=&mut dyn core::ops::function::FnMut<(&str), Output=core::result::Result<(), anyhow::Error>> @ 0x00007f133939f150, capture_output=<unavailable>) at process_builder.rs:248:22
    frame #9: 0x000055e1fdad7fed cargo`_$LT$cargo..core..compiler..DefaultExecutor$u20$as$u20$cargo..core..compiler..Executor$GT$::exec::hc2b9e38618027b47(self=<unavailable>, cmd=<unavailable>, _id=<unavailable>, _target=<unavailable>, _mode=<unavailable>, on_stdout_line=&mut dyn core::ops::function::FnMut<(&str), Output=core::result::Result<(), anyhow::Error>> @ 0x00007f5558e536f0, on_stderr_line=&mut dyn core::ops::function::FnMut<(&str), Output=core::result::Result<(), anyhow::Error>> @ 0x00007f5558e536f0) at mod.rs:133:9
    frame #10: 0x000055e1fdadd7c2 cargo`cargo::core::compiler::rustc::_$u7b$$u7b$closure$u7d$$u7d$::hd08ce3b824e17e69(state=<unavailable>) at mod.rs:318:13
    frame #11: 0x000055e1fd83759c cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h909d17ba2dd3a07c(self=Box<(dyn core::ops::function::FnOnce<(&cargo::core::compiler::job_queue::JobState), Output=core::result::Result<(), anyhow::Error>> + core::marker::Send), alloc::alloc::Global> @ 0x00007f5558e53650, args=(&cargo::core::compiler::job_queue::JobState) @ r15) at boxed.rs:1636:9
    frame #12: 0x000055e1fd837596 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] cargo::core::compiler::job::Work::call::he5314677cd002415(self=(inner = alloc::boxed::Box<(dyn core::ops::function::FnOnce<>, alloc::alloc::Global> @ 0x00007f5558e536f0), tx=0x00007f133939f7c8) at job.rs:31:9
    frame #13: 0x000055e1fd837596 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] cargo::core::compiler::job::Work::then::_$u7b$$u7b$closure$u7d$$u7d$::hee839d6bf751337f(state=0x00007f133939f7c8) at job.rs:36:13
    frame #14: 0x000055e1fd83758f cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f((null)=0x00007f1339f75e90, (null)=(&cargo::core::compiler::job_queue::JobState) @ r15) at function.rs:227:5
    frame #15: 0x000055e1fd83759c cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h909d17ba2dd3a07c(self=Box<(dyn core::ops::function::FnOnce<(&cargo::core::compiler::job_queue::JobState), Output=core::result::Result<(), anyhow::Error>> + core::marker::Send), alloc::alloc::Global> @ 0x00007f5558e53650, args=(&cargo::core::compiler::job_queue::JobState) @ r15) at boxed.rs:1636:9
    frame #16: 0x000055e1fd837596 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] cargo::core::compiler::job::Work::call::he5314677cd002415(self=(inner = alloc::boxed::Box<(dyn core::ops::function::FnOnce<>, alloc::alloc::Global> @ 0x00007f5558e536f0), tx=0x00007f133939f7c8) at job.rs:31:9
    frame #17: 0x000055e1fd837596 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] cargo::core::compiler::job::Work::then::_$u7b$$u7b$closure$u7d$$u7d$::hee839d6bf751337f(state=0x00007f133939f7c8) at job.rs:36:13
    frame #18: 0x000055e1fd83758f cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f((null)=0x00007f1339f75f20, (null)=(&cargo::core::compiler::job_queue::JobState) @ r15) at function.rs:227:5
    frame #19: 0x000055e1fdabb099 cargo`cargo::core::compiler::job_queue::DrainState::run::_$u7b$$u7b$closure$u7d$$u7d$::he62fa79974c9987b [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h909d17ba2dd3a07c(self=Box<(dyn core::ops::function::FnOnce<(&cargo::core::compiler::job_queue::JobState), Output=core::result::Result<(), anyhow::Error>> + core::marker::Send), alloc::alloc::Global> @ 0x00007f5558e53650, args=(&cargo::core::compiler::job_queue::JobState) @ r15) at boxed.rs:1636:9
    frame #20: 0x000055e1fdabb093 cargo`cargo::core::compiler::job_queue::DrainState::run::_$u7b$$u7b$closure$u7d$$u7d$::he62fa79974c9987b [inlined] cargo::core::compiler::job::Work::call::he5314677cd002415(self=(inner = alloc::boxed::Box<(dyn core::ops::function::FnOnce<>, alloc::alloc::Global> @ 0x00007f5558e536f0), tx=0x00007f133939f7c8) at job.rs:31:9
    frame #21: 0x000055e1fdabb093 cargo`cargo::core::compiler::job_queue::DrainState::run::_$u7b$$u7b$closure$u7d$$u7d$::he62fa79974c9987b [inlined] cargo::core::compiler::job::Job::run::heb80c1616b1f170f(state=0x00007f133939f7c8) at job.rs:62:9
    frame #22: 0x000055e1fdabb093 cargo`cargo::core::compiler::job_queue::DrainState::run::_$u7b$$u7b$closure$u7d$$u7d$::he62fa79974c9987b(state=JobState @ 0x00007f133939f7c8) at job_queue.rs:1040:34
    frame #23: 0x000055e1fd83904f cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hda7669d323da0eaf [inlined] cargo::core::compiler::job_queue::DrainState::run::_$u7b$$u7b$closure$u7d$$u7d$::hfe68ac7b45ac03ab at job_queue.rs:1098:21
    frame #24: 0x000055e1fd838ffd cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hda7669d323da0eaf at thread.rs:437:31
    frame #25: 0x000055e1fd838fd0 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hda7669d323da0eaf((null)=0x00007f1339e3aa40, (null)=<unavailable>) at function.rs:227:5
    frame #26: 0x000055e1fd81863e cargo`std::sys_common::backtrace::__rust_begin_short_backtrace::h24259d003811a78b [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::ha48699e8e8187cbd(self=Box<(dyn core::ops::function::FnOnce<(), Output=()> + core::marker::Send), alloc::alloc::Global> @ 0x00007f5558e536f0) at boxed.rs:1636:9
    frame #27: 0x000055e1fd81863b cargo`std::sys_common::backtrace::__rust_begin_short_backtrace::h24259d003811a78b [inlined] crossbeam_utils::thread::ScopedThreadBuilder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::hbf07fd2ee35affe3 at thread.rs:449:44
    frame #28: 0x000055e1fd81863b cargo`std::sys_common::backtrace::__rust_begin_short_backtrace::h24259d003811a78b(f=(closure = alloc::boxed::Box<(dyn core::ops::function::FnOnce<>, alloc::alloc::Global> @ 0x00007f5558e536f0)) at backtrace.rs:125:18
    frame #29: 0x000055e1fd8393ab cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 [inlined] std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h77f48923b3b16330 at mod.rs:481:17
    frame #30: 0x000055e1fd8393a6 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 [inlined] _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::hcf85bf374106e3c6(self=<unavailable>) at unwind_safe.rs:271:9
    frame #31: 0x000055e1fd8393a6 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 at panicking.rs:403:40
    frame #32: 0x000055e1fd8393a6 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 at panicking.rs:367:19
    frame #33: 0x000055e1fd8393a6 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 [inlined] std::panic::catch_unwind::hfac1c3b1661d60a6(f=<unavailable>) at panic.rs:129:14
    frame #34: 0x000055e1fd8393a6 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 [inlined] std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::hdfc489316504bc38 at mod.rs:480:30
    frame #35: 0x000055e1fd839350 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3((null)=0x00007f13395a8260, (null)=<unavailable>) at function.rs:227:5
    frame #36: 0x000055e1fdfe7cb3 cargo`std::sys::unix::thread::Thread::new::thread_start::he0eb13837439c3a8 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h8ffa971d6f77ff65(self=Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> @ 0x00007f5558e536f0) at boxed.rs:1636:9
    frame #37: 0x000055e1fdfe7cad cargo`std::sys::unix::thread::Thread::new::thread_start::he0eb13837439c3a8 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::hf48067d9dbade9e9(self=0x00007f1339e02cd0) at boxed.rs:1636:9
    frame #38: 0x000055e1fdfe7ca6 cargo`std::sys::unix::thread::Thread::new::thread_start::he0eb13837439c3a8(main=0x00007f1339e02cd0) at thread.rs:106:17
    frame #39: 0x00007f133a7f955c ld-musl-x86_64.so.1`start(p=0x00007f133939f900) at pthread_create.c:203:17
    frame #40: 0x00007f133a7fbfcb ld-musl-x86_64.so.1`__clone + 47
  thread #7, name = 'cargo', stop reason = signal SIGSTOP
    frame #0: 0x00007f133a7fc005 ld-musl-x86_64.so.1`__cp_end
    frame #1: 0x00007f133a7f7d4f ld-musl-x86_64.so.1`__syscall_cp_c(nr=0, u=<unavailable>, v=<unavailable>, w=<unavailable>, x=<unavailable>, y=<unavailable>, z=<unavailable>) at pthread_cancel.c:33:6
    frame #2: 0x00007f133a800dff ld-musl-x86_64.so.1`read(fd=<unavailable>, buf=<unavailable>, count=<unavailable>) at read.c:6:9
    frame #3: 0x000055e1fdfe883c cargo`std::sys::unix::process::process_inner::_$LT$impl$u20$std..sys..unix..process..process_common..Command$GT$::spawn::h1d6f77b3e0d6cc67 [inlined] std::sys::unix::fd::FileDesc::read::ha60ab583a50f0955(buf=(data_ptr = "", length = 8)) at fd.rs:65:13
    frame #4: 0x000055e1fdfe882a cargo`std::sys::unix::process::process_inner::_$LT$impl$u20$std..sys..unix..process..process_common..Command$GT$::spawn::h1d6f77b3e0d6cc67 [inlined] std::sys::unix::pipe::AnonPipe::read::he8533efcd2d11395(buf=(data_ptr = "", length = 8)) at pipe.rs:49:9
    frame #5: 0x000055e1fdfe882a cargo`std::sys::unix::process::process_inner::_$LT$impl$u20$std..sys..unix..process..process_common..Command$GT$::spawn::h1d6f77b3e0d6cc67(self=0x00007f1338f90180, default=Stdio @ 0x00007f1338f8fc60, needs_stdin=<unavailable>) at process_unix.rs:107:19
    frame #6: 0x000055e1fdfd988c cargo`std::process::Command::spawn::ha9bec88ec3740324(self=<unavailable>) at process.rs:879:9
    frame #7: 0x000055e1fdf9218e cargo`cargo_util::process_builder::ProcessBuilder::exec_with_streaming::h35d634369e7c6306 [inlined] cargo_util::process_builder::ProcessBuilder::exec_with_streaming::_$u7b$$u7b$closure$u7d$$u7d$::hee5747160a64ba2c at process_builder.rs:249:29
    frame #8: 0x000055e1fdf92188 cargo`cargo_util::process_builder::ProcessBuilder::exec_with_streaming::h35d634369e7c6306(self=<unavailable>, on_stdout_line=&mut dyn core::ops::function::FnMut<(&str), Output=core::result::Result<(), anyhow::Error>> @ 0x00007f1338f90140, on_stderr_line=&mut dyn core::ops::function::FnMut<(&str), Output=core::result::Result<(), anyhow::Error>> @ 0x00007f1338f90150, capture_output=<unavailable>) at process_builder.rs:248:22
    frame #9: 0x000055e1fdad7fed cargo`_$LT$cargo..core..compiler..DefaultExecutor$u20$as$u20$cargo..core..compiler..Executor$GT$::exec::hc2b9e38618027b47(self=<unavailable>, cmd=<unavailable>, _id=<unavailable>, _target=<unavailable>, _mode=<unavailable>, on_stdout_line=&mut dyn core::ops::function::FnMut<(&str), Output=core::result::Result<(), anyhow::Error>> @ 0x00007f5558e536f0, on_stderr_line=&mut dyn core::ops::function::FnMut<(&str), Output=core::result::Result<(), anyhow::Error>> @ 0x00007f5558e53650) at mod.rs:133:9
    frame #10: 0x000055e1fdadd7c2 cargo`cargo::core::compiler::rustc::_$u7b$$u7b$closure$u7d$$u7d$::hd08ce3b824e17e69(state=<unavailable>) at mod.rs:318:13
    frame #11: 0x000055e1fd83759c cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h909d17ba2dd3a07c(self=Box<(dyn core::ops::function::FnOnce<(&cargo::core::compiler::job_queue::JobState), Output=core::result::Result<(), anyhow::Error>> + core::marker::Send), alloc::alloc::Global> @ 0x00007f5558e536f0, args=(&cargo::core::compiler::job_queue::JobState) @ r15) at boxed.rs:1636:9
    frame #12: 0x000055e1fd837596 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] cargo::core::compiler::job::Work::call::he5314677cd002415(self=(inner = alloc::boxed::Box<(dyn core::ops::function::FnOnce<>, alloc::alloc::Global> @ 0x00007f5558e536f0), tx=0x00007f1338f907c8) at job.rs:31:9
    frame #13: 0x000055e1fd837596 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] cargo::core::compiler::job::Work::then::_$u7b$$u7b$closure$u7d$$u7d$::hee839d6bf751337f(state=0x00007f1338f907c8) at job.rs:36:13
    frame #14: 0x000055e1fd83758f cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f((null)=0x00007f1339e20be0, (null)=(&cargo::core::compiler::job_queue::JobState) @ r15) at function.rs:227:5
    frame #15: 0x000055e1fd83759c cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h909d17ba2dd3a07c(self=Box<(dyn core::ops::function::FnOnce<(&cargo::core::compiler::job_queue::JobState), Output=core::result::Result<(), anyhow::Error>> + core::marker::Send), alloc::alloc::Global> @ 0x00007f5558e53650, args=(&cargo::core::compiler::job_queue::JobState) @ r15) at boxed.rs:1636:9
    frame #16: 0x000055e1fd837596 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] cargo::core::compiler::job::Work::call::he5314677cd002415(self=(inner = alloc::boxed::Box<(dyn core::ops::function::FnOnce<>, alloc::alloc::Global> @ 0x00007f5558e53650), tx=0x00007f1338f907c8) at job.rs:31:9
    frame #17: 0x000055e1fd837596 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f [inlined] cargo::core::compiler::job::Work::then::_$u7b$$u7b$closure$u7d$$u7d$::hee839d6bf751337f(state=0x00007f1338f907c8) at job.rs:36:13
    frame #18: 0x000055e1fd83758f cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc40448ed66974f1f((null)=0x00007f1339e20c70, (null)=(&cargo::core::compiler::job_queue::JobState) @ r15) at function.rs:227:5
    frame #19: 0x000055e1fdabb099 cargo`cargo::core::compiler::job_queue::DrainState::run::_$u7b$$u7b$closure$u7d$$u7d$::he62fa79974c9987b [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h909d17ba2dd3a07c(self=Box<(dyn core::ops::function::FnOnce<(&cargo::core::compiler::job_queue::JobState), Output=core::result::Result<(), anyhow::Error>> + core::marker::Send), alloc::alloc::Global> @ 0x00007f555ce452c0, args=(&cargo::core::compiler::job_queue::JobState) @ r15) at boxed.rs:1636:9
    frame #20: 0x000055e1fdabb093 cargo`cargo::core::compiler::job_queue::DrainState::run::_$u7b$$u7b$closure$u7d$$u7d$::he62fa79974c9987b [inlined] cargo::core::compiler::job::Work::call::he5314677cd002415(self=(inner = alloc::boxed::Box<(dyn core::ops::function::FnOnce<>, alloc::alloc::Global> @ 0x00007f555ce452c0), tx=0x00007f1338f907c8) at job.rs:31:9
    frame #21: 0x000055e1fdabb093 cargo`cargo::core::compiler::job_queue::DrainState::run::_$u7b$$u7b$closure$u7d$$u7d$::he62fa79974c9987b [inlined] cargo::core::compiler::job::Job::run::heb80c1616b1f170f(state=0x00007f1338f907c8) at job.rs:62:9
    frame #22: 0x000055e1fdabb093 cargo`cargo::core::compiler::job_queue::DrainState::run::_$u7b$$u7b$closure$u7d$$u7d$::he62fa79974c9987b(state=JobState @ 0x00007f1338f907c8) at job_queue.rs:1040:34
    frame #23: 0x000055e1fd83904f cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hda7669d323da0eaf [inlined] cargo::core::compiler::job_queue::DrainState::run::_$u7b$$u7b$closure$u7d$$u7d$::hfe68ac7b45ac03ab at job_queue.rs:1098:21
    frame #24: 0x000055e1fd838ffd cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hda7669d323da0eaf at thread.rs:437:31
    frame #25: 0x000055e1fd838fd0 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hda7669d323da0eaf((null)=0x00007f1339bcab80, (null)=<unavailable>) at function.rs:227:5
    frame #26: 0x000055e1fd81863e cargo`std::sys_common::backtrace::__rust_begin_short_backtrace::h24259d003811a78b [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::ha48699e8e8187cbd(self=Box<(dyn core::ops::function::FnOnce<(), Output=()> + core::marker::Send), alloc::alloc::Global> @ 0x00007f555ce452c0) at boxed.rs:1636:9
    frame #27: 0x000055e1fd81863b cargo`std::sys_common::backtrace::__rust_begin_short_backtrace::h24259d003811a78b [inlined] crossbeam_utils::thread::ScopedThreadBuilder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::hbf07fd2ee35affe3 at thread.rs:449:44
    frame #28: 0x000055e1fd81863b cargo`std::sys_common::backtrace::__rust_begin_short_backtrace::h24259d003811a78b(f=(closure = alloc::boxed::Box<(dyn core::ops::function::FnOnce<>, alloc::alloc::Global> @ 0x00007f5558e53650)) at backtrace.rs:125:18
    frame #29: 0x000055e1fd8393ab cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 [inlined] std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h77f48923b3b16330 at mod.rs:481:17
    frame #30: 0x000055e1fd8393a6 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 [inlined] _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::hcf85bf374106e3c6(self=<unavailable>) at unwind_safe.rs:271:9
    frame #31: 0x000055e1fd8393a6 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 at panicking.rs:403:40
    frame #32: 0x000055e1fd8393a6 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 at panicking.rs:367:19
    frame #33: 0x000055e1fd8393a6 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 [inlined] std::panic::catch_unwind::hfac1c3b1661d60a6(f=<unavailable>) at panic.rs:129:14
    frame #34: 0x000055e1fd8393a6 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3 [inlined] std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::hdfc489316504bc38 at mod.rs:480:30
    frame #35: 0x000055e1fd839350 cargo`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf941baa3d65828b3((null)=0x00007f13395a8fe0, (null)=<unavailable>) at function.rs:227:5
    frame #36: 0x000055e1fdfe7cb3 cargo`std::sys::unix::thread::Thread::new::thread_start::he0eb13837439c3a8 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h8ffa971d6f77ff65(self=Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> @ 0x00007f5558e536f0) at boxed.rs:1636:9
    frame #37: 0x000055e1fdfe7cad cargo`std::sys::unix::thread::Thread::new::thread_start::he0eb13837439c3a8 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call

[Mark-Simulacrum]

cc @joshtriplett

Seems ... a little surprising? I wouldn't expect that PR to change cargo behavior.

[joshtriplett]

The most likely reason I can think of would be if the Gentoo sandbox tool doesn't understand the clone3 syscall.

[cuviper]

I can reproduce this in a gentoo container. On a RHEL 8 host (kernel 4.18, without clone3), it's fine. On Fedora 34 (kernel 5.14, with clone3) it's fine with -j1, sometimes hangs with -j2, and consistently hangs with more jobs (default 16 logical threads here). The child process is stuck here:

#0  0x00007f193f3bdf8b in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f193f3b6fd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x00007f193f3f65d0 in ?? () from /usr/lib64/libsandbox.so
#3  0x00007f193f3fa2f2 in access () from /usr/lib64/libsandbox.so
#4  0x00007f193f3fb168 in execvp () from /usr/lib64/libsandbox.so
#5  0x0000560a24303bdf in std::sys::unix::process::process_common::Command::do_exec ()
    at library/std/src/sys/unix/process/process_unix.rs:367
#6  0x0000560a24303577 in std::sys::unix::process::process_common::Command::spawn ()
    at library/std/src/sys/unix/process/process_unix.rs:77
#7  0x0000560a242f5bdc in std::process::Command::spawn () at library/std/src/process.rs:879
#8  0x0000560a2413aafe in cargo_util::process_builder::ProcessBuilder::exec_with_streaming ()

[cuviper]

I don't know anything about libsandbox.so, but I am suspicious of that fact that we're bypassing the system fork without the same clone flags and other internal bookkeeping. The glibc implementation usually clones this way:

/* Call the clone syscall with fork semantic.  The CTID address is used
   to store the child thread ID at its locationm, to erase it in child memory
   when the child exits, and do a wakeup on the futex at that address.

   The architecture with non-default kernel abi semantic should correctlly
   override it with one of the supported calling convention (check generic
   kernel-features.h for the clone abi variants).  */
static inline pid_t
arch_fork (void *ctid)
{
  const int flags = CLONE_CHILD_SETTID | CLONE_CHILD_CLEARTID | SIGCHLD;

... where ctid comes from _Fork, arch_fork (&THREAD_SELF->tid), and afterward that also initializes the child's robust mutext list. There's further fork handling that's supposed to happen after that in __libc_fork too.

In musl they have similar cleanups in the child, manually calling gettid and clearing lock state.

I suppose it's generally not safe to call pthread_mutex_lock from a forked child, but libsandbox adds some low-level hooks to make it work -- I see it has its own fork and exec* functions. But we've half-way bypassed it here, with a manual clone3 syscall and then a library-level execvp.

I think it's also possible in general that some fork-safe libc functions will be unsafe if you've bypassed the library fork.

[cuviper]

Perhaps we should initialize HAS_CLONE3 to false while we work this out -- it's only useful for the unstable pidfd stuff anyway. Or we could only use clone3 when directly needed for pidfd, which will be ~never for now.

[Mark-Simulacrum]

I agree that it seems like a good idea to either revert the patch wholesale on 1.56 (current beta) or apply some minimal fix that avoids this problem while we discuss -- @cuviper would you be up for posting a direct-to-beta branch patch making that change?

Once that lands we can bump this to track 1.57 rather than the 1.56 milestone to make sure we still keep it tracked, but we probably want to apply the partial "revert" on master as well to avoid a continuous cherry pick dance.

[joshtriplett]

I just read through the source of the Gentoo sandbox tool, and I don't think this is an issue with bypassing libc at all; this seems like a bug in the libsandbox library. And as far as I can tell, it isn't specific to clone3; this would happen with any program calling clone instead of fork to spawn a new process, as well. I think it might also happen when calling posix_spawn on glibc, since glibc will use clone rather than fork for that.

libsandbox hooks fork, and uses that hook as what it describes as "a poor man's pthread_atfork()"; it acquires its lock, calls the function, then releases its lock, to ensure that the fork doesn't happen while another thread holds the lock. However, libsandbox does not hook either clone or clone3. I would guess that libsandbox assumes that clone is only used for threads. I think the correct fix here would be for libsandbox to hook both clone and clone3, and perform the same logic whenever the flags do not include CLONE_VM.

I would propose that we temporarily disable HAS_CLONE3 to give more time to work around this issue, but I don't think we should permanently work around this issue. I think it needs fixing in libsandbox.

[joshtriplett]

There's a similar issue at https://bugs.gentoo.org/807832 involving java, and it appears to also involve clone3. Though, oddly, the strace there shows CLONE_VM, so that may be a different issue.

[cuviper]

See #89924 for restricting clone3 to pidfd needs.

I think the correct fix here would be for libsandbox to hook both clone and clone3, and perform the same logic whenever the flags do not include CLONE_VM.

I suppose they could hook clone, but they can't hook a syscall clone3 the same way. I haven't looked if they're doing syscall filtering at all, but if so, is there anything they could do besides outright disallowing it?

But I'm still concerned about bypassing libc in general. Suppose instead of "a poor man's pthread_atfork()", they could have used the real thing, or any other library might be trying to do the right thing here. Even libc internals will have state it's trying to maintain, like the TID and robust mutex list, so I fear we can't really call libc at all after this.