ui test failures (segfaults and nested panics) since libbacktrace update

[nikic]

Since the libbacktrace update in #50955 I'm getting 150 failures in ui tests for my local rustc build. The configuration is default apart from enabling LLVM assertions. The system is Ubuntu 16.04 on x86_64. The issue occurs with 74d0939, but not with 4f99f37 (commit prior to #50955 merge).

The failures are either due to segfaults or panics while panicking.

Sample segfault:

#0  __GI___libc_free (mem=0x1f8) at malloc.c:2951
#1  0x00007ffff7a08a1a in core::ptr::drop_in_place::h1ca2358faa81af6e ()
   from /home/nikic/rust/build/x86_64-unknown-linux-gnu/stage1/bin/../lib/librustc_driver-34cfa81bde6fac8c.so
#2  0x00007ffff7a1ad6f in rustc_driver::driver::compile_input::he84fc71bb26c3ae0 ()
   from /home/nikic/rust/build/x86_64-unknown-linux-gnu/stage1/bin/../lib/librustc_driver-34cfa81bde6fac8c.so
#3  0x00007fffffff8600 in ?? ()
#4  0x00007fffffff8b30 in ?? ()
#5  0x0000000000000001 in ?? ()
#6  0x00007fffffff8501 in ?? ()
#7  0x00007ffff7ab5dd5 in rustc_driver::run_compiler_with_pool::hf7847454e1489610 ()
   from /home/nikic/rust/build/x86_64-unknown-linux-gnu/stage1/bin/../lib/librustc_driver-34cfa81bde6fac8c.so
#8  0x00007ffff79d8643 in syntax::with_globals::h3467ea0b3a675539 ()
   from /home/nikic/rust/build/x86_64-unknown-linux-gnu/stage1/bin/../lib/librustc_driver-34cfa81bde6fac8c.so
#9  0x00007ffff79b0a5e in _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::hbeda43bf54c2692e
    ()
   from /home/nikic/rust/build/x86_64-unknown-linux-gnu/stage1/bin/../lib/librustc_driver-34cfa81bde6fac8c.so
#10 0x00007ffff76da08a in __rust_maybe_catch_panic ()
   from /home/nikic/rust/build/x86_64-unknown-linux-gnu/stage1/bin/../lib/libstd-fc6b9a3d7065b2e2.so
#11 0x00007ffff7ab3064 in rustc_driver::run::h69cfea9e720cab41 ()
   from /home/nikic/rust/build/x86_64-unknown-linux-gnu/stage1/bin/../lib/librustc_driver-34cfa81bde6fac8c.so
#12 0x00007ffff7ac08fb in rustc_driver::main::h8abbba663fdf77b9 ()
   from /home/nikic/rust/build/x86_64-unknown-linux-gnu/stage1/bin/../lib/librustc_driver-34cfa81bde6fac8c.so
#13 0x0000555555554b23 in std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h6c799bafc2cc7997 ()
#14 0x00007ffff76a46b3 in std::panicking::try::do_call::h5d13bd17c519af69 ()
   from /home/nikic/rust/build/x86_64-unknown-linux-gnu/stage1/bin/../lib/libstd-fc6b9a3d7065b2e2.so
#15 0x00007ffff76da08a in __rust_maybe_catch_panic ()
   from /home/nikic/rust/build/x86_64-unknown-linux-gnu/stage1/bin/../lib/libstd-fc6b9a3d7065b2e2.so
#16 0x00007ffff7690486 in std::rt::lang_start_internal::he15214c0a1fdba94 ()
   from /home/nikic/rust/build/x86_64-unknown-linux-gnu/stage1/bin/../lib/libstd-fc6b9a3d7065b2e2.so
#17 0x0000555555554b84 in main ()

Sample panic while panicking:

stack backtrace:
   0:     0x7f8e3cd9bc2e - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::ha82b397ea71be585
   1:     0x7f8e3cd71796 - std::sys_common::backtrace::print::h0f02aefa57d1295d
   2:     0x7f8e3cd782bd - std::panicking::default_hook::{{closure}}::hb248860985868cdd
   3:     0x7f8e3cd78030 - std::panicking::default_hook::h0dd63170d2171fd8
   4:     0x7f8e393070d5 - rustc::util::common::panic_hook::h5a856d10e946cf82
   5:     0x7f8e3cd78943 - std::panicking::rust_panic_with_hook::h470a7ba29e7f6e22
   6:     0x7f8e3cd7853c - std::panicking::begin_panic_fmt::hdda42c41c88b4167
   7:     0x7f8e3cd78501 - rust_begin_unwind
   8:     0x7f8e3cdcf722 - core::panicking::panic_fmt::hd47ce5be3b71cd97
   9:     0x7f8e3d15c4a5 - core::result::unwrap_failed::h38f2e7889fea9e60
  10:     0x7f8e3d182df4 - <arena::TypedArena<T> as core::ops::drop::Drop>::drop::h694e59aa0a4a1d47
  11:     0x7f8e3d0da9cc - core::ptr::drop_in_place::h0d7fde6281af2c84
  12:     0x7f8e3d0eeaf9 - rustc_driver::driver::compile_input::he84fc71bb26c3ae0
  13:                0x3 - <unknown>
thread panicked while panicking. aborting.

What stands out in both cases is that the trace contains a low-value address.

[kennytm]

Just to confirm, is the libbacktrace submodule up to date? (git submodule update --init --recursive)

Is it still reproducible after ./x.py clean?

[nikic]

@kennytm Yeah, the submodule is up to date and this still reproduced after doing a ./x.py clean --all.

[kennytm]

Could you verify if rolling back to an earlier version of libbacktrace would fix the problem?

1. Make sure config.toml has [build] submodules = false to stop x.py from automatically managing submodules
2. cd src/libbacktrace
3. git checkout 14d377e9be7c89511c472d728e1b88b4e96f1946
4. cd ../..
5. Build again

[Mark-Simulacrum]

None of the UI tests fail for me on Ubuntu 18.04 with latest master. Can you also check that running git reset --hard inside src/libbacktrace doesn't fix the problem for you?

[nikic]

@kennytm Nope, did not help. Still seeing the same failures.

@Mark-Simulacrum I just did an rm -rf rust and checked out and rebuilt everything from scratch -- still same result :/

[Mark-Simulacrum]

Can you reproduce running ./src/ci/docker/run.sh x86_64-gnu? That should isolate the environment as much as possible...

[nikic]

@Mark-Simulacrum Docker ran through cleanly (only some network related test failures).

[Mark-Simulacrum]

Unfortunate. I'm not sure what could be the cause of this then...

[nikic]

I've switched to a fresh Ubuntu 18.04 VM now, where the problem doesn't occur. As it seems like I'm the only one seeing this, I guess we can just chalk this up to some kind of one-time environment corruption.

[vorner]

I'm seeing this on my gentoo system (162 failures in ui tests, as described above). Yesterday master (01cc982), fresh clone, no modifications to config.toml except for low-priotity = true.

I'd like to provide more info if I knew which one.

[nikic]

Btw, as another data point, before I gave up on this I also tried just reverting the build-system changes but keeping the updated libbacktrace. Result was the same.

[Mark-Simulacrum]

@vorner What version of Gentoo?