spurious crash when compiling cargo on dist-mips64{el}-linux

[Noratrieb]

#110721 (comment)
#112695 (comment)
#112827 (comment)
#112616 (comment)
#111850 (comment)
#112591 (comment)
#112591 (comment)

Most recent ones were on mips64 and those don't seem to date back that far, but mips64el seems to have been a problem for several months.

see https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/MIPS.20failure.20on.20CI

[Noratrieb]

Somehow #112591 is blocked by this and cannot get past it.

[lqd]

• Rollup of 5 pull requests #113014 (comment)
• Refactor metadata emission to avoid visiting HIR #98867 (comment)
• Refactor metadata emission to avoid visiting HIR #98867 (comment)
• Normalize opaques with late-bound vars again #113108 (comment)
• Normalize opaques with late-bound vars again #113108 (comment)
• Normalize opaques with late-bound vars again #113108 (comment)

[Noratrieb]

I managed to get a backtrace by running the builder locally and then adding a bash call after the build invocation, then using gdb inside the container to rerun the command output by adding -vvv to the dist command.

gdb backtrace

#0  0x00007fffee326574 in llvm::MVT::getSizeInBits() const () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#1  0x00007fffef074652 in llvm::SDValue::getScalarValueSizeInBits() const [clone .isra.0] () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#2  0x00007fffef0ab57e in llvm::TargetLowering::SimplifyDemandedVectorElts(llvm::SDValue, llvm::APInt const&, llvm::APInt&, llvm::APInt&, llvm::TargetLowering::TargetLoweringOpt&, unsigned int, bool) const [clone .localalias] () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#3  0x00007fffef0a95b7 in llvm::TargetLowering::SimplifyDemandedVectorElts(llvm::SDValue, llvm::APInt const&, llvm::APInt&, llvm::APInt&, llvm::TargetLowering::TargetLoweringOpt&, unsigned int, bool) const [clone .localalias] () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#4  0x00007fffeeed92eb in (anonymous namespace)::DAGCombiner::SimplifyDemandedVectorElts(llvm::SDValue, llvm::APInt const&, bool) ()
   from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#5  0x00007fffeef0250a in (anonymous namespace)::DAGCombiner::visitEXTRACT_VECTOR_ELT(llvm::SDNode*) () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#6  0x00007fffeef650b5 in (anonymous namespace)::DAGCombiner::combine(llvm::SDNode*) () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#7  0x00007fffeef6654f in llvm::SelectionDAG::Combine(llvm::CombineLevel, llvm::AAResults*, llvm::CodeGenOpt::Level) ()
   from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#8  0x00007fffef05da86 in llvm::SelectionDAGISel::CodeGenAndEmitDAG() [clone .localalias] () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#9  0x00007fffef0605ca in llvm::SelectionDAGISel::SelectAllBasicBlocks(llvm::Function const&) [clone .localalias] ()
   from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#10 0x00007fffef062a06 in llvm::SelectionDAGISel::runOnMachineFunction(llvm::MachineFunction&) [clone .part.0] () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#11 0x00007ffff0b84bcf in llvm::MipsDAGToDAGISel::runOnMachineFunction(llvm::MachineFunction&) () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#12 0x00007fffef2a52a4 in llvm::MachineFunctionPass::runOnFunction(llvm::Function&) [clone .part.0] () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#13 0x00007ffff0288a70 in llvm::FPPassManager::runOnFunction(llvm::Function&) [clone .localalias] () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#14 0x00007ffff0288be4 in llvm::FPPassManager::runOnModule(llvm::Module&) [clone .localalias] () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#15 0x00007ffff02895e4 in llvm::legacy::PassManagerImpl::run(llvm::Module&) [clone .localalias] () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#16 0x00007fffee2dd598 in LLVMRustWriteOutputFile () from /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-803b3a685fc1945d.so
#17 0x00007fffee2a79ef in rustc_codegen_llvm::back::write::write_output_file () at compiler/rustc_codegen_llvm/src/back/write.rs:71
#18 0x00007fffee2abf88 in rustc_codegen_llvm::back::write::codegen::{closure#2} () at compiler/rustc_codegen_llvm/src/back/write.rs:749
#19 rustc_codegen_llvm::back::write::codegen::with_codegen<rustc_codegen_llvm::back::write::codegen::{closure_env#2}, core::result::Result<(), rustc_span::fatal_error::FatalError>> ()
    at compiler/rustc_codegen_llvm/src/back/write.rs:611
#20 rustc_codegen_llvm::back::write::codegen () at compiler/rustc_codegen_llvm/src/back/write.rs:748
#21 0x00007fffee1bf4d6 in rustc_codegen_llvm::{impl#3}::codegen () at compiler/rustc_codegen_llvm/src/lib.rs:233
#22 rustc_codegen_ssa::back::write::finish_intra_module_work<rustc_codegen_llvm::LlvmCodegenBackend> () at compiler/rustc_codegen_ssa/src/back/write.rs:913
#23 0x00007fffee27039f in rustc_codegen_ssa::back::write::execute_lto_work_item<rustc_codegen_llvm::LlvmCodegenBackend> () at compiler/rustc_codegen_ssa/src/back/write.rs:899
#24 rustc_codegen_ssa::back::write::spawn_work::{closure#0}<rustc_codegen_llvm::LlvmCodegenBackend> () at compiler/rustc_codegen_ssa/src/back/write.rs:1689
#25 rustc_codegen_llvm::{impl#2}::spawn_named_thread::{closure#0}<rustc_codegen_ssa::back::write::spawn_work::{closure_env#0}<rustc_codegen_llvm::LlvmCodegenBackend>, ()> ()
    at compiler/rustc_codegen_llvm/src/lib.rs:168
#26 std::sys_common::backtrace::__rust_begin_short_backtrace<rustc_codegen_llvm::{impl#2}::spawn_named_thread::{closure_env#0}<rustc_codegen_ssa::back::write::spawn_work::{closure_env#0}<rustc_codegen_llvm::LlvmCodegenBackend>, ()>, ()> () at library/std/src/sys_common/backtrace.rs:135
#27 0x00007fffee240a46 in std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure#0}<rustc_codegen_llvm::{impl#2}::spawn_named_thread::{closure_env#0}<rustc_codegen_ssa::back::write::spawn_work::{closure_env#0}<rustc_codegen_llvm::LlvmCodegenBackend>, ()>, ()> () at library/std/src/thread/mod.rs:529
#28 core::panic::unwind_safe::{impl#23}::call_once<(), std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<rustc_codegen_llvm::{impl#2}::spawn_named_thread::{closure_env#0}<rustc_codegen_ssa::back::write::spawn_work::{closure_env#0}<rustc_codegen_llvm::LlvmCodegenBackend>, ()>, ()>> () at library/core/src/panic/unwind_safe.rs:271
#29 std::panicking::try::do_call<core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<rustc_codegen_llvm::{impl#2}::spawn_named_thread::{closure_env#0}<rustc_codegen_ssa::back::write::spawn_work::{closure_env#0}<rustc_codegen_llvm::LlvmCodegenBackend>, ()>, ()>>, ()> () at library/std/src/panicking.rs:500
#30 std::panicking::try<(), core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<rustc_codegen_llvm::{impl#2}::spawn_named_thread::{closure_env#0}<rustc_codegen_ssa::back::write::spawn_work::{closure_env#0}<rustc_codegen_llvm::LlvmCodegenBackend>, ()>, ()>>> () at library/std/src/panicking.rs:464
#31 std::panic::catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<rustc_codegen_llvm::{impl#2}::spawn_named_thread::{closure_env#0}<rustc_codegen_ssa::back::write::spawn_work::{closure_env#0}<rustc_codegen_llvm::LlvmCodegenBackend>, ()>, ()>>, ()> () at library/std/src/panic.rs:142
#32 std::thread::{impl#0}::spawn_unchecked_::{closure#1}<rustc_codegen_llvm::{impl#2}::spawn_named_thread::{closure_env#0}<rustc_codegen_ssa::back::write::spawn_work::{closure_env#0}<rustc_codegen_llvm::LlvmCodegenBackend>, ()>, ()> () at library/std/src/thread/mod.rs:528
#33 core::ops::function::FnOnce::call_once<std::thread::{impl#0}::spawn_unchecked_::{closure_env#1}<rustc_codegen_llvm::{impl#2}::spawn_named_thread::{closure_env#0}<rustc_codegen_ssa::back::write::spawn_work::{closure_env#0}<rustc_codegen_llvm::LlvmCodegenBackend>, ()>, ()>, ()> () at library/core/src/ops/function.rs:250
#34 0x00007fffecf13bbe in alloc::boxed::{impl#47}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:1993
#35 alloc::boxed::{impl#47}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:1993
#36 std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#37 0x00007fffeccb9b43 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#38 0x00007fffecd4ba00 in ?? () from /lib/x86_64-linux-gnu/libc.so.6

I got this on bb7de68e2d432b4a695f23f1f76eeb2ad3f5c647, which reliably reproduces the crash for me.

[lqd]

cc @rust-lang/wg-llvm is the backtrace enough to understand the cause of the issue ?

[Noratrieb]

I have a single .ll file that reproduces the segfault when compiled with just llc, even outside the container with an llc i got off nixpkgs. I can't upload the file here right now since it's 416MB but I'll work on reducing it.
edit: https://cdn.discordapp.com/attachments/957720175619215380/1124044197008253080/bad-cgu.bc

[Noratrieb]

; ModuleID = 'reduced.bc'
target datalayout = "E-m:e-i8:8:32-i16:16:32-i64:64-n32:64-S128"
target triple = "mips64-unknown-linux-gnuabi64"

define internal fastcc void @uwu(<5 x i32> %self.8.val) {
start:
  store <5 x i32> %self.8.val, ptr null, align 4
  ret void
}

I'll open an upstream issue.