llvm segfaults during bootstrap of rustc_middle in stage 1

[matthiaskrgr]

When bootstrapping rustc, I can observe a crash inside llvm when building stage 1 rustc_middle.

  error: could not compile `rustc_middle`.

Caused by:
  process didn't exit successfully: `/home/matthias/.cargo/bin/sccache /home/matthias/vcs/github/rust/build/bootstrap/debug/rustc --crate-name rustc_middle --edition=2018 compiler/rustc_middle/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C codegen-units=4 -C debuginfo=0 -C metadata=d04a1078c718a5ca -C extra-filename=-d04a1078c718a5ca --out-dir /home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps -L dependency=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/release/deps --extern bitflags=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libbitflags-70d28355c7ba065c.rmeta --extern byteorder=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libbyteorder-e3dd620d9578e39d.rmeta --extern chalk_ir=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libchalk_ir-c1f762b67e0905bf.rmeta --extern measureme=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libmeasureme-36a1068e4a5ba5a7.rmeta --extern polonius_engine=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libpolonius_engine-9c9005f4c2ee0bc8.rmeta --extern rustc_rayon_core=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_rayon_core-837c6250b0008557.rmeta --extern rustc_apfloat=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_apfloat-f41ce9e4d2735cf3.rmeta --extern rustc_arena=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_arena-2e633fc954377d89.rmeta --extern rustc_ast=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_ast-178de7fd822a2b16.rmeta --extern rustc_attr=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_attr-a692c7c6d2859faa.rmeta --extern rustc_data_structures=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_data_structures-277b9d6f8e7c56be.rmeta --extern rustc_errors=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_errors-04226f690745f132.rmeta --extern rustc_feature=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_feature-540909ef5385f3a4.rmeta --extern rustc_hir=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_hir-4fe87b7e14efe82a.rmeta --extern rustc_index=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_index-725f016597ca9da3.rmeta --extern rustc_macros=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/release/deps/librustc_macros-66fc5a3cd2593a47.so --extern rustc_query_system=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_query_system-c0820170a4d74415.rmeta --extern rustc_serialize=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_serialize-983e836ae27a89c6.rmeta --extern rustc_session=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_session-460c56b3f5719dbd.rmeta --extern rustc_span=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_span-1dbb07394d590c27.rmeta --extern rustc_target=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_target-b01ab5104179568a.rmeta --extern smallvec=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libsmallvec-5b86fc59326ca1e7.rmeta --extern tracing=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libtracing-6132f0f4621eff27.rmeta -C target-cpu=native -Zmacro-backtrace '-Clink-args=-Wl,-rpath,$ORIGIN/../lib' -Zunstable-options '-Wrustc::internal' -Cprefer-dynamic -Zbinary-dep-depinfo -L native=/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/build/psm-fefe209cb5a48701/out` (exit code: 254)
warning: build failed, waiting for other jobs to finish...

In dmsg I could see this:

[336285.100146] rustc[2173805]: segfault at a5 ip 00007f845ed6e310 sp 00007f844bffc200 error 4 in libLLVM-11-rust-dev.so[7f845eb90000+20ce000]
[336285.100157] Code: 24 20 85 c9 74 04 48 89 0c 08 4c 8b b4 24 98 00 00 00 4c 8b 44 24 18 45 8d 48 01 48 89 6c 24 48 48 8b 4c 24 28 48 89 4c 24 50 <49> 8b 42 10 48 8b 10 48 85 c9 74 2f 48 6b f1 38 31 ff 31 c0 66 2e
[336285.100212] audit: type=1701 audit(1598967540.990:822): auid=1000 uid=1000 gid=1000 ses=2 pid=2173044 comm="rustc" exe="/home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1/bin/rustc" sig=11 res=1

I did a bisect and found out that the crash happens since 1d22f75
cc @davidtwco

my cpu is a Intel(R) Core(TM) i5-7200U CPU

The timing of the crash makes me think that this might happen somewhere during LTOing of the rustc_middle crate, rustc did not show a backtrace for the crash though, since it happens inside llvm.
I already did purge my rustc build cache and the sccache before I saw the segfault happens inside llvm (it didn't help anyway). 😅

rustc @ d9cd4a3

[davidtwco]

cc @eddyb

[Aaron1011]

@matthiaskrgr: Was a coredump generated? If so, should be able to generate a backtrace by running gdb corefile, followed by the bt command.

[matthiaskrgr]

[New LWP 2248402]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
0x00007fbaf234a9b7 in __pthread_clockjoin_ex () from /usr/lib/libpthread.so.0
(gdb) continue
Continuing.
[New Thread 0x7fbae514e640 (LWP 2248426)]
[New Thread 0x7fbae4b18640 (LWP 2248427)]
[New Thread 0x7fbae4225640 (LWP 2248432)]
[New Thread 0x7fbaddffe640 (LWP 2248438)]
[New Thread 0x7fbaddffe640 (LWP 2248474)]
[Thread 0x7fbaddffe640 (LWP 2248438) exited]
[New Thread 0x7fbaf17fd640 (LWP 2248491)]
[Thread 0x7fbae4225640 (LWP 2248432) exited]
[Thread 0x7fbaddffe640 (LWP 2248474) exited]
[Thread 0x7fbaf17fd640 (LWP 2248491) exited]
[New Thread 0x7fbaf17fd640 (LWP 2248499)]
[New Thread 0x7fbaddffe640 (LWP 2248500)]

Thread 9 "rustc" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fbaf17fd640 (LWP 2248499)]
0x00007fbaf343c310 in llvm::CallInst::Create(llvm::FunctionType*, llvm::Value*, llvm::ArrayRef<llvm::Value*>, llvm::ArrayRef<llvm::OperandBundleDefT<llvm::Value*> >, llvm::Twine const&, llvm::Instruction*) () from /home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1/lib/libLLVM-11-rust-dev.so
(gdb) bt
#0  0x00007fbaf343c310 in llvm::CallInst::Create(llvm::FunctionType*, llvm::Value*, llvm::ArrayRef<llvm::Value*>, llvm::ArrayRef<llvm::OperandBundleDefT<llvm::Value*> >, llvm::Twine const&, llvm::Instruction*) () from /home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1/lib/libLLVM-11-rust-dev.so
#1  0x00007fbaf343bf11 in llvm::IRBuilderBase::CreateCall(llvm::FunctionType*, llvm::Value*, llvm::ArrayRef<llvm::Value*>, llvm::Twine const&, llvm::MDNode*) ()
   from /home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1/lib/libLLVM-11-rust-dev.so
#2  0x00007fbaf350121d in llvm::IRBuilderBase::CreateMaskedGather(llvm::Value*, llvm::Align, llvm::Value*, llvm::Value*, llvm::Twine const&) ()
   from /home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1/lib/libLLVM-11-rust-dev.so
#3  0x00007fbaf45f9aeb in llvm::InnerLoopVectorizer::vectorizeMemoryInstruction(llvm::Instruction*, llvm::VPTransformState&, llvm::VPValue*, llvm::VPValue*, llvm::VPValue*) ()
   from /home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1/lib/libLLVM-11-rust-dev.so
#4  0x00007fbaf466bdb4 in llvm::VPBasicBlock::execute(llvm::VPTransformState*) () from /home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1/lib/libLLVM-11-rust-dev.so
#5  0x00007fbaf466d23d in llvm::VPlan::execute(llvm::VPTransformState*) () from /home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1/lib/libLLVM-11-rust-dev.so
#6  0x00007fbaf461a9ee in llvm::LoopVectorizationPlanner::executePlan(llvm::InnerLoopVectorizer&, llvm::DominatorTree*) ()
   from /home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1/lib/libLLVM-11-rust-dev.so
#7  0x00007fbaf46256e8 in llvm::LoopVectorizePass::processLoop(llvm::Loop*) () from /home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1/lib/libLLVM-11-rust-dev.so
#8  0x00007fbaf4627237 in llvm::LoopVectorizePass::runImpl(llvm::Function&, llvm::ScalarEvolution&, llvm::LoopInfo&, llvm::TargetTransformInfo&, llvm::DominatorTree&, llvm::BlockFrequencyInfo&, llvm::TargetLibraryInfo*, llvm::DemandedBits&, llvm::AAResults&, llvm::AssumptionCache&, std::function<llvm::LoopAccessInfo const& (llvm::Loop&)>&, llvm::OptimizationRemarkEmitter&, llvm::ProfileSummaryInfo*) () from /home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1/lib/libLLVM-11-rust-dev.so
#9  0x00007fbaf462acab in (anonymous namespace)::LoopVectorize::runOnFunction(llvm::Function&) () from /home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1/lib/libLLVM-11-rust-dev.so
#10 0x00007fbaf353d3b2 in llvm::FPPassManager::runOnFunction(llvm::Function&) () from /home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1/lib/libLLVM-11-rust-dev.so
#11 0x00007fbaf3544023 in llvm::FPPassManager::runOnModule(llvm::Module&) () from /home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1/lib/libLLVM-11-rust-dev.so
#12 0x00007fbaf353ddfa in llvm::legacy::PassManagerImpl::run(llvm::Module&) () from /home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1/lib/libLLVM-11-rust-dev.so
#13 0x00007fbaf34a08fa in LLVMRunPassManager () from /home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1/lib/libLLVM-11-rust-dev.so
#14 0x00007fbaf6db4054 in rustc_codegen_llvm::back::lto::run_pass_manager () from /home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-1c5da71f0b2b75c2.so
#15 0x00007fbaf6dff3ef in <rustc_codegen_llvm::LlvmCodegenBackend as rustc_codegen_ssa::traits::write::WriteBackendMethods>::optimize_thin ()
   from /home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-1c5da71f0b2b75c2.so
#16 0x00007fbaf6bffbf0 in rustc_codegen_ssa::back::lto::LtoModuleCodegen<B>::optimize ()
   from /home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-1c5da71f0b2b75c2.so
#17 0x00007fbaf6d44d77 in rustc_codegen_ssa::back::write::execute_work_item () from /home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-1c5da71f0b2b75c2.so
#18 0x00007fbaf6c6953e in std::sys_common::backtrace::__rust_begin_short_backtrace () from /home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-1c5da71f0b2b75c2.so
#19 0x00007fbaf6c732d7 in core::ops::function::FnOnce::call_once{{vtable-shim}} () from /home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-1c5da71f0b2b75c2.so
#20 0x00007fbaf57c3cc8 in std::sys::unix::thread::Thread::new::thread_start () from /home/matthias/vcs/github/rust/build/x86_64-unknown-linux-gnu/stage1/lib/libstd-a52de4cbb300e5a7.so
#21 0x00007fbaf23493e9 in start_thread () from /usr/lib/libpthread.so.0
#22 0x00007fbaf55d1293 in clone () from /usr/lib/libc.so.6

does this help?

[matthiaskrgr]

config.toml via cat config.toml | grep -v "^\#" | grep "."

[llvm]
thin-lto = true
ninja = true
targets = "X86"
link-jobs = 2
[build]
extended = true
tools = ["cargo", "rls", "clippy", "rustfmt", "analysis", "src", "miri", "rust-analyzer"]
sanitizers = true
profiler = true
low-priority = true
[install]
[rust]
codegen-units = 0
backtrace-on-ice = true
[target.x86_64-unknown-linux-gnu]
ar = "llvm-ar"
ranlib = "llvm-ranlib"
[dist]

[matthiaskrgr]

There is no segfault if I build without -Ctarget-cpu=native

[Dylan-DPC-zz]

Marking this as P-high based on the wg-prioritization discussion

[Aaron1011]

@matthiaskrgr: What setup are you using to pass -Ctarget-cpu=native?

[matthiaskrgr]

I had export RUSTFLAGS="-C target-cpu=native" in my ~/.zsh.rs initially.

RUSTFLAGS="-Ctarget-cpu=native" ./x.py build reproduces the segfault as well.