rustc overflowed its stack when compiling 148,492 lines of code.

[SunHao-0]

Rustc overflowed its stack when compiling 148,492 lines of code.
Rust is great, but this limitation is unacceptable.

Error

> cargo build
thread 'rustc' has overflowed its stack
fatal runtime error: stack overflow
error: could not compile `hl-fuzzer`
Caused by:
  process didn't exit successfully: `rustc --crate-name hl_fuzzer --edition=2018 fuzzer/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="amd64-linux"' --cfg 'feature="default"' -C metadata=258ea3d5d4c9e7e0 -C extra-filename=-258ea3d5d4c9e7e0 --out-dir .../target/debug/deps -C incremental=.../target/debug/incremental -L dependency=.../target/debug/deps --extern hlang=.../target/debug/deps/libhlang-7040598e8269a156.rmeta` (signal: 6, SIGABRT: process abort signal)

Crate

Loc: 148,492
Large file: fuzzer/syscalls/linux/amd64.rs
Link: https://github.com/SunHao-0/healer/tree/refactor

Hardware

Cpu

Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   39 bits physical, 48 bits virtual
CPU(s):                          16
On-line CPU(s) list:             0-15
Thread(s) per core:              2
Core(s) per socket:              8
Socket(s):                       1
*Memory*
RANGE                                  SIZE  STATE REMOVABLE  BLOCK
0x0000000000000000-0x000000009fffffff  2.5G online       yes   0-19
0x0000000100000000-0x000000045fffffff 13.5G online       yes 32-139

Memory block size:       128M
Total online memory:      16G
Total offline memory:      0B

Resource limitation

-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             1048576
-c: core file size (blocks)         unlimited
-m: resident set size (kbytes)      unlimited
-u: processes                       unlimited
-n: file descriptors                1024
-l: locked-in-memory size (kbytes)  64
-v: address space (kbytes)          unlimited
...

Meta

rustc 1.49.0-nightly (ffa2e7ae8 2020-10-24)
binary: rustc
commit-hash: ffa2e7ae8fbf9badc035740db949b9dae271c29f
commit-date: 2020-10-24
host: x86_64-unknown-linux-gnu
release: 1.49.0-nightly
LLVM version: 11.0

[Mark-Simulacrum]

Can you collect a backtrace on the overflowing stack?

[SunHao-0]

> gdb --args rustc --crate-name hl_fuzzer --edition=2018 fuzzer/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="amd64-linux"' --cfg 'feature="default"' -C metadata=258ea3d5d4c9e7e0 -C extra-filename=-258ea3d5d4c9e7e0 --out-dir .../target/debug/deps -C incremental=.../target/debug/incremental -L dependency=.../target/debug/deps --extern hlang=.../target/debug/deps/libhlang-7040598e8269a156.rmeta

# Recursion starts here, leads to stack overflow.
...
#74319 0x00007ffff5510bb9 in rustc_data_structures::graph::iterate::post_order_walk () from .../nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74320 0x00007ffff5510bb9 in rustc_data_structures::graph::iterate::post_order_walk () from .../nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74321 0x00007ffff5510bb9 in rustc_data_structures::graph::iterate::post_order_walk () from .../nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74325 0x00007ffff5510ced in rustc_data_structures::graph::iterate::reverse_post_order () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74326 0x00007ffff5276f8d in rustc_data_structures::graph::dominators::dominators () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74327 0x00007ffff559b40b in rustc_mir::borrow_check::do_mir_borrowck () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74328 0x00007ffff50285e6 in rustc_infer::infer::InferCtxtBuilder::enter () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74329 0x00007ffff5596f49 in rustc_mir::borrow_check::mir_borrowck () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74330 0x00007ffff5560e35 in core::ops::function::FnOnce::call_once () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74331 0x00007ffff3fa7d25 in rustc_middle::ty::query::<impl rustc_query_system::query::config::QueryAccessors<rustc_middle::ty::context::TyCtxt> for rustc_middle::ty::query::queries::mir_borrowck>::compute () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74332 0x00007ffff3f4cf81 in rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74333 0x00007ffff3f11e87 in rustc_query_system::dep_graph::graph::DepGraph<K>::with_task_impl () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74334 0x00007ffff3fa9c41 in rustc_data_structures::stack::ensure_sufficient_stack () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74335 0x00007ffff3ef0b5c in rustc_query_system::query::plumbing::get_query_impl () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74336 0x00007ffff3ef5ff9 in rustc_query_system::query::plumbing::ensure_query_impl () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74337 0x00007ffff3fa4f73 in rustc_interface::passes::analysis () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74338 0x00007ffff3cded7b in rustc_middle::ty::query::<impl rustc_query_system::query::config::QueryAccessors<rustc_middle::ty::context::TyCtxt> for rustc_middle::ty::query::queries::analysis>::compute () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74339 0x00007ffff3ce0df8 in rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74340 0x00007ffff3d7d0e4 in rustc_query_system::dep_graph::graph::DepGraph<K>::with_task_impl () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74341 0x00007ffff3d783df in rustc_middle::ty::query::plumbing::<impl rustc_query_system::query::QueryContext for rustc_middle::ty::context::TyCtxt>::start_query::{{closure}}::{{closure}}::{{closure}} () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74342 0x00007ffff3d0a444 in rustc_query_system::query::plumbing::get_query_impl () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74343 0x00007ffff3ce112b in rustc_interface::passes::QueryContext::enter () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74344 0x00007ffff3d7a4a2 in rustc_interface::queries::<impl rustc_interface::interface::Compiler>::enter () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74345 0x00007ffff3d412a2 in rustc_span::with_source_map () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74346 0x00007ffff3d7b9c2 in rustc_interface::interface::create_compiler_and_run () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74347 0x00007ffff3d63d5a in scoped_tls::ScopedKey<T>::set () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74348 0x00007ffff3d809b5 in std::sys_common::backtrace::__rust_begin_short_backtrace () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74349 0x00007ffff3cfe3ce in core::ops::function::FnOnce::call_once{{vtable-shim}} () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74350 0x00007ffff34a621a in <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once () at /rustc/ffa2e7ae8fbf9badc035740db949b9dae271c29f/library/alloc/src/boxed.rs:1042
#74351 <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once () at /rustc/ffa2e7ae8fbf9badc035740db949b9dae271c29f/library/alloc/src/boxed.rs:1042
#74352 std::sys::unix::thread::Thread::new::thread_start () at library/std/src/sys/unix/thread.rs:89
#74353 0x00007ffff33d0609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#74354 0x00007ffff32e4293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
`

[SunHao-0]

By the way, if this pr(#78588) solved the problem, how can I get the patched executable rustc as soon as possible? Will CI upload the compiled rustc to somewhere?

[Mark-Simulacrum]

rustup-toolchain-install-master 761d47a65a41279f919a16d8b59f188b14a70617 should work; see https://crates.io/crates/rustup-toolchain-install-master/ to install it.

[SunHao-0]

Problem not solved

rustup-toolchain-install-master 761d47a65a41279f919a16d8b59f188b14a70617 should work;

Rustc with 761d47 commit hash still overflowed, when compiling my project.
The backtrace is the same as above.

...
#74319 0x00007ffff5510bb9 in rustc_data_structures::graph::iterate::post_order_walk () from .../nightly-x86_64-unknown-linux- gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74320 0x00007ffff5510bb9 in rustc_data_structures::graph::iterate::post_order_walk () from .../nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74321 0x00007ffff5510bb9 in rustc_data_structures::graph::iterate::post_order_walk () from .../nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74325 0x00007ffff5510ced in rustc_data_structures::graph::iterate::reverse_post_order () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
#74326 0x00007ffff5276f8d in rustc_data_structures::graph::dominators::dominators () from .../toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-49c72af66a9d25fd.so
...

Rustc version

rustc 1.49.0-nightly (761d47a65 2020-10-31)
binary: rustc
commit-hash: 761d47a65a41279f919a16d8b59f188b14a70617
commit-date: 2020-10-31
host: x86_64-unknown-linux-gnu
release: 1.49.0-nightly

[SunHao-0]

Thanks for the quick response. Unfortunately, this pr(#78607) didn't fix the problem, rustc still overflowed it stack but with a slower speed. For now, I'll change the code generation logic of my crate to split the long function into multiple shorter functions. However, this kind of limitation should not exist in rustc.

Version

> rustc --version --verbose
rustc 1.49.0-nightly (b0d26b017 2020-11-02)
binary: rustc
commit-hash: b0d26b01718f8a86b4ee8dfdb0508d3a5d49e19b
commit-date: 2020-11-02
host: x86_64-unknown-linux-gnu
release: 1.49.0-nightly

Hardware(updated)

> lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              80
On-line CPU(s) list: 0-79
Thread(s) per core:  2
Core(s) per socket:  20
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               85
...

> lsmem
RANGE                                  SIZE  STATE REMOVABLE BLOCK
0x0000000000000000-0x000000007fffffff    2G online       yes     0
0x0000000100000000-0x000000207fffffff  126G online       yes  2-64

Memory block size:         2G
Total online memory:     128G
Total offline memory:      0B