Compiler OOM when creating extremely large array of `String` using `const` with const evaluation

[5225225]

I tried this code:

pub fn other() {
    const S: String = String::new();
    let _ = [S; usize::MAX >> 5];
}

I expected to see this happen: Either an error about the value being too large for the current architecture, or nothing (because errors about too large arrays don't seem to happen for values bound to _)

Instead, this happened:

# rustc main.rs        
memory allocation of 13835058055282163688 bytes failed
zsh: IOT instruction (core dumped)  rustc main.rs

Meta

rustc --version --verbose:

rustc 1.61.0-nightly (532d3cda9 2022-02-23)
binary: rustc
commit-hash: 532d3cda90b8a729cd982548649d32803d265052
commit-date: 2022-02-23
host: x86_64-unknown-linux-gnu
release: 1.61.0-nightly
LLVM version: 14.0.0

Valgrind with trace-children=yes gives me the following location for the error

Backtrace

==2782416== Argument 'size' of function malloc has a fishy (possibly negative) value: -4611686018427387928
==2782416==    at 0x4845899: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==2782416==    by 0x6F4982E: _RNvMs_NtCsc8QzWtcr2xm_5alloc7raw_vecINtB4_6RawVecINtNtCsd3w0T8MxtOr_12rustc_middle3mir14ProjectionElemNtBO_5LocalNtNtBQ_2ty2TyEE11allocate_inCs3FhhpBtephy_19rustc_mir_transform (in /home/jess/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-920796763ff499ad.so)
==2782416==    by 0x50D8F5F: _RNvXNtNtCsc8QzWtcr2xm_5alloc3vec14spec_from_iterINtB4_3VecTNtNtCsd3w0T8MxtOr_12rustc_middle3mir5PlaceINtNtCsffm2ZuYuepl_4core6option6OptionuEEEINtB2_12SpecFromIterBU_INtNtNtNtB1G_4iter8adapters3map3MapINtNtNtB1G_3ops5range5RangeyENCNvMs0_NtCsdKNwjzyVHJ1_18rustc_mir_dataflow15elaborate_dropsINtB3O_8DropCtxtNtNtCs3FhhpBtephy_19rustc_mir_transform4shim18DropShimElaboratorE19open_drop_for_array0EE9from_iterB4Z_ (in /home/jess/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-920796763ff499ad.so)
==2782416==    by 0x6F08403: _RNvMs0_NtCsdKNwjzyVHJ1_18rustc_mir_dataflow15elaborate_dropsINtB5_8DropCtxtNtNtCs3FhhpBtephy_19rustc_mir_transform4shim18DropShimElaboratorE19open_drop_for_arrayB1f_ (in /home/jess/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-920796763ff499ad.so)
==2782416==    by 0x6F05E89: _RNvMs0_NtCsdKNwjzyVHJ1_18rustc_mir_dataflow15elaborate_dropsINtB5_8DropCtxtNtNtCs3FhhpBtephy_19rustc_mir_transform4shim18DropShimElaboratorE14elaborate_dropB1f_.llvm.4668469828716773882 (in /home/jess/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-920796763ff499ad.so)
==2782416==    by 0x6F4343A: _RNvNtCs3FhhpBtephy_19rustc_mir_transform4shim9make_shim.llvm.8089105803349852858 (in /home/jess/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-920796763ff499ad.so)
==2782416==    by 0x71DB720: _RINvNtNtCsjJfV0yDsyeG_18rustc_query_system5query8plumbing17try_execute_queryNtNtCsgV5xg9Y7H18_16rustc_query_impl8plumbing9QueryCtxtINtNtB4_6caches10ArenaCacheNtNtNtCsd3w0T8MxtOr_12rustc_middle2ty8instance11InstanceDefNtNtB2C_3mir4BodyEEB1g_.llvm.2504427033025671314 (in /home/jess/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-920796763ff499ad.so)
==2782416==    by 0x7265451: _RINvNtNtCsjJfV0yDsyeG_18rustc_query_system5query8plumbing9get_queryNtNtCsgV5xg9Y7H18_16rustc_query_impl7queries9mir_shimsNtNtB17_8plumbing9QueryCtxtEB17_ (in /home/jess/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-920796763ff499ad.so)
==2782416==    by 0x72763B3: _RNvXs8A_CsgV5xg9Y7H18_16rustc_query_implNtB6_7QueriesNtNtNtCsd3w0T8MxtOr_12rustc_middle2ty5query11QueryEngine9mir_shims (in /home/jess/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-920796763ff499ad.so)
==2782416==    by 0x6C322F1: _RNvMsE_NtCsd3w0T8MxtOr_12rustc_middle2tyNtNtB5_7context6TyCtxt12instance_mir (in /home/jess/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-920796763ff499ad.so)
==2782416==    by 0x61A1915: _RNvNtCsl186E7FnW1R_18rustc_monomorphize9collector18collect_neighbours (in /home/jess/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-920796763ff499ad.so)
==2782416==    by 0x619CAC3: _RNvNtCsl186E7FnW1R_18rustc_monomorphize9collector17collect_items_rec (in /home/jess/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-920796763ff499ad.so)
==2782416== 
memory allocation of 13835058055282163688 bytes failed
==2782416== 
==2782416== Process terminating with default action of signal 6 (SIGABRT): dumping core
==2782416==    at 0x8BFE34C: __pthread_kill_implementation (in /usr/lib/libc.so.6)
==2782416==    by 0x8BB14B7: raise (in /usr/lib/libc.so.6)
==2782416==    by 0x8B9B533: abort (in /usr/lib/libc.so.6)
==2782416==    by 0x88A8876: std::sys::unix::abort_internal (mod.rs:259)
==2782416==    by 0x8862E85: std::process::abort (process.rs:2003)
==2782416==    by 0x889B12D: rust_oom (alloc.rs:330)
==2782416==    by 0x88EB146: __rg_oom (alloc.rs:411)
==2782416==    by 0x88EB0C6: alloc::alloc::handle_alloc_error::rt_error (alloc.rs:377)
==2782416==    by 0x88EAE96: core::ops::function::FnOnce::call_once (function.rs:227)
==2782416==    by 0x88EB5C5: core::intrinsics::const_eval_select (intrinsics.rs:2366)
==2782416==    by 0x8864585: alloc::alloc::handle_alloc_error (alloc.rs:381)
==2782416==    by 0x6F49870: _RNvMs_NtCsc8QzWtcr2xm_5alloc7raw_vecINtB4_6RawVecINtNtCsd3w0T8MxtOr_12rustc_middle3mir14ProjectionElemNtBO_5LocalNtNtBQ_2ty2TyEE11allocate_inCs3FhhpBtephy_19rustc_mir_transform (in /home/jess/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-920796763ff499ad.so)

Which I traced down to probably this line. I'm assuming size is the array size here, but I have not verified that.

rust/compiler/rustc_mir_dataflow/src/elaborate_drops.rs

Lines 737 to 751 in c8a49fc

	let fields: Vec<(Place<'tcx>, Option<D::Path>)> = (0..size)
	.map(|i| {
	(
	tcx.mk_place_elem(
	self.place,
	ProjectionElem::ConstantIndex {
	offset: i,
	min_length: size,
	from_end: false,
	},
	),
	self.elaborator.array_subpath(self.path, i, size),
	)
	})
	.collect();

[5225225]

The fix here might be similar to the fix in #86255? Try and reserve the Vec::with_capacity(size), then if that fails, report an error? Not sure, especially since it looks like this whole area of code has no way to report errors, so that would need to be added.

[RalfJung]

Cc @rust-lang/wg-const-eval

[Enselic]

Triage: This seems fixed now:

$ cat src/main.rs
pub fn main() {
    const S: String = String::new();
    let _ = [S; usize::MAX >> 5];
}
$ cargo build
   Compiling minimal v0.1.0 (/home/martin/src/bin)
error: values of the type `[String; 576460752303423487]` are too big for the current architecture

error: could not compile `minimal` (bin "minimal") due to 1 previous error

I'm going to assume a regression test has been added when this was fixed, because I would be very surprised if that wasn't the case. Closing.

[RalfJung]

I think that just indicates that we now properly skip const-eval if the type is truly unrepresentable.

But for a type that's big but within the size limit, I don't think anything got fixed here. Should be easy to reproduce by making the array size smaller until the "too big for the current architecture" error disappears.

[5225225]

Hm...

pub fn other() {
    const S: String = String::new();
    let _ = [S; usize::MAX >> 22];
}

fn main() {
    other();
}

does not give any error, and it compiles fine. (The binary it produces gives a stack overflow, but that is to be expected.) If rustc truly was trying to actually allocate that many Strings, that would have failed compilation.

I'll try bisecting this later and seeing what caused this to work, and if it was intentionally done or if it's just work that happened to be skipped.

[5225225]

unfortunately,

********************************************************************************
Regression in nightly-2023-04-03
********************************************************************************
ending github query because we found starting sha: 0599b6b931816ab46ab79072189075f543931cbd
get_commits_between returning commits, len: 8
  commit[0] 2023-04-01: Auto merge of #109483 - joboet:optimize_lazycell, r=Mark-Simulacrum
  commit[1] 2023-04-01: Auto merge of #109833 - klensy:bs-tty, r=Mark-Simulacrum
  commit[2] 2023-04-02: Auto merge of #109834 - Nilstrieb:mailmap-myself, r=Mark-Simulacrum
  commit[3] 2023-04-02: Auto merge of #109836 - clubby789:param-non-exhaustive, r=Nilstrieb
  commit[4] 2023-04-02: Auto merge of #109701 - Amanieu:binaryheap_retain, r=ChrisDenton
  commit[5] 2023-04-02: Auto merge of #109852 - Nilstrieb:rollup-g3mgxxw, r=Nilstrieb
  commit[6] 2023-04-02: Auto merge of #109008 - clubby789:drop-elaborate-array, r=davidtwco
  commit[7] 2023-04-02: Auto merge of #109811 - jyn514:symlink-fixes, r=Mark-Simulacrum
ERROR: no CI builds available between 0599b6b931816ab46ab79072189075f543931cbd and 3a8a131e9509c478ece1c58fe0ea2d49463d2300 within last 167 days

was fixed quite a while ago. I'll go through these later, it might be pretty obvious which one it is (i hope :))

[RalfJung]

Oh right, this isn't actually a huge const, just a huge array of consts.

Still, there may or may not have been a regression test added.

[5225225]

#109008 looks the most relevant here

in particular, the regression test there is:

const SZ: usize = 64_000_000;
type BigDrop = [String; SZ];

fn f(_dropme: BigDrop) {}

fn f2(_moveme: BigDrop) -> String {
    let [a, ..] = _moveme;
    a
}

fn main() {
    f(std::array::from_fn(|_| String::new()));
    f2(std::array::from_fn(|_| String::new()));
}

is probably the issue that was being ran into here. (I think this issue is a dupe of #109004).

[RalfJung]

Ah, yes that looks very similar. Thanks for digging this out!
I agree this can be closed then.