ICE on extremely long string literal (larger than 4G)

[oxalica]

I know that having huge literal in source is a bad idea, but it should not cause ICE :(
It looks like something overflowed.

Version: rustc 1.37.0-nightly (4edff843d 2019-06-16) running on x86_64-unknown-linux-gnu

Command, output and backtrace:

$ RUST_BACKTRACE=full rustc ./huge.rs
thread 'rustc' panicked at 'begin <= end (22 <= 21) when slicing `const HELLO: &str = r"
hello!!
hello!!
hello!!
hello!!
hello!!
hello!!
hello!!
hello!!
hello!!
hello!!
hello!!
hello!!
hello!!
hello!!
hello!!
hello!!
hello!!
hello!!
hello!!
hello!!
hello!!
hello!!
hello!!
hello!!
hello!!
hello!!
hello!!
hello!!
hello!!
h`[...]', src/libcore/str/mod.rs:2021:5
stack backtrace:
   0:     0x7fc8de00011b - backtrace::backtrace::libunwind::trace::h912e0d04a7708cf2
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.29/src/backtrace/libunwind.rs:88
   1:     0x7fc8de00011b - backtrace::backtrace::trace_unsynchronized::hfb708595cf87b63f
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.29/src/backtrace/mod.rs:66
   2:     0x7fc8de00011b - std::sys_common::backtrace::_print::ha79f66d09fb9689e
                               at src/libstd/sys_common/backtrace.rs:47
   3:     0x7fc8de00011b - std::sys_common::backtrace::print::hbbf6608241e58d95
                               at src/libstd/sys_common/backtrace.rs:36
   4:     0x7fc8de00011b - std::panicking::default_hook::{{closure}}::h5463a2f4723f6cb3
                               at src/libstd/panicking.rs:198
   5:     0x7fc8ddfffe1e - std::panicking::default_hook::h79a408572fcba4ed
                               at src/libstd/panicking.rs:212
   6:     0x7fc8dbd92351 - rustc::util::common::panic_hook::h318156cfb41f5413
   7:     0x7fc8de000979 - std::panicking::rust_panic_with_hook::hb67cd2b0c77d9895
                               at src/libstd/panicking.rs:479
   8:     0x7fc8de000412 - std::panicking::continue_panic_fmt::he44145e3b8e029cd
                               at src/libstd/panicking.rs:382
   9:     0x7fc8de0002f6 - rust_begin_unwind
                               at src/libstd/panicking.rs:309
  10:     0x7fc8de02e11d - core::panicking::panic_fmt::h7327567cfbeab565
                               at src/libcore/panicking.rs:85
  11:     0x7fc8de030740 - core::str::slice_error_fail::hba2c24f2962448b8
                               at src/libcore/str/mod.rs:0
  12:     0x7fc8db0dd012 - core::str::traits::<impl core::slice::SliceIndex<str> for core::ops::range::Range<usize>>::index::{{closure}}::he3b193b38a91659a
  13:     0x7fc8db0ebf50 - syntax::parse::lexer::StringReader::with_str_from_to::h0d0fa368b8b05ffc
  14:     0x7fc8db0e9916 - syntax::parse::lexer::StringReader::advance_token::hc406a5a0c98fe430
  15:     0x7fc8db0e766d - syntax::parse::lexer::StringReader::try_next_token::habfb2fe795d28efa
  16:     0x7fc8db0e7a9f - syntax::parse::lexer::StringReader::real_token::hfbd846ba27af2f28
  17:     0x7fc8db115ef1 - syntax::parse::lexer::tokentrees::TokenTreesReader::parse_token_tree::h8061afbb3e86c76b
  18:     0x7fc8db1155ce - syntax::parse::lexer::tokentrees::TokenTreesReader::parse_all_token_trees::h8018bcf24cc2951d
  19:     0x7fc8db0e72f5 - syntax::parse::lexer::tokentrees::<impl syntax::parse::lexer::StringReader>::into_token_trees::h35e2e07fb40a0abc
  20:     0x7fc8db0035af - syntax::parse::maybe_file_to_stream::h77846087608aa34a
  21:     0x7fc8db002840 - syntax::parse::maybe_source_file_to_parser::h109680085ff17678
  22:     0x7fc8db0022b4 - syntax::parse::source_file_to_parser::h7859d86f2a1b8e8a
  23:     0x7fc8db00164a - syntax::parse::parse_crate_from_file::h72b9587b203687b8
  24:     0x7fc8dda73bfd - rustc_interface::passes::parse::{{closure}}::h73af6c8d58be2c8e
  25:     0x7fc8dda6e476 - rustc::util::common::time::hedf145db853fee22
  26:     0x7fc8dda12c6f - rustc_interface::passes::parse::h6339ba714c071498
  27:     0x7fc8dda02653 - rustc_interface::queries::Query<T>::compute::h59a144469961cc35
  28:     0x7fc8ddaebb84 - rustc_interface::queries::<impl rustc_interface::interface::Compiler>::parse::hc1befc73f5446e68
  29:     0x7fc8de2f3a71 - rustc_interface::interface::run_compiler_in_existing_thread_pool::h14bf4a38c2714175
  30:     0x7fc8de3650a6 - std::thread::local::LocalKey<T>::with::h1ef421860970c1a1
  31:     0x7fc8de2ff0ae - scoped_tls::ScopedKey<T>::set::hae697baf57baa69a
  32:     0x7fc8de331124 - syntax::with_globals::h2ba7ae4c00aa57d8
  33:     0x7fc8de34fdfd - std::sys_common::backtrace::__rust_begin_short_backtrace::hbacd46537732a9e0
  34:     0x7fc8de01181a - __rust_maybe_catch_panic
                               at src/libpanic_unwind/lib.rs:82
  35:     0x7fc8de2dc2d9 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h5f09d41df1db8d36
  36:     0x7fc8ddfe3e1f - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h6a25020f908b1369
                               at /rustc/4edff843dd219cf19a5fede6c78c7ce95402e1f5/src/liballoc/boxed.rs:746
  37:     0x7fc8de010500 - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::he4b68ab6cba491dc
                               at /rustc/4edff843dd219cf19a5fede6c78c7ce95402e1f5/src/liballoc/boxed.rs:746
  38:     0x7fc8de010500 - std::sys_common::thread::start_thread::hf0f92ea133daa112
                               at src/libstd/sys_common/thread.rs:13
  39:     0x7fc8de010500 - std::sys::unix::thread::Thread::new::thread_start::h9d375b1cdab5f2d3
                               at src/libstd/sys/unix/thread.rs:79
  40:     0x7fc8ddf7eef7 - start_thread
  41:     0x7fc8ddea322f - __GI___clone
  42:                0x0 - <unknown>
query stack during panic:
end of query stack

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.37.0-nightly (4edff843d 2019-06-16) running on x86_64-unknown-linux-gnu

Here is the generator of the source:

out="./huge.rs"
echo 'const HELLO: &str = r"' >$out
yes "hello!!" | head -c 4294967296 >>$out
echo '";fn main() { println!("{}", HELLO); }' >>$out

[Centril]

cc @petrochenkov @matklad

[matklad]

Took a quick look. We certainly have a restriction that any file can't be that large, because we use u32 for indexing. And looks like we never try to enforce it anywhere. I think the place to do that would be SourceFile::new.

Do we have any way to panic without this being marked as ICE? I don't want to thread Handler to each SourceFile::new, and this seems like a marginal error where it's totally OK to panic. This is even OK in the IDE context I think: IDE should enforce reasonable max size of the file (something like 10mb, not 4gb) at the virtual file system layer.

The "just panic" fix is here: matklad@8fd5b42

[petrochenkov]

Do we have any way to panic without this being marked as ICE?

FatalError.raise() or span_fatal if the context is available.

[tesuji]

I tried to compile the same thing in gcc and clang:

#!/bin/sh
printf '#include <stdio.h>\nchar s[] = "' > hello.c
python -c 'print("h" * 4294967296)' >> hello.c
truncate -s -1 hello.c

printf '";' >> hello.c
cat >> hello.c << EOF

int main()
{
  printf("%s\n", s);
  return 0;
}
EOF
gcc hello.c -o hello_gcc
clang hello.c -o hello_clang # crash

With gcc:

% gdb hello_gcc
Reading symbols from a.out...(no debugging symbols found)...done.
(gdb) disas main
Dump of assembler code for function main:
   0x0000000000001130 <+0>:     push   rbp
   0x0000000000001131 <+1>:     mov    rbp,rsp
   0x0000000000001134 <+4>:     lea    rdi,[rip+0x2ec6]        # 0x4001 <s>
   0x000000000000113b <+11>:    call   0x1210 <puts@plt>
   0x0000000000001140 <+16>:    mov    eax,0x0
   0x0000000000001145 <+21>:    pop    rbp
   0x0000000000001146 <+22>:    ret
End of assembler dump.
(gdb) x/s 0x4001
0x4001 <s>:     ""
(gdb) quit

With clang:

% clang hello.c
Stack dump:
0.      Program arguments: /home/lzutao/.local/bin/clang-8 -cc1 -triple x86_64-unknown-linux-gnu -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names -main-file-name hello.c -mrelocation-model static -mthread-model posix -mdisable-fp-elim -fmat
h-errno -masm-verbose -mconstructor-aliases -munwind-tables -fuse-init-array -target-cpu x86-64 -dwarf-column-info -debugger-tuning=gdb -resource-dir /home/lzutao/.local/lib/clang/8.0.0 -internal-isystem /usr/local/include -internal-isystem /home/lzutao/.local/lib/clang/
8.0.0/include -internal-externc-isystem /home/lzutao/.local/include -internal-externc-isystem /usr/lib/gcc/x86_64-linux-gnu/6/include-fixed -internal-externc-isystem /usr/include -internal-externc-isystem /usr/local/include -internal-externc-isystem /usr/include/x86_64-l
inux-gnu -fdebug-compilation-dir /home/lzutao/forked/rust/check -ferror-limit 19 -fmessage-length 135 -fobjc-runtime=gcc -fdiagnostics-show-option -fcolor-diagnostics -o /tmp/hello-27434e.o -x c hello.c -faddrsig
1.      clang-8: error: unable to execute command: Alarm clock
clang-8: error: clang frontend command failed due to signal (use -v to see invocation)
clang version 8.0.0
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/lzutao/.local/bin
clang-8: note: diagnostic msg: PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace, preprocessed source, and associated run script.
clang-8: note: diagnostic msg:
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang-8: note: diagnostic msg: /tmp/hello-943593.c
clang-8: note: diagnostic msg: /tmp/hello-943593.sh
clang-8: note: diagnostic msg:

********************

Meta

% clang --version
clang version 8.0.0
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/lzutao/.local/bin
% gcc --version
gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

EDIT: Opened https://bugs.llvm.org/show_bug.cgi?id=42301