unstable fingerprints for item_children - ICE on Rust 1.52.0 stable

[ohsayan]

Code

Skytable is quite a large project and I'm trying to produce a MCVE as soon as I can.
You can find the relevant logs on our CI server: https://build.skytable.io/skytable/skytable/98/1/3. From what I can see, this only happens on 1.52.0 and I'm supposing this is a regression from stable. For what it's worth, our CI server is running on an M1 Chip (aarch64-apple-darwin). From the backtrace it seems like the compiler panics while trying to build the sky-bench component or libsky (which is the smallest part of the project)

Update [August-08-2021]: After upgrading to 1.54, ICEs continue to appear. See this comment

Update [August-09-2021] @Aaron1011 has investigated this extensively. Please refer to this comment and see the log here

Runs

• Run 1: https://build.skytable.io/skytable/skytable/98/1/3
• Run 2: https://build.skytable.io/skytable/skytable/99/1/3
• Run 3: https://build.skytable.io/skytable/skytable/100/1/3

(can repro on every re-run)

Meta

rustc --version --verbose:

rustc 1.52.0 (88f19c6da 2021-05-03)
binary: rustc
commit-hash: 88f19c6dab716c6281af7602e30f413e809c5974
commit-date: 2021-05-03
host: aarch64-apple-darwin
release: 1.52.0
LLVM version: 12.0.0

Backtrace

thread 'rustc' panicked at 'found unstable fingerprints for item_children(libsky[d7cb]): [Export { ident: terrapipe#0, res: Def(Mod, DefId(27:3 ~ libsky[d7cb]::terrapipe)), span: /private/tmp/drone-XTLrTmOnXHGBgymS/drone/src/libsky/src/lib.rs:31:1: 31:19 (#0), vis: Public }, Export { ident: util#0, res: Def(Mod, DefId(27:24 ~ libsky[d7cb]::util)), span: /private/tmp/drone-XTLrTmOnXHGBgymS/drone/src/libsky/src/lib.rs:32:1: 32:14 (#0), vis: Public }, Export { ident: TResult#0, res: Def(TyAlias, DefId(27:61 ~ libsky[d7cb]::TResult)), span: /private/tmp/drone-XTLrTmOnXHGBgymS/drone/src/libsky/src/lib.rs:35:1: 35:49 (#0), vis: Public }, Export { ident: BUF_CAP#0, res: Def(Const, DefId(27:63 ~ libsky[d7cb]::BUF_CAP)), span: /private/tmp/drone-XTLrTmOnXHGBgymS/drone/src/libsky/src/lib.rs:37:1: 37:37 (#0), vis: Public }, Export { ident: Error#0, res: Def(Trait, DefId(1:2324 ~ std[abff]::error::Error)), span: /private/tmp/drone-XTLrTmOnXHGBgymS/drone/src/libsky/src/lib.rs:33:5: 33:22 (#0), vis: Restricted(DefId(27:0 ~ libsky[d7cb])) }]', /rustc/88f19c6dab716c6281af7602e30f413e809c5974/compiler/rustc_query_system/src/query/plumbing.rs:593:5
stack backtrace:
   0:        0x1074f6c54 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h7cf4b69c45a99b1c
   1:        0x107552400 - core::fmt::write::h05b8a9712099e9e5
   2:        0x1074eb850 - std::io::Write::write_fmt::hfe8917b7b12925fd
   3:        0x1074fa3d0 - std::panicking::default_hook::{{closure}}::hda4b0dbd83d08efe
   4:        0x1074f9ef8 - std::panicking::default_hook::h713df0508ca1a369
   5:        0x100b9de9c - rustc_driver::report_ice::h621c43280551f1ad
   6:        0x1074faaf4 - std::panicking::rust_panic_with_hook::h70f0bcacc0251305
   7:        0x1074fa6b8 - std::panicking::begin_panic_handler::{{closure}}::h0a11da6e0d868b2b
   8:        0x1074f70e4 - std::sys_common::backtrace::__rust_end_short_backtrace::h0e50d651e2e8352a
   9:        0x1074fa620 - _rust_begin_unwind
  10:        0x1075785a4 - std::panicking::begin_panic_fmt::h27d289fcb95248ef
  11:        0x102faf318 - rustc_query_system::query::plumbing::incremental_verify_ich::h3b795f48fda514bf
  12:        0x102fd24b4 - rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory::hd489dd83686ae20b
  13:        0x103207d18 - rustc_data_structures::stack::ensure_sufficient_stack::h984bf9678df70bcb
  14:        0x102f178ec - rustc_query_system::query::plumbing::get_query_impl::h8acc6a801e6ac255
  15:        0x10311aa60 - <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::item_children::hec4321eb38d39236
  16:        0x10397a768 - core::ops::function::FnOnce::call_once::h43db234dfbd2fd0c
  17:        0x1031d0a54 - rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps::h04742382a4263eef
  18:        0x102fcaa70 - rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory::h7682e4beeea09fbc
  19:        0x1031fb2f0 - rustc_data_structures::stack::ensure_sufficient_stack::h4cbe0bdcb561e992
  20:        0x102f00fe0 - rustc_query_system::query::plumbing::get_query_impl::h5216c9b499f69736
  21:        0x10311b034 - <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::visible_parent_map::h3768502f3a14842a
  22:        0x103fe7694 - rustc_middle::ty::print::pretty::PrettyPrinter::try_print_visible_def_path_recur::h824307ccd6064e73
  23:        0x103ff0de0 - <rustc_middle::ty::print::pretty::FmtPrinter<F> as rustc_middle::ty::print::Printer>::print_def_path::hd4ae041fed231929
  24:        0x1040002bc - rustc_middle::ty::print::Printer::default_print_def_path::h99e342df73393194
  25:        0x103ff0f44 - <rustc_middle::ty::print::pretty::FmtPrinter<F> as rustc_middle::ty::print::Printer>::print_def_path::hd4ae041fed231929
  26:        0x1040016b0 - <rustc_middle::ty::print::pretty::TraitRefPrintOnlyTraitPath as core::fmt::Display>::fmt::h305888562eeeeb09
  27:        0x107552400 - core::fmt::write::h05b8a9712099e9e5
  28:        0x103ecfed4 - rustc_middle::ty::print::pretty::<impl rustc_middle::ty::print::Print<P> for rustc_middle::ty::sty::TraitRef>::print::h20133c6e2b9ec55e
  29:        0x103ed0ccc - rustc_middle::ty::print::pretty::<impl core::fmt::Display for rustc_middle::ty::sty::TraitRef>::fmt::h7333ed8f621fd019
  30:        0x103ec77b8 - rustc_middle::ty::structural_impls::<impl core::fmt::Debug for rustc_middle::ty::sty::TraitRef>::fmt::ha19f762841df7f58
  31:        0x107552400 - core::fmt::write::h05b8a9712099e9e5
  32:        0x1075530dc - <core::fmt::Formatter as core::fmt::Write>::write_fmt::h9a11944cd06b3d42
  33:        0x103fd8ca0 - rustc_middle::ty::structural_impls::<impl core::fmt::Debug for rustc_middle::ty::PredicateKind>::fmt::h567256ec1ce234c5
  34:        0x10754f3a4 - core::fmt::builders::DebugTuple::field::hded5bed1a041d37c
  35:        0x103ed4c7c - <rustc_middle::ty::sty::Binder<T> as core::fmt::Debug>::fmt::hcc79fe5855ca8a5b
  36:        0x107552400 - core::fmt::write::h05b8a9712099e9e5
  37:        0x1075530dc - <core::fmt::Formatter as core::fmt::Write>::write_fmt::h9a11944cd06b3d42
  38:        0x103fca38c - <&T as core::fmt::Debug>::fmt::hab8caed5401f622b
  39:        0x10754f3a4 - core::fmt::builders::DebugTuple::field::hded5bed1a041d37c
  40:        0x103fcc060 - <&T as core::fmt::Debug>::fmt::hfc29699970eaaae2
  41:        0x10754f54c - core::fmt::builders::DebugInner::entry::h09f4589557fabc12
  42:        0x10754f624 - core::fmt::builders::DebugList::entry::hc0c7a1b71658e789
  43:        0x103fcb508 - <&T as core::fmt::Debug>::fmt::he19086ac17751cf5
  44:        0x10754f098 - core::fmt::builders::DebugStruct::field::h5db80813e9a9e355
  45:        0x104044184 - <rustc_middle::ty::generics::GenericPredicates as core::fmt::Debug>::fmt::h5372e3d76d9a0b2b
  46:        0x107552400 - core::fmt::write::h05b8a9712099e9e5
  47:        0x1074fa844 - <std::panicking::begin_panic_handler::PanicPayload as core::panic::BoxMeUp>::get::hf279008f0b9b3ed6
  48:        0x1074faae0 - std::panicking::rust_panic_with_hook::h70f0bcacc0251305
  49:        0x1074fa6b8 - std::panicking::begin_panic_handler::{{closure}}::h0a11da6e0d868b2b
  50:        0x1074f70e4 - std::sys_common::backtrace::__rust_end_short_backtrace::h0e50d651e2e8352a
  51:        0x1074fa620 - _rust_begin_unwind
  52:        0x1075785a4 - std::panicking::begin_panic_fmt::h27d289fcb95248ef
  53:        0x102faa728 - rustc_query_system::query::plumbing::incremental_verify_ich::h081990c57d7d94df
  54:        0x102fc46d4 - rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory::h24cd89b9507887d5
  55:        0x103207bf4 - rustc_data_structures::stack::ensure_sufficient_stack::h981b3c6b557f243f
  56:        0x102f2ac38 - rustc_query_system::query::plumbing::get_query_impl::hb11df1f05dfb76d7
  57:        0x103114598 - <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::explicit_predicates_of::hf6924c2f9352ea7e
  58:        0x102d7aa28 - rustc_typeck::collect::predicates_defined_on::hc0531c0faa0a74ba
  59:        0x1031de130 - rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps::h9ab49bb9d6457e24
  60:        0x102fc46b0 - rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory::h24cd89b9507887d5
  61:        0x103207bf4 - rustc_data_structures::stack::ensure_sufficient_stack::h981b3c6b557f243f
  62:        0x102f2ac38 - rustc_query_system::query::plumbing::get_query_impl::hb11df1f05dfb76d7
  63:        0x1031143f8 - <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::predicates_defined_on::hb165e1c73c1eae33
  64:        0x102d7ae14 - rustc_typeck::collect::predicates_of::h3f8101518405123b
  65:        0x1031de130 - rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps::h9ab49bb9d6457e24
  66:        0x102fc46b0 - rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory::h24cd89b9507887d5
  67:        0x103207bf4 - rustc_data_structures::stack::ensure_sufficient_stack::h981b3c6b557f243f
  68:        0x102f2ac38 - rustc_query_system::query::plumbing::get_query_impl::hb11df1f05dfb76d7
  69:        0x103113070 - <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::predicates_of::h064a2c2ca9452515
  70:        0x10293e734 - rustc_ty_utils::ty::param_env::hb6eeb7e7c677dea4
  71:        0x1031d0670 - rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps::h042062de78523ca3
  72:        0x103164c04 - rustc_query_system::dep_graph::graph::DepGraph<K>::with_task_impl::hacbe5f5272934978
  73:        0x1031f4758 - rustc_data_structures::stack::ensure_sufficient_stack::h1df559daf6453029
  74:        0x102f7aed0 - rustc_query_system::query::plumbing::force_query_with_job::h61f5e9e8714f6c08
  75:        0x102f57258 - rustc_query_system::query::plumbing::force_query_impl::h7a7cd43b5a0a77c5
  76:        0x10305a0cc - rustc_query_impl::query_callbacks::param_env::force_from_dep_node::hc390c9c3fab64c5d
  77:        0x103171ff0 - rustc_query_system::dep_graph::graph::DepGraph<K>::try_mark_previous_green::hff03ccf2328beb39
  78:        0x103171fdc - rustc_query_system::dep_graph::graph::DepGraph<K>::try_mark_previous_green::hff03ccf2328beb39
  79:        0x103171fdc - rustc_query_system::dep_graph::graph::DepGraph<K>::try_mark_previous_green::hff03ccf2328beb39
  80:        0x103171fdc - rustc_query_system::dep_graph::graph::DepGraph<K>::try_mark_previous_green::hff03ccf2328beb39
  81:        0x103171e74 - rustc_query_system::dep_graph::graph::DepGraph<K>::try_mark_green_and_read::h74cbceb96d078716
  82:        0x102f44698 - rustc_query_system::query::plumbing::ensure_must_run::h01abf8e313dcc6ab
  83:        0x103116750 - <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::mir_borrowck::h47b2168b23282e74
  84:        0x100cba75c - rustc_middle::ty::<impl rustc_middle::ty::context::TyCtxt>::par_body_owners::h11d601215cfdd011
  85:        0x100ca6950 - rustc_interface::passes::analysis::hb9c4401f82ff9f3a
  86:        0x1031d6f54 - rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps::h992778ae60ca3cee
  87:        0x10315e968 - rustc_query_system::dep_graph::graph::DepGraph<K>::with_task_impl::h8520a5b2ebd4eaa3
  88:        0x103214a38 - rustc_data_structures::stack::ensure_sufficient_stack::hf95bdd7dad659377
  89:        0x102f8fe40 - rustc_query_system::query::plumbing::force_query_with_job::hdc420bb164faa63c
  90:        0x102efffc4 - rustc_query_system::query::plumbing::get_query_impl::h4ff04ac3bd9178f1
  91:        0x103112f1c - <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::analysis::hc202e75378cba7a2
  92:        0x100bf3304 - rustc_interface::passes::QueryContext::enter::h17d1690d8b523d81
  93:        0x100bb0ea8 - rustc_interface::queries::<impl rustc_interface::interface::Compiler>::enter::h4150fe8ab9fd33d3
  94:        0x100b9f724 - rustc_span::with_source_map::h35812be89751f825
  95:        0x100bb1d98 - rustc_interface::interface::create_compiler_and_run::h812f164e1b5516df
  96:        0x100ba5980 - scoped_tls::ScopedKey<T>::set::h0f359b94c4699e7d
  97:        0x100bb2b84 - std::sys_common::backtrace::__rust_begin_short_backtrace::hdc801d619b0f2f18
  98:        0x100bcc388 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h806a923c7f067c24
  99:        0x1075066ec - std::sys::unix::thread::Thread::new::thread_start::h5b09a471d4a0a1b5
 100:        0x19942a06c - _pthread_kill

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.52.0 (88f19c6da 2021-05-03) running on aarch64-apple-darwin

note: compiler flags: -C embed-bitcode=no -C debuginfo=2 -C incremental

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
thread panicked while processing panic. aborting.
error: could not compile `sky-bench`

Caused by:
  process didn't exit successfully: `rustc --crate-name sky_bench --edition=2018 sky-bench/src/main.rs --error-format=json --json=diagnostic-rendered-ansi --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 --test -C metadata=7117c4f6a5247e4a -C extra-filename=-7117c4f6a5247e4a --out-dir [secret:cargotarget]/debug/deps -C incremental=[secret:cargotarget]/debug/incremental -L dependency=[secret:cargotarget]/debug/deps --extern clap=[secret:cargotarget]/debug/deps/libclap-cefe834593a8a5ca.rlib --extern devtimer=[secret:cargotarget]/debug/deps/libdevtimer-1244bbe38dff8712.rlib --extern libsky=[secret:cargotarget]/debug/deps/liblibsky-34ea91518ed018b5.rlib --extern rand=[secret:cargotarget]/debug/deps/librand-90c3cd98bbead71e.rlib --extern serde=[secret:cargotarget]/debug/deps/libserde-ade142c231b30f6f.rlib --extern serde_json=[secret:cargotarget]/debug/deps/libserde_json-b30be567a22934fa.rlib` (signal: 5, SIGTRAP: trace/breakpoint trap)
warning: build failed, waiting for other jobs to finish...
thread 'rustc' panicked at 'found unstable fingerprints for item_children(libsky[d7cb]): [Export { ident: terrapipe#0, res: Def(Mod, DefId(26:3 ~ libsky[d7cb]::terrapipe)), span: /private/tmp/drone-XTLrTmOnXHGBgymS/drone/src/libsky/src/lib.rs:31:1: 31:19 (#0), vis: Public }, Export { ident: util#0, res: Def(Mod, DefId(26:24 ~ libsky[d7cb]::util)), span: /private/tmp/drone-XTLrTmOnXHGBgymS/drone/src/libsky/src/lib.rs:32:1: 32:14 (#0), vis: Public }, Export { ident: TResult#0, res: Def(TyAlias, DefId(26:61 ~ libsky[d7cb]::TResult)), span: /private/tmp/drone-XTLrTmOnXHGBgymS/drone/src/libsky/src/lib.rs:35:1: 35:49 (#0), vis: Public }, Export { ident: BUF_CAP#0, res: Def(Const, DefId(26:63 ~ libsky[d7cb]::BUF_CAP)), span: /private/tmp/drone-XTLrTmOnXHGBgymS/drone/src/libsky/src/lib.rs:37:1: 37:37 (#0), vis: Public }, Export { ident: Error#0, res: Def(Trait, DefId(1:2324 ~ std[abff]::error::Error)), span: /private/tmp/drone-XTLrTmOnXHGBgymS/drone/src/libsky/src/lib.rs:33:5: 33:22 (#0), vis: Restricted(DefId(26:0 ~ libsky[d7cb])) }]', /rustc/88f19c6dab716c6281af7602e30f413e809c5974/compiler/rustc_query_system/src/query/plumbing.rs:593:5
stack backtrace:
   0:        0x109b5ac54 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h7cf4b69c45a99b1c
   1:        0x109bb6400 - core::fmt::write::h05b8a9712099e9e5
   2:        0x109b4f850 - std::io::Write::write_fmt::hfe8917b7b12925fd
   3:        0x109b5e3d0 - std::panicking::default_hook::{{closure}}::hda4b0dbd83d08efe
   4:        0x109b5def8 - std::panicking::default_hook::h713df0508ca1a369
   5:        0x103201e9c - rustc_driver::report_ice::h621c43280551f1ad
   6:        0x109b5eaf4 - std::panicking::rust_panic_with_hook::h70f0bcacc0251305
   7:        0x109b5e6b8 - std::panicking::begin_panic_handler::{{closure}}::h0a11da6e0d868b2b
   8:        0x109b5b0e4 - std::sys_common::backtrace::__rust_end_short_backtrace::h0e50d651e2e8352a
   9:        0x109b5e620 - _rust_begin_unwind
  10:        0x109bdc5a4 - std::panicking::begin_panic_fmt::h27d289fcb95248ef
  11:        0x105613318 - rustc_query_system::query::plumbing::incremental_verify_ich::h3b795f48fda514bf
  12:        0x1056364b4 - rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory::hd489dd83686ae20b
  13:        0x10586bd18 - rustc_data_structures::stack::ensure_sufficient_stack::h984bf9678df70bcb
  14:        0x10557b8ec - rustc_query_system::query::plumbing::get_query_impl::h8acc6a801e6ac255
  15:        0x10577ea60 - <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::item_children::hec4321eb38d39236
  16:        0x105fde768 - core::ops::function::FnOnce::call_once::h43db234dfbd2fd0c
  17:        0x105834a54 - rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps::h04742382a4263eef
  18:        0x10562ea70 - rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory::h7682e4beeea09fbc
  19:        0x10585f2f0 - rustc_data_structures::stack::ensure_sufficient_stack::h4cbe0bdcb561e992
  20:        0x105564fe0 - rustc_query_system::query::plumbing::get_query_impl::h5216c9b499f69736
  21:        0x10577f034 - <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::visible_parent_map::h3768502f3a14842a
  22:        0x10664b694 - rustc_middle::ty::print::pretty::PrettyPrinter::try_print_visible_def_path_recur::h824307ccd6064e73
  23:        0x106654de0 - <rustc_middle::ty::print::pretty::FmtPrinter<F> as rustc_middle::ty::print::Printer>::print_def_path::hd4ae041fed231929
  24:        0x1066642bc - rustc_middle::ty::print::Printer::default_print_def_path::h99e342df73393194
  25:        0x106654f44 - <rustc_middle::ty::print::pretty::FmtPrinter<F> as rustc_middle::ty::print::Printer>::print_def_path::hd4ae041fed231929
  26:        0x1066656b0 - <rustc_middle::ty::print::pretty::TraitRefPrintOnlyTraitPath as core::fmt::Display>::fmt::h305888562eeeeb09
  27:        0x109bb6400 - core::fmt::write::h05b8a9712099e9e5
  28:        0x106533ed4 - rustc_middle::ty::print::pretty::<impl rustc_middle::ty::print::Print<P> for rustc_middle::ty::sty::TraitRef>::print::h20133c6e2b9ec55e
  29:        0x106534ccc - rustc_middle::ty::print::pretty::<impl core::fmt::Display for rustc_middle::ty::sty::TraitRef>::fmt::h7333ed8f621fd019
  30:        0x10652b7b8 - rustc_middle::ty::structural_impls::<impl core::fmt::Debug for rustc_middle::ty::sty::TraitRef>::fmt::ha19f762841df7f58
  31:        0x109bb6400 - core::fmt::write::h05b8a9712099e9e5
  32:        0x109bb70dc - <core::fmt::Formatter as core::fmt::Write>::write_fmt::h9a11944cd06b3d42
  33:        0x10663cca0 - rustc_middle::ty::structural_impls::<impl core::fmt::Debug for rustc_middle::ty::PredicateKind>::fmt::h567256ec1ce234c5
  34:        0x109bb33a4 - core::fmt::builders::DebugTuple::field::hded5bed1a041d37c
  35:        0x106538c7c - <rustc_middle::ty::sty::Binder<T> as core::fmt::Debug>::fmt::hcc79fe5855ca8a5b
  36:        0x109bb6400 - core::fmt::write::h05b8a9712099e9e5
  37:        0x109bb70dc - <core::fmt::Formatter as core::fmt::Write>::write_fmt::h9a11944cd06b3d42
  38:        0x10662e38c - <&T as core::fmt::Debug>::fmt::hab8caed5401f622b
  39:        0x109bb33a4 - core::fmt::builders::DebugTuple::field::hded5bed1a041d37c
  40:        0x106630060 - <&T as core::fmt::Debug>::fmt::hfc29699970eaaae2
  41:        0x109bb354c - core::fmt::builders::DebugInner::entry::h09f4589557fabc12
  42:        0x109bb3624 - core::fmt::builders::DebugList::entry::hc0c7a1b71658e789
  43:        0x10662f508 - <&T as core::fmt::Debug>::fmt::he19086ac17751cf5
  44:        0x109bb3098 - core::fmt::builders::DebugStruct::field::h5db80813e9a9e355
  45:        0x1066a8184 - <rustc_middle::ty::generics::GenericPredicates as core::fmt::Debug>::fmt::h5372e3d76d9a0b2b
  46:        0x109bb6400 - core::fmt::write::h05b8a9712099e9e5
  47:        0x109b5e844 - <std::panicking::begin_panic_handler::PanicPayload as core::panic::BoxMeUp>::get::hf279008f0b9b3ed6
  48:        0x109b5eae0 - std::panicking::rust_panic_with_hook::h70f0bcacc0251305
  49:        0x109b5e6b8 - std::panicking::begin_panic_handler::{{closure}}::h0a11da6e0d868b2b
  50:        0x109b5b0e4 - std::sys_common::backtrace::__rust_end_short_backtrace::h0e50d651e2e8352a
  51:        0x109b5e620 - _rust_begin_unwind
  52:        0x109bdc5a4 - std::panicking::begin_panic_fmt::h27d289fcb95248ef
  53:        0x10560e728 - rustc_query_system::query::plumbing::incremental_verify_ich::h081990c57d7d94df
  54:        0x1056286d4 - rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory::h24cd89b9507887d5
  55:        0x10586bbf4 - rustc_data_structures::stack::ensure_sufficient_stack::h981b3c6b557f243f
  56:        0x10558ec38 - rustc_query_system::query::plumbing::get_query_impl::hb11df1f05dfb76d7
  57:        0x105778598 - <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::explicit_predicates_of::hf6924c2f9352ea7e
  58:        0x1053dea28 - rustc_typeck::collect::predicates_defined_on::hc0531c0faa0a74ba
  59:        0x105842130 - rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps::h9ab49bb9d6457e24
  60:        0x1056286b0 - rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory::h24cd89b9507887d5
  61:        0x10586bbf4 - rustc_data_structures::stack::ensure_sufficient_stack::h981b3c6b557f243f
  62:        0x10558ec38 - rustc_query_system::query::plumbing::get_query_impl::hb11df1f05dfb76d7
  63:        0x1057783f8 - <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::predicates_defined_on::hb165e1c73c1eae33
  64:        0x1053dee14 - rustc_typeck::collect::predicates_of::h3f8101518405123b
  65:        0x105842130 - rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps::h9ab49bb9d6457e24
  66:        0x1056286b0 - rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory::h24cd89b9507887d5
  67:        0x10586bbf4 - rustc_data_structures::stack::ensure_sufficient_stack::h981b3c6b557f243f
  68:        0x10558ec38 - rustc_query_system::query::plumbing::get_query_impl::hb11df1f05dfb76d7
  69:        0x105777070 - <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::predicates_of::h064a2c2ca9452515
  70:        0x104fa2734 - rustc_ty_utils::ty::param_env::hb6eeb7e7c677dea4
  71:        0x105834670 - rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps::h042062de78523ca3
  72:        0x1057c8c04 - rustc_query_system::dep_graph::graph::DepGraph<K>::with_task_impl::hacbe5f5272934978
  73:        0x105858758 - rustc_data_structures::stack::ensure_sufficient_stack::h1df559daf6453029
  74:        0x1055deed0 - rustc_query_system::query::plumbing::force_query_with_job::h61f5e9e8714f6c08
  75:        0x1055bb258 - rustc_query_system::query::plumbing::force_query_impl::h7a7cd43b5a0a77c5
  76:        0x1056be0cc - rustc_query_impl::query_callbacks::param_env::force_from_dep_node::hc390c9c3fab64c5d
  77:        0x1057d5ff0 - rustc_query_system::dep_graph::graph::DepGraph<K>::try_mark_previous_green::hff03ccf2328beb39
  78:        0x1057d5fdc - rustc_query_system::dep_graph::graph::DepGraph<K>::try_mark_previous_green::hff03ccf2328beb39
  79:        0x1057d5fdc - rustc_query_system::dep_graph::graph::DepGraph<K>::try_mark_previous_green::hff03ccf2328beb39
  80:        0x1057d5fdc - rustc_query_system::dep_graph::graph::DepGraph<K>::try_mark_previous_green::hff03ccf2328beb39
  81:        0x1057d5fdc - rustc_query_system::dep_graph::graph::DepGraph<K>::try_mark_previous_green::hff03ccf2328beb39
  82:        0x1057d5fdc - rustc_query_system::dep_graph::graph::DepGraph<K>::try_mark_previous_green::hff03ccf2328beb39
  83:        0x1057d5e74 - rustc_query_system::dep_graph::graph::DepGraph<K>::try_mark_green_and_read::h74cbceb96d078716
  84:        0x1055a8698 - rustc_query_system::query::plumbing::ensure_must_run::h01abf8e313dcc6ab
  85:        0x105779e60 - <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::check_mod_item_types::hd4ddf75c5a421224
  86:        0x1052b437c - rustc_session::utils::<impl rustc_session::session::Session>::time::h35db09ce2c38ad14
  87:        0x1052d6630 - rustc_typeck::check_crate::h0a22b7ed6869eee9
  88:        0x10330a8ec - rustc_interface::passes::analysis::hb9c4401f82ff9f3a
  89:        0x10583af54 - rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps::h992778ae60ca3cee
  90:        0x1057c2968 - rustc_query_system::dep_graph::graph::DepGraph<K>::with_task_impl::h8520a5b2ebd4eaa3
  91:        0x105878a38 - rustc_data_structures::stack::ensure_sufficient_stack::hf95bdd7dad659377
  92:        0x1055f3e40 - rustc_query_system::query::plumbing::force_query_with_job::hdc420bb164faa63c
  93:        0x105563fc4 - rustc_query_system::query::plumbing::get_query_impl::h4ff04ac3bd9178f1
  94:        0x105776f1c - <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::analysis::hc202e75378cba7a2
  95:        0x103257304 - rustc_interface::passes::QueryContext::enter::h17d1690d8b523d81
  96:        0x103214ea8 - rustc_interface::queries::<impl rustc_interface::interface::Compiler>::enter::h4150fe8ab9fd33d3
  97:        0x103203724 - rustc_span::with_source_map::h35812be89751f825
  98:        0x103215d98 - rustc_interface::interface::create_compiler_and_run::h812f164e1b5516df
  99:        0x103209980 - scoped_tls::ScopedKey<T>::set::h0f359b94c4699e7d
 100:        0x103216b84 - std::sys_common::backtrace::__rust_begin_short_backtrace::hdc801d619b0f2f18
 101:        0x103230388 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h806a923c7f067c24
 102:        0x109b6a6ec - std::sys::unix::thread::Thread::new::thread_start::h5b09a471d4a0a1b5
 103:        0x19942a06c - _pthread_kill

error: internal compiler error: unexpected panic

Related issues: #85003, #83291, #84970 (kept open as this is aarch64-apple-darwin)

[ohsayan]

@rustbot label: +regression-from-stable-to-stable

[ohsayan]

Can confirm this doesn't happen on 1.51.0 stable

[hellow554]

I can't reproduce it on linux neither with cargo build nor with cargo test. Is it an platform specific problem?

[ohsayan]

I can't reproduce it on linux neither with cargo build nor with cargo test. Is it an platform specific problem?

Neither can I repro on our Linux/macOS/Windows (Intel x86) machines

[ohsayan]

Our CI continues to reproduce this exact error in subsequent runs:

• https://build.skytable.io/skytable/skytable/101/1/3
• https://build.skytable.io/skytable/skytable/103/1/3
• https://build.skytable.io/skytable/skytable/104/1/3

Oddly enough, it passes with: https://build.skytable.io/skytable/skytable/102/1/3 and fails right on the next ones. Since deps were upgraded, there were some changes to the target folder which I'm going to assume caused the build to pass. I'm making this assumption from the build log which shows 'fresh' for run 102, but doesn't on subsequent runs

[ohsayan]

We continue to reproduce this error; I'm creating a new storage pool for the CI runs to see if it helps; all relevant failure logs can be found here: https://build.skytable.io/skytable/skytable

Edit: By new storage pool I mean we're technically removing all the earlier cache
Update: After clearing the cache the build passes; seems like this is an issue with the target directory. I've kept the old cache just in case anyone would like to use it for debugging
Update: The build crashed right now; so it's clearly pre-existing compiled deps/resources that is causing problems (incremental compilation cache)

[Mark-Simulacrum]

Please see #84970 - you should likely be able to fix with CARGO_INCREMENTAL=0 for the time being.