error: found unstable fingerprints for evaluate_obligation

[aurexav]

Code

Meta

rustc --version --verbose:

xavier on xavier.zone in common at  rococo-v1 <!?> via 𝓡 v1.52.0-nightly takes 1m6s
Fri Mar 19 13:14:22 2021 - 50%
✗ rustc --version --verbose
rustc 1.52.0-nightly (1705a7d64 2021-03-18)
binary: rustc
commit-hash: 1705a7d64b833d1c4b69958b0627bd054e6d764b
commit-date: 2021-03-18
host: x86_64-unknown-linux-gnu
release: 1.52.0-nightly
LLVM version: 12.0.0

Error output

xavier on xavier.zone in common at  rococo-v1 <!?> via 𝓡 v1.52.0-nightly takes 13ms
Fri Mar 19 13:12:30 2021 - 45%
λ cargo b --release                     
   Compiling darwinia-ethereum-relay v2.0.0 (/home/xavier/Documents/darwinia-network/common/frame/bridge/ethereum/relay)
   Compiling darwinia-ethereum-linear-relay v2.0.0 (/home/xavier/Documents/darwinia-network/common/frame/bridge/ethereum/linear-relay)
   Compiling pangolin-runtime v2.0.0 (/home/xavier/Documents/darwinia-network/common/bin/node/runtime/pangolin)
thread 'rustc' panicked at 'found unstable fingerprints for evaluate_obligation(1746fd5d1304336-c1d2d55e63e10b17)', /rustc/1705a7d64b833d1c4b69958b0627bd054e6d764b/compiler/rustc_query_system/src/query/plumbing.rs:593:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.52.0-nightly (1705a7d64 2021-03-18) running on x86_64-unknown-linux-gnu

note: compiler flags: -C opt-level=3 -C embed-bitcode=no -C incremental --crate-type lib

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
#0 [evaluate_obligation] evaluating trait selection obligation `impls::DealWithFees: sp_api_hidden_includes_construct_runtime::hidden_include::traits::OnUnbalanced<darwinia_balances::NegativeImbalance<Runtime, darwinia_balances::Instance0>>`
#1 [normalize_projection_ty] normalizing `Canonical { max_universe: U0, variables: [], value: ParamEnvAnd { param_env: ParamEnv { caller_bounds: [], reveal: All }, value: ProjectionTy { substs: [pallet_transaction_payment::CurrencyAdapter<darwinia_balances::Module<Runtime, darwinia_balances::Instance0>, impls::DealWithFees>, Runtime], item_def_id: DefId(394:165 ~ pallet_transaction_payment[3044]::payment::OnChargeTransaction::Balance) } } }`
end of query stack
error: could not compile `pangolin-runtime`

To learn more, run the command again with --verbose.

Backtrace

xavier on xavier.zone in common at  rococo-v1 <!?> via 𝓡 v1.52.0-nightly takes 62ms
Fri Mar 19 13:14:24 2021 - 46%
λ RUST_BACKTRACE=1 cargo b --release 
   Compiling librocksdb-sys v6.11.4
   Compiling sc-network v0.9.0 (https://github.com/darwinia-network/substrate.git?tag=rococo-v1.1#645299e8)
   Compiling pangolin-runtime v2.0.0 (/home/xavier/Documents/darwinia-network/common/bin/node/runtime/pangolin)
thread 'rustc' panicked at 'found unstable fingerprints for evaluate_obligation(1746fd5d1304336-c1d2d55e63e10b17)', /rustc/1705a7d64b833d1c4b69958b0627bd054e6d764b/compiler/rustc_query_system/src/query/plumbing.rs:593:5
stack backtrace:
   0: rust_begin_unwind
             at /rustc/1705a7d64b833d1c4b69958b0627bd054e6d764b/library/std/src/panicking.rs:493:5
   1: std::panicking::begin_panic_fmt
             at /rustc/1705a7d64b833d1c4b69958b0627bd054e6d764b/library/std/src/panicking.rs:435:5
   2: rustc_query_system::query::plumbing::incremental_verify_ich
   3: rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory
   4: rustc_data_structures::stack::ensure_sufficient_stack
   5: rustc_query_system::query::plumbing::get_query_impl
   6: <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::evaluate_obligation
   7: <rustc_infer::infer::InferCtxt as rustc_trait_selection::traits::query::evaluate_obligation::InferCtxtExt>::evaluate_obligation
   8: <rustc_infer::infer::InferCtxt as rustc_trait_selection::traits::query::evaluate_obligation::InferCtxtExt>::evaluate_obligation_no_overflow
   9: rustc_trait_selection::traits::fulfill::FulfillProcessor::process_trait_obligation
  10: rustc_trait_selection::traits::fulfill::FulfillProcessor::progress_changed_obligations
  11: rustc_data_structures::obligation_forest::ObligationForest<O>::process_obligations
  12: <rustc_trait_selection::traits::fulfill::FulfillmentContext as rustc_infer::traits::engine::TraitEngine>::select_where_possible
  13: <rustc_infer::infer::InferCtxtBuilder as rustc_trait_selection::infer::InferCtxtBuilderExt>::enter_canonical_trait_query
  14: rustc_traits::normalize_projection_ty::normalize_projection_ty
  15: rustc_query_impl::<impl rustc_query_system::query::config::QueryAccessors<rustc_query_impl::plumbing::QueryCtxt> for rustc_query_impl::queries::normalize_projection_ty>::compute
  16: rustc_query_system::query::config::QueryVtable<CTX,K,V>::compute
  17: rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps
  18: rustc_query_system::dep_graph::graph::DepGraph<K>::with_ignore
  19: rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory
  20: rustc_data_structures::stack::ensure_sufficient_stack
  21: rustc_query_system::query::plumbing::get_query_impl
  22: <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::normalize_projection_ty
  23: <rustc_trait_selection::traits::query::normalize::QueryNormalizer as rustc_middle::ty::fold::TypeFolder>::fold_ty
  24: rustc_middle::ty::fold::TypeFoldable::fold_with
  25: rustc_middle::ty::structural_impls::<impl rustc_middle::ty::fold::TypeFoldable for &rustc_middle::ty::TyS>::super_fold_with
  26: <rustc_trait_selection::traits::query::normalize::QueryNormalizer as rustc_middle::ty::fold::TypeFolder>::fold_ty
  27: rustc_middle::ty::fold::TypeFoldable::fold_with
  28: rustc_middle::ty::structural_impls::<impl rustc_middle::ty::fold::TypeFoldable for &rustc_middle::ty::TyS>::super_fold_with
  29: <rustc_trait_selection::traits::query::normalize::QueryNormalizer as rustc_middle::ty::fold::TypeFolder>::fold_ty
  30: <rustc_infer::infer::at::At as rustc_trait_selection::traits::query::normalize::AtExt>::normalize
  31: rustc_infer::infer::InferCtxtBuilder::enter
  32: rustc_traits::normalize_erasing_regions::normalize_generic_arg_after_erasing_regions
  33: rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps
  34: rustc_query_system::dep_graph::graph::DepGraph<K>::with_ignore
  35: rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory
  36: rustc_data_structures::stack::ensure_sufficient_stack
  37: rustc_query_system::query::plumbing::get_query_impl
  38: <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::normalize_generic_arg_after_erasing_regions
  39: <rustc_middle::ty::normalize_erasing_regions::NormalizeAfterErasingRegionsFolder as rustc_middle::ty::fold::TypeFolder>::fold_ty
  40: rustc_middle::ty::instance::Instance::subst_mir_and_normalize_erasing_regions
  41: <rustc_mir::monomorphize::collector::MirNeighborCollector as rustc_middle::mir::visit::Visitor>::visit_terminator
  42: rustc_mir::monomorphize::collector::collect_neighbours
  43: rustc_mir::monomorphize::collector::collect_items_rec
  44: rustc_mir::monomorphize::collector::collect_items_rec
  45: rustc_mir::monomorphize::collector::collect_items_rec
  46: rustc_mir::monomorphize::collector::collect_items_rec
  47: rustc_mir::monomorphize::collector::collect_items_rec
  48: rustc_mir::monomorphize::collector::collect_items_rec
  49: rustc_mir::monomorphize::collector::collect_items_rec
  50: rustc_mir::monomorphize::collector::collect_items_rec
  51: rustc_mir::monomorphize::collector::collect_items_rec
  52: rustc_mir::monomorphize::collector::collect_items_rec
  53: rustc_mir::monomorphize::collector::collect_items_rec
  54: rustc_mir::monomorphize::collector::collect_items_rec
  55: rustc_mir::monomorphize::collector::collect_crate_mono_items
  56: rustc_mir::monomorphize::partitioning::collect_and_partition_mono_items
  57: rustc_query_impl::<impl rustc_query_system::query::config::QueryAccessors<rustc_query_impl::plumbing::QueryCtxt> for rustc_query_impl::queries::collect_and_partition_mono_items>::compute
  58: rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps
  59: rustc_query_system::dep_graph::graph::DepGraph<K>::with_task_impl
  60: rustc_data_structures::stack::ensure_sufficient_stack
  61: rustc_query_system::query::plumbing::force_query_with_job
  62: rustc_query_system::query::plumbing::get_query_impl
  63: <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::collect_and_partition_mono_items
  64: <rustc_codegen_llvm::LlvmCodegenBackend as rustc_codegen_ssa::traits::backend::CodegenBackend>::codegen_crate
  65: rustc_interface::passes::QueryContext::enter
  66: rustc_interface::queries::Queries::ongoing_codegen
  67: rustc_interface::queries::<impl rustc_interface::interface::Compiler>::enter
  68: rustc_span::with_source_map
  69: rustc_interface::interface::create_compiler_and_run
  70: scoped_tls::ScopedKey<T>::set
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.52.0-nightly (1705a7d64 2021-03-18) running on x86_64-unknown-linux-gnu

note: compiler flags: -C opt-level=3 -C embed-bitcode=no -C incremental --crate-type lib

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
#0 [evaluate_obligation] evaluating trait selection obligation `impls::DealWithFees: sp_api_hidden_includes_construct_runtime::hidden_include::traits::OnUnbalanced<darwinia_balances::NegativeImbalance<Runtime, darwinia_balances::Instance0>>`
#1 [normalize_projection_ty] normalizing `Canonical { max_universe: U0, variables: [], value: ParamEnvAnd { param_env: ParamEnv { caller_bounds: [], reveal: All }, value: ProjectionTy { substs: [pallet_transaction_payment::CurrencyAdapter<darwinia_balances::Module<Runtime, darwinia_balances::Instance0>, impls::DealWithFees>, Runtime], item_def_id: DefId(394:165 ~ pallet_transaction_payment[3044]::payment::OnChargeTransaction::Balance) } } }`
#2 [normalize_generic_arg_after_erasing_regions] normalizing `for<'r> fn(&'r mut _::_parity_scale_codec::depth_limit::DepthTrackingInput<&[u8]>) -> std::result::Result<<<<Runtime as pallet_transaction_payment::Config>::OnChargeTransaction as pallet_transaction_payment::OnChargeTransaction<Runtime>>::Balance as _::_parity_scale_codec::HasCompact>::Type, _::_parity_scale_codec::Error> {<<<<Runtime as pallet_transaction_payment::Config>::OnChargeTransaction as pallet_transaction_payment::OnChargeTransaction<Runtime>>::Balance as _::_parity_scale_codec::HasCompact>::Type as _::_parity_scale_codec::Decode>::decode::<_::_parity_scale_codec::depth_limit::DepthTrackingInput<&[u8]>>}`
#3 [collect_and_partition_mono_items] collect_and_partition_mono_items
end of query stack
error: could not compile `pangolin-runtime`

To learn more, run the command again with --verbose.

[jyn514]

Could you post the code this crashes on? It doesn't have to be small, but this is hard to fix without a way to replicate the crash.

[aurexav]

Could you post the code this crashes on? It doesn't have to be small, but this is hard to fix without a way to replicate the crash.

I'm not sure what causes this. Can't give a minimal env to reproduce this, sorry.

I'm compiling a substrate runtime. Really big project.

darwinia-network/darwinia-common@81fb0c5

The first time it can be compile success. And I do some modifies to code, then it panics.

And cargo clean is a workaround.

[Aaron1011]

This will be fixed by #83220

[aurexav]

Fixed in 3-23

But in 3-24 panic again

[teor2345]

I'm also seeing a similar error intermittently, when compiling https://github.com/ZcashFoundation/zebra using clippy --all-targets --all-features. See #83381 for more details.

It seems like nightly-2021-03-13-x86_64-unknown-linux-gnu is unaffected (or the issue is very rare).

2021-03-14 to 2021-03-21 are affected occasionally. (Maybe 1 in 10 runs.)

I haven't tested anything more recent, but I'll do that over the next few days.