unstable fingerprints for optimized_mir

[KalitaAlexey]

Code

git clone -b v2.0.0 --depth 1 https://github.com/substrate-developer-hub/substrate-node-template

Steps

rustup target add wasm32-unknown-unknown --toolchain nightly-2020-10-05
export RUSTFLAGS="-Z incremental-info=yes -Z incremental-verify-ich=yes"
cargo build
Add a comment (modify a file with no actual impact)
cargo build

Meta

rustc --version --verbose:

rustc 1.49.0-nightly (beb5ae474 2020-10-04)
binary: rustc
commit-hash: beb5ae474d2835962ebdf7416bd1c9ad864fe101
commit-date: 2020-10-04
host: x86_64-apple-darwin
release: 1.49.0-nightly
LLVM version: 11.0

Error output

thread 'rustc' panicked at 'found unstable fingerprints for optimized_mir(node_template_runtime[3b47]::_::{impl#0}::decode)', /rustc/beb5ae474d2835962ebdf7416bd1c9ad864fe101/compiler/rustc_query_system/src/query/plumbing.rs:555:5

Backtrace

stack backtrace:
   0: _rust_begin_unwind
   1: std::panicking::begin_panic_fmt
   2: rustc_query_system::query::plumbing::incremental_verify_ich
   3: rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory
   4: rustc_data_structures::stack::ensure_sufficient_stack
   5: rustc_query_system::query::plumbing::get_query_impl
   6: rustc_metadata::rmeta::encoder::EncodeContext::encode_optimized_mir
   7: <rustc_metadata::rmeta::encoder::EncodeContext as rustc_hir::intravisit::Visitor>::visit_item
   8: rustc_hir::hir::Crate::visit_all_item_likes
   9: rustc_metadata::rmeta::encoder::EncodeContext::encode_crate_root
  10: rustc_metadata::rmeta::encoder::encode_metadata_impl
  11: rustc_data_structures::sync::join
  12: rustc_metadata::rmeta::decoder::cstore_impl::<impl rustc_middle::middle::cstore::CrateStore for rustc_metadata::creader::CStore>::encode_metadata
  13: rustc_middle::ty::context::TyCtxt::encode_metadata
  14: rustc_interface::passes::QueryContext::enter
  15: rustc_interface::queries::Queries::ongoing_codegen
  16: rustc_interface::queries::<impl rustc_interface::interface::Compiler>::enter
  17: rustc_span::with_source_map
  18: rustc_interface::interface::create_compiler_and_run
  19: scoped_tls::ScopedKey<T>::set

[camelid]

Meta

rustc --version --verbose:

rustc 1.49.0-nightly (beb5ae474 2020-10-04)
binary: rustc
commit-hash: beb5ae474d2835962ebdf7416bd1c9ad864fe101
commit-date: 2020-10-04
host: x86_64-apple-darwin
release: 1.49.0-nightly
LLVM version: 11.0

@KalitaAlexey Can you please try with a recent nightly and see if it works? E.g. try with 2020-12-22.

[KalitaAlexey]

@camelid

rustc --version --verbose
rustc 1.50.0-nightly (7f9c43cf9 2020-12-23)
binary: rustc
commit-hash: 7f9c43cf98cfe1c369045399929cb098155b8374
commit-date: 2020-12-23
host: x86_64-pc-windows-msvc
release: 1.50.0-nightly

git clone https://github.com/paritytech/substrate.git
cd substrate/bin/node-template/node
export RUSTFLAGS="-Z incremental-info=yes -Z incremental-verify-ich=yes"
cargo build
// Make a change to src/lib.rs
cargo build

cargo build
   Compiling node-template v2.0.0 (D:\Projects\Rust\substrate\bin\node-template\node)
[incremental] session directory: 259 files hard-linked
[incremental] session directory: 0 files copied
thread 'rustc' panicked at 'found unstable fingerprints for evaluate_obligation(537eb1fcc2ac3041-cafb908c05fc9c89)', /rustc/7f9c43cf98cfe1c369045399929cb098155b8374\compiler\rustc_query_system\src\query\plumbing.rs:566:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: internal compiler error: unexpected panic
note: the compiler unexpectedly panicked. this is a bug.
note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md
note: rustc 1.50.0-nightly (7f9c43cf9 2020-12-23) running on x86_64-pc-windows-msvc
note: compiler flags: -Z incremental-info=yes -Z incremental-verify-ich=yes -C embed-bitcode=no -C debuginfo=2 -C incremental --crate-type lib
note: some of the compiler flags provided by cargo are hidden

query stack during panic:
#0 [evaluate_obligation] evaluating trait selection obligation `sc_service::Arc<dyn sp_state_machine::trie_backend_essence::Storage<sp_runtime::traits::BlakeTwo256>>: sp_state_machine::trie_backend_essence::TrieBackendStorage<sp_runtime::traits::BlakeTwo256>`
#1 [normalize_projection_ty] normalizing `Canonical { max_universe: U0, variables: [], value: ParamEnvAnd { param_env: ParamEnv { caller_bounds: [], reveal: All }, value: ProjectionTy { substs: [sc_client_db::storage_cache::SyncingCachingState<sc_client_db::RefTrackingState<sp_runtime::generic::Block<sp_runtime::generic::Header<u32, sp_runtime::traits::BlakeTwo256>, sp_runtime::OpaqueExtrinsic>>, sp_runtime::generic::Block<sp_runtime::generic::Header<u32, sp_runtime::traits::BlakeTwo256>, sp_runtime::OpaqueExtrinsic>>, sp_runtime::traits::BlakeTwo256], item_def_id: DefId(195:26 ~ sp_state_machine[4107]::backend::Backend::Transaction) } } }`
end of query stack
error: could not compile `node-template`

[Aaron1011]

This looks like a legitimate issue with evaluate_obligation. In the initial successful run, we get the following result (I added some extra logging locally):

DEBUG rustc_traits::evaluate_obligation evaluate_obligation(Canonical {
    max_universe: U0,
    variables: [
        CanonicalVarInfo {
            kind: Region(
                U0,
            ),
        },
    ],
    value: ParamEnvAnd {
        param_env: ParamEnv {
            caller_bounds: [],
            reveal: All,
        },
        value: TraitPredicate(<sc_service::Arc<dyn sp_state_machine::trie_backend_essence::Storage<sp_runtime::traits::BlakeTwo256>> as sp_state_machine::trie_backend_essence::TrieBackendStorage<sp_runtime::traits::BlakeTwo256>>),
    },
}) = Ok(EvaluatedToOk)

On the second (modified incremental) run, we get this result:

DEBUG rustc_traits::evaluate_obligation evaluate_obligation(Canonical {
    max_universe: U0,
    variables: [
        CanonicalVarInfo {
            kind: Region(
                U0,
            ),
        },
    ],
    value: ParamEnvAnd {
        param_env: ParamEnv {
            caller_bounds: [],
            reveal: All,
        },
        value: TraitPredicate(<sc_service::Arc<dyn sp_state_machine::trie_backend_essence::Storage<sp_runtime::traits::BlakeTwo256>> as sp_state_machine::trie_backend_essence::TrieBackendStorage<sp_runtime::traits::BlakeTwo256>>),
    },
}) = Ok(EvaluatedToOkModuloRegions)

The return value of evaluate_obligation on the same input differs between compilation sessions - we get EvaluatedToOk with an empty incremental cache, but we get EvaluatedToOkModuloRegions with a partially outdated cache.

[Aaron1011]

The issue is caused by the way that we handle regions in the evaluation cache. When we insert a result into the evaluation cache (either infcx.evaluation_cache or tcx.evaluation_cache, we use a 'freshened' version of the original TraitPredicate as the key. The 'freshened' TraitPredicate has all non-late-bound regions erased.

Unfortunately, this can lead to issues if we try to evaluate the following two predicates:

impl SomeTrait for SomeType<'static>

SomeType<'static> as SomeTrait
SomeType<'#_r> as SomeTrait

When we evaluate SomeType<'static> as SomeTrait, we'll get EvaluatedToOk, since the region parameter in our trait ref is known to match the impl. We will then cache the result as <SomeType<'erased> as SomeTrait> -> EvaluatedToOk.

If we later try to evaluate SomeType<'#_r> as SomeTrait, we will end up matching the evaluation cache entry, giving us a result of EvaluatedToOk. However, we should have gotten a result of EvaluatedToOkModuloRegions, since we don't know that '#_r == 'static holds.

This is really difficult to observe in practice, for a number of reasons:

• The relevant trait predicates need to get evaluated, not just registered in a FulfillmentContext.
• Trait evaluation usually goes through the evaluate_obligation query, which canonicalizes the regions in the input trait ref. To end up trying to evaluate SomeType<'static> as SomeTrait, we need to end up calling evaluate_predicate_recursively on it, with a different original trait ref used in the original query.
• Using EvaluatedToOkModuloRegions instead of EvaluatedToOk only seems to cause an error when it results in the incremental hash changing (I don't know if it's possible to weaponize this into extending a lifetime).

As a result, I haven't been able to minimize this.