ICE: no errors encountered even though `delay_span_bug` issued (regression on 2023-10-12)

[xxchan]

Can be reproduced by:

# I believe which commit doesn't matter. But tested on f27182f35cad6831b5751f677f32645aa1321613, and also 7761c4ff18b19718d8dbde5e862505e8bbdef74e
git clone https://github.com/risingwavelabs/risingwave 
cargo +nightly check

Sorry for not having a minimal example yet.

Meta

rust version 1.75.0-nightly (475c71d 2023-10-11)

(A little background: We have a GHA to check whether build fails on latest nighty rust. It succeeds on 2023-10-11, but failed with ICE on 2023-10-12)

https://github.com/risingwavelabs/risingwave/actions/runs/6502231902/job/17660915782

Error output

error: internal compiler error: no errors encountered even though `delay_span_bug` issued

error: internal compiler error: ambiguity performing ParamEnvAnd { param_env: ParamEnv { caller_bounds: [Binder { value: OutlivesPredicate(ReEarlyBound(DefId(0:589 ~ risingwave_compute[0de4]::rpc::service::exchange_service::{impl#2}::get_stream::'life0), 0, 'life0), ReEarlyBound(DefId(0:590 ~ risingwave_compute[0de4]::rpc::service::exchange_service::{impl#2}::get_stream::'async_trait), 1, 'async_trait)), bound_vars: [] }, Binder { value: OutlivesPredicate(rpc::service::exchange_service::ExchangeServiceImpl, ReEarlyBound(DefId(0:590 ~ risingwave_compute[0de4]::rpc::service::exchange_service::{impl#2}::get_stream::'async_trait), 1, 'async_trait)), bound_vars: [] }], reveal: UserFacing }, value: ProvePredicate { predicate: Binder { value: TraitPredicate(<std::pin::Pin<std::boxed::Box<{async block@src/compute/src/rpc/service/exchange_service.rs:77:71: 116:6}>> as std::ops::CoerceUnsized<std::pin::Pin<std::boxed::Box<dyn futures::Future<Output = std::result::Result<madsim_tonic::Response<impl tokio_stream::Stream<Item = std::result::Result<risingwave_pb::task_service::GetStreamResponse, madsim_tonic::Status>>>, madsim_tonic::Status>> + std::marker::Send>>>>, polarity:Positive), bound_vars: [] } } }
   --> src/compute/src/rpc/service/exchange_service.rs:77:71
    |
77  |       ) -> std::result::Result<Response<Self::GetStreamStream>, Status> {
    |  _______________________________________________________________________^
78  | |         use risingwave_pb::task_service::get_stream_request::*;
79  | |
80  | |         let peer_addr = request
...   |
115 | |         )))
116 | |     }
    | |_____^
    |
note: delayed at /rustc/475c71da0710fd1d40c046f9cee04b733b5b2b51/compiler/rustc_trait_selection/src/traits/query/type_op/mod.rs:170:26
         0: <rustc_errors::HandlerInner>::emit_diagnostic
         1: <rustc_errors::Handler>::delay_span_bug::<rustc_span::span_encoding::Span, alloc::string::String>
         2: <rustc_borrowck::type_check::TypeChecker>::prove_trait_ref
         3: <rustc_borrowck::type_check::TypeChecker>::typeck_mir
         4: rustc_borrowck::type_check::type_check
         5: rustc_borrowck::nll::compute_regions
         6: rustc_borrowck::do_mir_borrowck
         7: rustc_borrowck::mir_borrowck
         8: rustc_query_impl::plumbing::__rust_begin_short_backtrace::<rustc_query_impl::query_impl::mir_borrowck::dynamic_query::{closure#2}::{closure#0}, rustc_middle::query::erase::Erased<[u8; 8]>>
         9: rustc_query_system::query::plumbing::try_execute_query::<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::VecCache<rustc_span::def_id::LocalDefId, rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt, false>
        10: rustc_query_impl::query_impl::mir_borrowck::get_query_non_incr::__rust_end_short_backtrace
        11: <rustc_hir_analysis::collect::type_of::opaque::TaitConstraintLocator>::check
        12: <rustc_hir_analysis::collect::type_of::opaque::TaitConstraintLocator as rustc_hir::intravisit::Visitor>::visit_impl_item
        13: <rustc_hir_analysis::collect::type_of::opaque::TaitConstraintLocator as rustc_hir::intravisit::Visitor>::visit_item
        14: <rustc_hir_analysis::collect::type_of::opaque::TaitConstraintLocator as rustc_hir::intravisit::Visitor>::visit_item
        15: rustc_hir_analysis::collect::type_of::type_of_opaque
        16: rustc_query_impl::plumbing::__rust_begin_short_backtrace::<rustc_query_impl::query_impl::type_of_opaque::dynamic_query::{closure#2}::{closure#0}, rustc_middle::query::erase::Erased<[u8; 8]>>
        17: rustc_query_system::query::plumbing::try_execute_query::<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::DefaultCache<rustc_span::def_id::DefId, rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt, false>
        18: rustc_query_impl::query_impl::type_of_opaque::get_query_non_incr::__rust_end_short_backtrace
        19: rustc_middle::query::plumbing::query_get_at::<rustc_query_system::query::caches::DefaultCache<rustc_span::def_id::DefId, rustc_middle::query::erase::Erased<[u8; 8]>>>
        20: rustc_hir_analysis::collect::type_of::type_of
        21: rustc_query_impl::plumbing::__rust_begin_short_backtrace::<rustc_query_impl::query_impl::type_of::dynamic_query::{closure#2}::{closure#0}, rustc_middle::query::erase::Erased<[u8; 8]>>
        22: rustc_query_system::query::plumbing::try_execute_query::<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::DefaultCache<rustc_span::def_id::DefId, rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt, false>
        23: rustc_query_impl::query_impl::type_of::get_query_non_incr::__rust_end_short_backtrace
        24: rustc_middle::query::plumbing::query_get_at::<rustc_query_system::query::caches::DefaultCache<rustc_span::def_id::DefId, rustc_middle::query::erase::Erased<[u8; 8]>>>
        25: rustc_hir_analysis::check::check::check_mod_item_types
        26: rustc_query_impl::plumbing::__rust_begin_short_backtrace::<rustc_query_impl::query_impl::check_mod_item_types::dynamic_query::{closure#2}::{closure#0}, rustc_middle::query::erase::Erased<[u8; 0]>>
        27: rustc_query_system::query::plumbing::try_execute_query::<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::DefaultCache<rustc_span::def_id::LocalModDefId, rustc_middle::query::erase::Erased<[u8; 0]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt, false>
        28: rustc_query_impl::query_impl::check_mod_item_types::get_query_non_incr::__rust_end_short_backtrace
        29: rustc_hir_analysis::check_crate
        30: rustc_interface::passes::analysis
        31: rustc_query_impl::plumbing::__rust_begin_short_backtrace::<rustc_query_impl::query_impl::analysis::dynamic_query::{closure#2}::{closure#0}, rustc_middle::query::erase::Erased<[u8; 1]>>
        32: rustc_query_system::query::plumbing::try_execute_query::<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::SingleCache<rustc_middle::query::erase::Erased<[u8; 1]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt, false>
        33: rustc_query_impl::query_impl::analysis::get_query_non_incr::__rust_end_short_backtrace
        34: std::sys_common::backtrace::__rust_begin_short_backtrace::<rustc_interface::util::run_in_thread_with_globals<rustc_interface::interface::run_compiler<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure#1}>::{closure#0}, core::result::Result<(), rustc_span::ErrorGuaranteed>>::{closure#0}::{closure#0}, core::result::Result<(), rustc_span::ErrorGuaranteed>>
        35: <<std::thread::Builder>::spawn_unchecked_<rustc_interface::util::run_in_thread_with_globals<rustc_interface::interface::run_compiler<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure#1}>::{closure#0}, core::result::Result<(), rustc_span::ErrorGuaranteed>>::{closure#0}::{closure#0}, core::result::Result<(), rustc_span::ErrorGuaranteed>>::{closure#1} as core::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
        36: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
                   at /rustc/475c71da0710fd1d40c046f9cee04b733b5b2b51/library/alloc/src/boxed.rs:2007:9
        37: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
                   at /rustc/475c71da0710fd1d40c046f9cee04b733b5b2b51/library/alloc/src/boxed.rs:2007:9
        38: std::sys::unix::thread::Thread::new::thread_start
                   at /rustc/475c71da0710fd1d40c046f9cee04b733b5b2b51/library/std/src/sys/unix/thread.rs:108:17
        39: <unknown>
        40: <unknown>
   --> src/compute/src/rpc/service/exchange_service.rs:77:71
    |
77  |       ) -> std::result::Result<Response<Self::GetStreamStream>, Status> {
    |  _______________________________________________________________________^
78  | |         use risingwave_pb::task_service::get_stream_request::*;
79  | |
80  | |         let peer_addr = request
...   |
115 | |         )))
116 | |     }
    | |_____^

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: please attach the file at `/home/runner/work/risingwave/risingwave/rustc-ice-2023-10-13T00:18:36.192779807Z-56318.txt` to your bug report

note: compiler flags: --crate-type lib -C lto=off -C embed-bitcode=no -C target-feature=+avx2 -C link-arg=-fuse-ld=lld -C link-arg=-Wl,--no-rosegment

Backtrace

<backtrace>

[matthiaskrgr]

This might be a duplicate of #108498 @aliemjay will know more I believe :)

[albertlarsan68]

I am minimizing this huge repo, I'll report when I have a result.

[matthiaskrgr]

https://github.com/matthiaskrgr/glacier2/blob/master/ices/108498.rs reproduces an ice with very similar stacktrace

[aliemjay]

does the code compile before #114586 (i.e in nightly-10-09-2023)?

[xxchan]

Yes, it compiles in 2023-10-09.

…

On Sat, Oct 14, 2023 at 7:10 AM Ali MJ Al-Nasrawy ***@***.***> wrote: does the code compiles before #114586 <#114586> (i.e in nightly-10-09-2023)? — Reply to this email directly, view it on GitHub <#116684 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJBQZNPT5CQMXQ6D6WOAISTX7HC6ZAVCNFSM6AAAAAA56UVS5GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRSGM3DMMRXGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[aliemjay]

Yes, it compiles in 2023-10-09.

sorry, wrong date format. I meant 2023-09-10

[xxchan]

Yes. 2023-09-10 is ok. BTW, we are currently mainly using 2023-09-09 ( https://github.com/risingwavelabs/risingwave/blob/main/ci/rust-toolchain)

…

On Sat, Oct 14, 2023 at 11:56 AM Ali MJ Al-Nasrawy ***@***.***> wrote: Yes, it compiles in 2023-10-09. sorry, wrong date format. I meant 2023-09-10 — Reply to this email directly, view it on GitHub <#116684 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJBQZNJSJHNBZUNXUYYWSUTX7IENVAVCNFSM6AAAAAA56UVS5GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRSGU2TANJQHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[aliemjay]

@rustbot claim