Assertion failed: Cannot set `break_last_token` and have trailing token

[jruderman]

Found with fuzz-rustc plus a new AST-guided mutator

Code

fn main() {
    x::<#[a]y::<z>>
}

What's going on

• The attribute #[a] applies to y::<z>
• The > was initially lexed as part of a >> token
• This seems to be exactly the case described by the comment above the assertion

Meta

rustc --version --verbose:

rustc 1.66.0-nightly (6b3ede3f7 2022-10-13)
binary: rustc
commit-hash: 6b3ede3f7bc502eba7bbd202b4b9312d812adcd7
commit-date: 2022-10-13
host: x86_64-apple-darwin
release: 1.66.0-nightly
LLVM version: 15.0.2

Error output

thread 'rustc' panicked at 'assertion failed: `(left == right)`
  left: `MaybeComma`,
 right: `None`: Cannot set `break_last_token` and have trailing token', compiler/rustc_parse/src/parser/attr_wrapper.rs:295:13

Backtrace

stack backtrace:
   0: _rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::panicking::assert_failed_inner
   3: core::panicking::assert_failed::<rustc_parse::parser::TrailingToken, rustc_parse::parser::TrailingToken>
   4: <rustc_parse::parser::Parser>::collect_tokens_trailing_token::<rustc_ast::ptr::P<rustc_ast::ast::Expr>, <rustc_parse::parser::Parser>::collect_tokens_for_expr<<rustc_parse::parser::Parser>::parse_dot_or_call_expr::{closure#0}>::{closure#0}>
   5: <rustc_parse::parser::Parser>::parse_prefix_expr
   6: <rustc_parse::parser::Parser>::parse_assoc_expr_with
   7: <rustc_parse::parser::Parser>::parse_generic_arg
   8: <rustc_parse::parser::Parser>::parse_angle_args
   9: <rustc_parse::parser::Parser>::parse_path_segment
  10: <rustc_parse::parser::Parser>::parse_path_segments
  11: <rustc_parse::parser::Parser>::parse_path_inner
  12: <rustc_parse::parser::Parser>::parse_stmt_path_start
  13: <rustc_parse::parser::Parser>::parse_stmt_without_recovery
  14: <rustc_parse::parser::Parser>::parse_full_stmt
  15: <rustc_parse::parser::Parser>::parse_block_tail
  16: <rustc_parse::parser::Parser>::parse_block_common
  17: <rustc_parse::parser::Parser>::parse_fn
  18: <rustc_parse::parser::Parser>::parse_item_common
  19: <rustc_parse::parser::Parser>::parse_mod
  20: rustc_parse::parse_crate_from_file
  21: <rustc_session::session::Session>::time::<core::result::Result<rustc_ast::ast::Crate, rustc_errors::diagnostic_builder::DiagnosticBuilder<rustc_errors::ErrorGuaranteed>>, rustc_interface::passes::parse::{closure#0}>
  22: rustc_interface::passes::parse
  23: <rustc_interface::queries::Queries>::parse
  24: <rustc_interface::interface::Compiler>::enter::<rustc_driver::run_compiler::{closure#1}::{closure#2}, core::result::Result<core::option::Option<rustc_interface::queries::Linker>, rustc_errors::ErrorGuaranteed>>
  25: rustc_span::with_source_map::<core::result::Result<(), rustc_errors::ErrorGuaranteed>, rustc_interface::interface::create_compiler_and_run<core::result::Result<(), rustc_errors::ErrorGuaranteed>, rustc_driver::run_compiler::{closure#1}>::{closure#1}>
  26: rustc_interface::interface::create_compiler_and_run::<core::result::Result<(), rustc_errors::ErrorGuaranteed>, rustc_driver::run_compiler::{closure#1}>
  27: <scoped_tls::ScopedKey<rustc_span::SessionGlobals>>::set::<rustc_interface::interface::run_compiler<core::result::Result<(), rustc_errors::ErrorGuaranteed>, rustc_driver::run_compiler::{closure#1}>::{closure#0}, core::result::Result<(), rustc_errors::ErrorGuaranteed>>
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[chenyukang]

@rustbot claim

[chenyukang]

@Aaron1011 Do you have any suggestion for this case?

The code path is going like this:

Parser eat the first one of '>' at here and set break_last_token:

rust/compiler/rustc_parse/src/parser/mod.rs

Line 706 in dbaf3e6

self.token_cursor.break_last_token = true;

fall pass at here:

rust/compiler/rustc_parse/src/parser/expr.rs

Line 3087 in 53728ff

TrailingToken::MaybeComma

So this assertion is failed:

rust/compiler/rustc_parse/src/parser/attr_wrapper.rs

Line 295 in c0983a9

assert_eq!(

The result is:
trailing == TrailingToken::MaybeComma and break_last_token is true.

Seems the assertion missed this scenario, do we need to add a TrailingToken::Gt ?

[Aaron1011]

@chenyukang The assertion I wrote is overly conservative - it really just needs to check that we don't simultaneously break a token and 'really' capture a trailing token. Specifically, we only need to panic if we actually incremented end_pos in the match block.

I think the simplest fix would be to add a captured_trailing variable, which gets set to true in the parts of the match block that increment end_pos. The assertion can then be changed to assert!(!captured_trailing, "Cannot set break_last_token and have trailing token")