Transition rustc Parser to proc_macro token model

[matklad]

Currently, there are two different approaches for dealing with composite tokens like >> in rustc.

1. Keep tokens in composed form, and split into pieces, > and >, when necessary.
2. Keep tokens decomposed, with jointness information, and join tokens when necessary.

At the moment, the first approach is used by the parser, and the second approach is used by the proc_macro API. It would be good to move the parser to the decomposed approach as well, as it is somewhat more natural, more future-compatible (one can introduce new tokens) and having two of a thing is bad in itself!

Here are some relevant bits of the code that handle composed model:

• Composed tokens as produced by rustc_lexer
• Composed tokens preserved by the token cooking
• Here's the bit when we produce a TokenTree, consumed by the parser. Note that, although we are tracking jointness here, the tokens are composed.
• Here's the bit of the parser which decomposes tokens on the fly.

Here are the bits relevant to decomposed model:

• Gluing tokens in TokenStreamBuilder
• Token::glue

Note that the tt matcher in macro_rules eats one composed token, and this is affects language specification.
That is, when we transition to decomposed model, we'll need to fix this code to eat one composed token to maintain backwards compatibility.

[matklad]

cc @petrochenkov

[matklad]

made some initial stabs in matklad@0d46730.

The idea is to remove cases from Token::glue one by one, until no tokens are glued together, except for tt matcher.

Faced a couple of problems:

• parse_assert accepts an &[TokenTree], which throughs away jointness info, so assert!(1 != 1) does not parse
• NtTT should be changed from holding a TokenTree to holding an TokenStream, to account for the fact that $tt:tt eats <<, which are two tokens in the new model

EDIT: more stabs at https://github.com/matklad/rust/tree/decomposed-neq. fixed all parser problems, not it looks like we are loosing jointness info somewhere..

[matklad]

Fond next obstacle: in macros by example, quoted::TokenTree erases jointness information, so

macro_rules! m { () => (==) }

produces = =. Note that jointness seems to be correctly preserved by macro invocations. I guess we should change quoted::Delimited to store token trees with jointness, to better mirror the TokenStream.

[matklad]

@petrochenkov is ^ a good plan? Or are there any bigger plans for refactoring quote, which we should do first?

[petrochenkov]

I never thought about this case.
Preserving jointness would probably be a good start unconditionally.

IIRC, the stuff in syntax::ext::tt::quoted should behave exactly like usual token streams, it's just re-hashed slightly for more convenient work with macro_rules DSL.
(Maybe it doesn't even add too much convenience and can be removed? Who knows.)
Anyway, if it behaves differently than regular token streams, it's something that's better fixed.

[matklad]

(Maybe it doesn't even add too much convenience and can be removed? Who knows.)

Hm, so that we represent $var:ty as literally $var:ty, and just match them on the fly while transcribing? I guess I'll try this approach for mbe in rust-analyzer. Currently, we blindly copy rustc approach with duplicated TokenStream.

For rustc, I feel like it's better to stick with the current model for the time being