Stop using intermediate macros in definition of symbols

[sivadeilra]

Currently, the rustc_macros::symbols macro generates two
macro_rules! macros as its output. These two macros are
used in rustc_span/src/symbol.rs.

This means that each Symbol that we define is represented
in the AST of rustc_symbols twice: once in the definition
of the define_symbols! macro (similarly for the
`keywords! macro), and once in the rustc_span::symbols
definition.

That would be OK if there were only a handful of symbols,
but currently we define over 1100 symbols. The definition
of the define_symbols! macro contains the expanded definition
of each symbol, so that's a lot of AST storage wasted on a
macro that is used exactly once.

This commit removes the define_symbols macro, and simply
allows the proc macro to directly generate the
rustc_symbols::symbol::sym module.

The benefit is mainly in reducing memory wasted during
compilation of rustc itself. It should also reduce memory used
by Rust Analyzer.

This commit also reduces the size of the AST for symbol
definitions, by moving two #[allow(...)] attributes from
the symbol constants to the sym module. This eliminates 2200+
attribute nodes.

This commit also eliminates the need for the digits_array
constant. There's no need to store an array of Symbol values
for digits. We can simply define a constant of the base value,
and add to that base value.

I left the sym::integer function in rustc_span/src/symbol.rs
instead of moving it into rustc_macros/src/symbols.rs for two
reasons. First, because it's human-written code; it doesn't need
to be generated by the proc-macro. Second, because I didn't want
the #[allow(...)] attributes that I moved to the sym module
scope to apply to this function. The sym module re-exports the
integer function from its parent module.

[rust-highfive]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @oli-obk (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

[petrochenkov]

r? @petrochenkov

[petrochenkov]

The current structure was motivated by IDE experience first of all.
If code is not generated by a proc macro, then "Go to Definition" works on it, otherwise it does not.
(E.g. you can go to mod sym or mod kw and see where they are defined and which macros generate symbol lists.)
So as much code as possible was moved to outside of the macro.

I've just checked this branch with vscode + rust-analyzer, and the difference is pretty much the same as it was when the macro was introduced.

[sivadeilra]

The current structure was motivated by IDE experience first of all.

I can see how that would be useful. Go-to-definition could still be supported, by having the proc-macro generate sym_internal and then having rustc_span/src/symbol.rs contain pub use sym_internal as sym;, immediately before the symbols! { ... } table.

[rylev]

@sivadeilra Do we have any numbers on how much this helps memory usage? It would help to assess any possible tradeoffs in the experience of rust-analyzer.

[petrochenkov]

I think something like this would be optimal.
I.e. we keep the old structure, but use glob reexports instead of macros.
This way mod kw can be found by both go-to-definition and by textual search, and it's generally more clear what happens.

[sivadeilra]

That's pretty much what my second update in this PR does. It uses 2 re-exports, rather than macros.

[petrochenkov]

@sivadeilra
I mean reexporting only the macro-generated entities rather than the whole modules.

[sivadeilra]

I mean reexporting only the macro-generated entities rather than the whole modules.

That sounds good. I'll update with that.

[petrochenkov]

Thanks!
r=me with commits squashed.

[sivadeilra]

Rebased & squashed. Thanks!

[petrochenkov]

@bors r+

[bors]

📌 Commit 2b2462e has been approved by petrochenkov

[bors]

⌛ Testing commit 2b2462e with merge 8d006c0...

[bors]

☀️ Test successful - checks-actions
Approved by: petrochenkov
Pushing 8d006c0 to master...