Expand and clarify tokens.

[ehuss]

No description provided.

[ehuss]

I'm open to feedback on this. I felt like tokens were underdocumented and widely dispersed.

Notes:

• I kinda like the cross-referencing, but I'm not sure if it is a good idea.
• The cross references include some potentially confusing parsing situations. For example, the token >>= is normally for shift right assignment, but could also be nested generics followed by assignment. I'm not sure if I covered every situation where rust splits the tokens based on context. Perhaps this should be explicitly described?
• The symbol names are otherwise unused, so I'm not sure if they serve a purpose.
• This also introduces some new terminology (punctuation, delimiters) which doesn't have much precedent.
• I renamed :: to PathSep from ModSep since it can be used in much more than just modules and I felt that could be confusing.
• I noticed some parts of the reference grammar do not make token separation clear. For example an inner attribute is made of separate tokens #, !, [,..., ]. I could in a separate PR attempt to make that clearer?
• I think I got everything correct, but there were a few tricky bits. For example, underscore isn't really mentioned anywhere as a token, and in rustc internally it is parsed as a keyword, but I think it makes more sense to keep it as a punctuation token since keywords are typically actual words.
• I couldn't find _ used in inferred types documented anywhere.
• I left out deprecated/unused/unstable tokens.

[Havvy]

A first pass look over this and it looks good to me.

[Havvy]

💟 Thanks! The table is definitely much more useful than just linking to the operands page.