Closed
Description
Bacause the tokenizer only returns pointers to the beginning and the end of the token, calculating line numbers/column offsets is more complicated than needed.
Feature or enhancement
Instead of the tokenizer returning a token type and settings pointers, we wanna return the token type (remains as is) and then set a struct token
that has the following information:
- Pointers to beginning and end
- Location information (
lineno
,col_offset
, etc.) - Level (the level in the
parenstack
)
This way the parser will have a much easier job of setting line numbers & column offsets in the generated AST numbers and will make some of our work on the f-strings parsing easier.