Skip to content

Tokenizer returnings pointers make calculation of position info more complicated #97973

Closed
@lysnikolaou

Description

@lysnikolaou

Bacause the tokenizer only returns pointers to the beginning and the end of the token, calculating line numbers/column offsets is more complicated than needed.

Feature or enhancement

Instead of the tokenizer returning a token type and settings pointers, we wanna return the token type (remains as is) and then set a struct token that has the following information:

  1. Pointers to beginning and end
  2. Location information (lineno, col_offset, etc.)
  3. Level (the level in the parenstack)

This way the parser will have a much easier job of setting line numbers & column offsets in the generated AST numbers and will make some of our work on the f-strings parsing easier.

Metadata

Metadata

Assignees

Labels

type-featureA feature request or enhancement

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions