Supporting leading infix operators

[susliko]

We have an issue (#141) to support leading infix operators
The doc defines them as

• a symbolic identifier such as +, or approx_==, or an identifier in backticks that
• starts a new line, and
• is not following a blank line, and
• is followed by at least one whitespace character and a token that can start an expression.
• Furthermore, if the operator appears on its own line, the next line must have at least the same indentation width as the operator

What is a symbolic identifier, formally? Is it op from the syntax reference?

opchar ::= ‘!’ | ‘#’ | ‘%’ | ‘&’ | ‘*’ | ‘+’ | ‘-’ | ‘/’ | ‘:’ | ‘<’ | ‘=’ | ‘>’ | ‘?’ | ‘@’ | ‘\’ | ‘^’ | ‘|’ | ‘~’
op ::= opchar {opchar}

Our tree-sitter grammar does not define a separate rule this type of identifier, it's being a part of $.operator_identifier

Probably, if we had a separate rule for this symbolic identifer, we could tweak the external scanner to stop issuing AUTOMATED_SEMICOLON tokens when encountering a valid position for leading infix expression

if (valid_symbols[AUTOMATIC_SEMICOLON] && newline_count > 0) {
...
  if (valid_symbols[LEADING_INFIX_OP] && newline_count == 1) {
    if (lookahead_matches_opchar(lexer)) {
      while (lookahead_matches_opchar(lexer)) {
        advance(lexer)
      }
      if (iswspace(lexer->lookahead) {
        return false // no AUTOMATIC_SEMICOLON should be produced
      }
    }
  }

Do you have any thoughts on this? 🤔

[eed3si9n]

What is a symbolic identifier, formally? Is it op from the syntax reference?

Looking at the implementation it's:

1. A backquoted identifier
2. An identifier whose last character is an opchar (This includes both op like + and alphaid like approx_==)

I am wondering if we need to implement identifier scanner in the external scanner eventually.