PHP 8.0+ does no longer accept a comment in the whitespace between ampersand and argument in function signature

[olleharstedt]

Description

The following code:

<?php

function foo(
array
#if __PHP__
&
#endif
$bodies
)
{
}

Resulted in this output:

    Parse error: syntax error, unexpected variable "$bodies" in /in/aq2rT on line 8

    Process exited with code 255.

But I expected this output instead:

<Empty, should compile without syntax error as in PHP 7.4

PHP Version

PHP 8.1 and above, works fine in PHP 7.4

Operating System

No response

[iluuu1994]

This is expected and caused by some parser/tokenizer hackery for intersection types that was necessary to avoid ambiguity. See 069a9fa.

[olleharstedt]

Won't fix = impossible to fix?

[iluuu1994]

Yes 🙂

[olleharstedt]

😢

[iluuu1994]

I think in theory we could adjust the tokens so that variables are assumed after & /* comment */ instead of intersection types, but one case will always not work.

[olleharstedt]

Are comments part of the token stream? That's the problem? Because whitespace and newline works fine.

[iluuu1994]

No. The problem is that the grammar is ambiguous. Our LR(1) parser only gets one token of lookahead, but code like function foo(Foo |& Bar $param) { (| denoting the current position) requires multiple tokens of lookahead to understand if this is a param by reference (Foo &$param) or an intersection type. We avoid this ambiguity by creating different tokens for & depending on what follows, but this has limitations in that comments will influence tokenization.

php-src/Zend/zend_language_scanner.l

Lines 1872 to 1879 in b96b88b

	<ST_IN_SCRIPTING>"&"[ \t\r\n]*("$"|"...") {
	yyless(1);
	RETURN_TOKEN(T_AMPERSAND_FOLLOWED_BY_VAR_OR_VARARG);
	}
	<ST_IN_SCRIPTING>"&" {
	RETURN_TOKEN(T_AMPERSAND_NOT_FOLLOWED_BY_VAR_OR_VARARG);
	}

[olleharstedt]

Ah ok. But expanding <ST_IN_SCRIPTING>"&"[ \t\r\n]*("$"|"...") { to also eat comments could fix it? If that's possible during lexing. But maybe not good for lexing performance. Or possible only eat the single-line comments # and //.