Failures found using fuzz testing

[c42f]

Using the tools from #379 uncovers several bugs. For example

fuzz_test(try_parseall_failure, product_token_fuzz(cutdown_tokens, 2))
fuzz_test(try_hook_failure, product_token_fuzz(cutdown_tokens, 2))
fuzz_test(try_parseall_failure, random_token_fuzz(cutdown_tokens, 10, 10_000_000))

Some errors found with this:

Parser errors

• parseall(Expr, "@(") (Fix crash with empty macro name when parsing "@(" #382)
• parseall(SyntaxNode, "function(where") (Fix crash when parsing malformed function(where #388)
• parseall(SyntaxNode, "x{primitive\ntype"), parseall(SyntaxNode, "(primitive\ntype") (Disallow newline between contextual keyword pairs in parentheses #386)
• parseall(SyntaxNode, "var\"\"``\$"), parseall(SyntaxNode, "+||where'``\$") (Fix crashes due to lexing ambiguity of string delimiters #394)

Tree building errors

• parseall(SyntaxNode, "using . ...", ignore_errors=true)
• parseall(Expr, "\x04'\0", ignore_errors=true)
• parseall(Expr, "@(var\"", ignore_errors=true)
• parseall(Expr, "for\n\n<:", ignore_errors=true) (Fix bug parsing unary subtypes with newlines #393)
• parseall(Expr, "x:y:<", ignore_errors=true)

Hook errors

• Meta_parseall("x.") (Fix error in hooks when parsing incomplete x. #385)

[vtjnash]

Are these also fuzz errors:

Parse core.jl: Test Failed at /home/pkgeval/.julia/packages/JuliaSyntax/r20Wu/test/test_utils.jl:241
  Expression: reduced_failures == []
   Evaluated: AbstractString["for f() in 1:3\n        push!(a, f())\n        push!(fs, f)\n    end",
                             " {{}}where{}"] == Any[]

https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_hash/1fc9d54_vs_72cd63c/JuliaSyntax.primary.log

[c42f]

Thanks for pointing that out. Also now visible on JuliaSyntax dev branch with nightly (eg, https://github.com/JuliaLang/JuliaSyntax.jl/actions/runs/6968830137/job/18963527621#step:21:414)

Here's a slightly less minimal but more realistic reproduction of the new failure of parsing comparison of JuliaLang test/core.jl:

julia> dump(JuliaSyntax.fl_parse(Expr, "A where {T} == Z"))
Expr
  head: Symbol where
  args: Array{Any}((2,))
    1: Symbol A
    2: Expr
      head: Symbol call
      args: Array{Any}((3,))
        1: Symbol ==
        2: Expr
          head: Symbol braces
          args: Array{Any}((1,))
            1: Symbol T
        3: Symbol Z

julia> dump(JuliaSyntax.parsestmt(Expr, "A where {T} == Z"))
Expr
  head: Symbol call
  args: Array{Any}((3,))
    1: Symbol ==
    2: Expr
      head: Symbol where
      args: Array{Any}((2,))
        1: Symbol A
        2: Symbol T
    3: Symbol Z

vs the following without the curlies which parses the same in both parsers

julia> dump(JuliaSyntax.parsestmt(Expr, "A where T == Z"))
Expr
  head: Symbol where
  args: Array{Any}((2,))
    1: Symbol A
    2: Expr
      head: Symbol call
      args: Array{Any}((3,))
        1: Symbol ==
        2: Symbol T
        3: Symbol Z