Syntax Highlighting Errors when Diffing - Fixed Format Fortran

[smf007]

Description

The chroma library had added a fixed format fortran syntax highlighter back in version v0.9.2, which gitea started using as of v1.15.0-RC1. The fortran fixed format syntax highlighting works for displaying files in gitea, however it is broken when displaying a diff of changes.

This can be see using the try.gitea portal:

The syntax highlighting is correct
https://try.gitea.io/test1765/fortran_test/src/branch/main/fortran_fixed.f

The syntax highlighting is incorrect when doing a diff
https://try.gitea.io/test1765/fortran_test/commit/228401fa1860d7aea71f3511937e49ee39d10938

Gitea Version

Tried both 1.16.4, 1.16.7

Can you reproduce the bug on the Gitea demo site?

Yes

Log Gist

No response

Screenshots

See https://try.gitea.io/test1765/fortran_test/commit/228401fa1860d7aea71f3511937e49ee39d10938

Git Version

No response

Operating System

ROCKY 8.5 (RHEL 8.5)

How are you running Gitea?

Used pre-built binaries for Linux x64, running on ROCKY Linux 8.5.

Database

No response

[wxiaoguang]

It seems to be similar with #19508

• Diff highlighting doesn't always include all changes #19508

(hmm, maybe not the same reason)

[wxiaoguang]

The root problem is different from #19508

The problem here is that DiffInlineWithHighlightCode -> highlight.Code -> lexer.Tokenise & formatter.Format doesn't work correctly for the inline Fortran code.

The result of highlight.Code is incorrect from beginning.

[wxiaoguang]

Another test shows that the problem is related to github.com/alecthomas/chroma@v0.10.0/lexers/f/fortran_fixed.go, which is used by inline diff highlighter

If you rename the file from test.f to test.f90 (use lexers/f/fortran.go), then the highlight in diff view is correct.

[wxiaoguang]

Workaround, either:

• use *.f90
• [or] set highlight.mapping in app.ini to force the *.f use Fortran instead of FortranFixed lexer.

[smf007]

Workaround, either:

* use `*.f90`

* [or] set `highlight.mapping` in app.ini to force the `*.f` use `Fortran` instead of `FortranFixed` lexer.

I can confirm that overriding the highlight mapping makes the diff more usable.

Looking at this more closely though, the fixed format highlighting performed by the library is itself incorrect regardless of using fortranfixed or fortran. Fortranfixed is not resolving comments as actual comments. This can be seen in the example, lines 1-5 should be greyed out as comments, but are not https://try.gitea.io/test1765/fortran_test/src/branch/main/fortran_fixed.f So my original statement above that the fixed format highlighting was correct is actually false.

[wxiaoguang]

There is a tricky thing in Gitea.

When doing whole file rendering, the extension name is not respected, instead, Gitea will guess what's the language of the source file by its content, so even if the ext name is .f, the full file view can still be rendered correctly like .f90

However, when Gitea is doing inline diff, it's done line by line without the guess, and only use the ext name as the language, then FortranFixed is used for .f, the rendered result is not ideal.

[smf007]

Seems I forgot to update this issue. I had done some work upstream in Chroma for the parsing of fortran.

• Update fortran lexer alecthomas/chroma#646
• Tweak Fortran Lexer Parsing of Whitespace within keywords alecthomas/chroma#647
• Revise fortran fixed format lexer to recognize comments specified using the "!" alecthomas/chroma#648

646 & 648 were included in Chroma 2.3.0, and 647 is in master awaiting a version tag.

Chroma 2.3.0 has been merged into Gitea 1.18.0 branch.

As the demo site is kept up to date, I need to check if the Chroma 2.3.0 inclusion has resolved most of this issue.