UI unresponsive when sketch has a very long line

[per1234]

Describe the problem

Arduino sketches may contain large machine generated arrays for data such as images. These may span many columns, using a block that follows the dimensions of the source data (e.g., an array with 320 elements per line might be generated for a 320x240 pixel image), or even be all on a single long line.

🐛 The Arduino IDE UI becomes noticeably laggy or even completely unresponsive when the sketch contains a long line

To reproduce

1. Download the following demonstration sketch, which contains a line 18432 characters long:
   LongLine.zip
2. Unzip the downloaded file.
3. Open the "LongLine" sketch in the Arduino IDE.
   🐛 The IDE UI is completely unresponsive.
4. Force close the Arduino IDE.
5. Start the Arduino IDE (making sure it loads an innocuous sketch on startup).
6. Open the "Command Palette" via the Ctrl+Shift+P shortcut (Command+Shift+P for macOS users).
7. Select the "Preferences: Open Settings (UI)" command.
8. In the "Search Settings" field, type editor.maxTokenizationLineLength
9. Change the value of the "Editor: Max Tokenization Line Length" setting from the default 20000 to 500
10. Open the "LongLine" sketch in the Arduino IDE.
    🙂 The IDE remains perfectly responsive.

Expected behavior

IDE is usable when the sketch contains long lines.

Arduino IDE version

2.0.0-rc9.2.snapshot-de32bdd

Operating system

Windows

Operating system version

10

Additional context

I am able reproduce the issue in Theia Blueprint (but not in VS Code), so the inability to handle such content is not a bug in the Arduino IDE codebase.

I see that this was reported in the Theia project and fixed by reducing the default value of the editor.maxTokenizationLineLength to 400: eclipse-theia/theia#8021

So a similar change should be made in Arduino IDE as well.

---

I used a ridiculously long line in the demo sketch (though it was generated from an image of only 32x32 px, using an established tool). More reasonable line lengths result in less dramatic impact, but still make the IDE unpleasant to use. A real world file was provided here:

https://forum.arduino.cc/t/2-0-slows-down-if-very-long-lines-but-ok-with-crs-line-feeds-inserted/1021335/5

---

It seems that changes to the editor.maxTokenizationLineLength setting are not applied to sketches which have already been "tokenized", so make sure to reload the sketch if you are experimenting with the setting.

---

The issue is not related to the Arduino Language Server because it occurs even when the language server is not running due to not having a board open.

---

Originally reported at https://forum.arduino.cc/t/2-0-slows-down-if-very-long-lines-but-ok-with-crs-line-feeds-inserted/1021335

Issue checklist

• I searched for previous reports in the issue tracker
• I verified the problem still occurs when using the latest nightly build
• My report contains all necessary details

[kittaakos]

I see that this was reported in the Theia project and fixed by reducing the default value of the editor.maxTokenizationLineLength to 400: eclipse-theia/theia#8021

https://github.com/eclipse-theia/theia/blob/2289366b82f6d013fd4de7b990b21d9037a74c32/packages/editor/src/browser/editor-generated-preference-schema.ts#L118-L124

We should add a default preference in IDE2. I will take care of it.