Invalidate core cache if external dependency was modified

[Bodmer]

Describe the request

Add an option to prevent caching or to clear the cache.

Describe the current behavior

Some libraries and board package use a build_opt.h file in the sketch to configure the board core or indeed a library.

See for example here:
https://github.com/stm32duino/wiki/wiki/Customize-build-options-using-build_opt.h

The problem is that swapping between sketches that use slightly different build options will fail to use the new options.

The crude work-around is to select a different processor, compile (even if this generates errors) so the cache is deleted/changed, then swap back to the correct processor and compile again. Clearly this is a rather clumsy way of forcing a cache clear.

If there is a neater solution then please post it. Otherwise please update the IDE with an option to use/not use caching. Clearly this option can default to the current arrangement.

Demo

$ arduino-cli version

arduino-cli.exe  Version: git-snapshot Commit: b82a519e Date: 2023-10-21T11:43:36Z

$ export ARDUINO_DIRECTORIES_DATA="/tmp/arduino-cli-directories/data"  # Use a throwaway directories.data for the demo

$ arduino-cli core install arduino:avr@1.8.6  # Install an arbitrary platform for use in the demo

[...]

$ echo 'compiler.cpp.extra_flags="-I{build.source.path}"' > "${ARDUINO_DIRECTORIES_DATA}/packages/arduino/hardware/avr/1.8.6/platform.local.txt"  # Add the sketch folder to the compiler's search path

$ printf '#include <SomeSketchHeaderFile.h>\n#ifdef MACRO_FROM_SKETCH_HEADER_FILE\n#error "MACRO_FROM_SKETCH_HEADER_FILE is defined"\n#endif' > "${ARDUINO_DIRECTORIES_DATA}/packages/arduino/hardware/avr/1.8.6/cores/arduino/DependsOnSomeSketchHeaderFile.cpp"  # Add source file to the core that depends on a header file from the sketch

$ SKETCH_PATH="/tmp/FooSketch"

$ arduino-cli sketch new "$SKETCH_PATH"

$ touch "${SKETCH_PATH}/SomeSketchHeaderFile.h"

$ arduino-cli compile --fqbn arduino:avr:uno --verbose "$SKETCH_PATH" # Create core cache

[...]
Compiling core...
[...]
Archiving built core (caching) in: C:\Users\per\AppData\Local\Temp\arduino\cores\arduino_avr_uno_5958a1d0f1df880da5aa74c2f1a07415\core.a
[...]

$ echo "#define MACRO_FROM_SKETCH_HEADER_FILE" > "${SKETCH_PATH}/SomeSketchHeaderFile.h"

$ arduino-cli compile --fqbn arduino:avr:uno --verbose "$SKETCH_PATH"

[...]
Compiling core...
Using precompiled core: C:\Users\per\AppData\Local\Temp\arduino\cores\arduino_avr_uno_5958a1d0f1df880da5aa74c2f1a07415\core.a
[...]
Sketch uses 444 bytes (1%) of program storage space. Maximum is 32256 bytes.
Global variables use 9 bytes (0%) of dynamic memory, leaving 2039 bytes for local variables. Maximum is 2048 bytes.

🙁 The cached core was used even though a dependency had changed.

The expected result is that the cache would be invalidated, which in this simplified demo would result in the compilation failing:

$ arduino-cli compile --clean --fqbn arduino:avr:uno --verbose "$SKETCH_PATH"

[...]
Compiling core...
[...]
C:\Users\per\AppData\Local\Temp\arduino-cli-directories\data\packages\arduino\hardware\avr\1.8.6\cores\arduino\DependsOnSomeSketchHeaderFile.cpp:3:2: error: #error "MACRO_FROM_SKETCH_HEADER_FILE is defined"
 #error "MACRO_FROM_SKETCH_HEADER_FILE is defined"
  ^~~~~
[...]

Arduino CLI version

Original report

0.27.1

Last verified with

b82a519

Operating system

Windows

Operating system version

10/11

Additional context

See:

https://github.com/stm32duino/wiki/wiki/Customize-build-options-using-build_opt.h

Note that the suggested approach of closing the IDE and opening again no longer works.

Related

• Provide an option for a complete recompile, including any board specific cached code arduino-ide#419
• https://forum.arduino.cc/t/cache-clear-option/1040106

Issue checklist

• I searched for previous requests in the issue tracker
• I verified the feature was still missing when using the nightly build
• My request contains all necessary details

[matthijskooijman]

I believe that arduino-cli compile has a --clean option that would achieve what you need.

However, I also think that the caching code should handle this automatically, e.g. by tracking the include paths used for compiling the core files and noticing that these change when compiling another sketch (this implies that sketches using build_opt.h or otherwise include sketch dir in the core include path can never share cached cores between different sketches, but that's just how it is, I guess).

[Bodmer]

I tried setting:

cache.enable=false
compiler.cache_core=false

in preferences.tft but that did not work so perhaps this setting in now deprecated?

[per1234]

Hi @Bodmer

in preferences.tft

Arduino IDE 2.x uses a completely separate preferences store than Arduino IDE 1.x. The preferences.txt file is not used at all by Arduino IDE 2.x. There is no setting in Arduino IDE 2.x to control caching behavior.

I saw you also posted about this on the Arduino Forum:

https://forum.arduino.cc/t/cache-clear-option/1040106

I'm looking forward to your opinion on the alternative proposal I described in my reply there.

However, I always prefer a solution that allows the caching to always be handled correctly automatically without any need for special action from the user, as matthijskooijman described.

I had transferred this issue from its original location in the arduino/arduino-cli repository to arduino/arduino-ide because the feature you request would only require changes in the Arduino IDE 2.x codebase (since Arduino CLI should already have the necessary capability). But if the solution is to be improving the caching behavior then the work would be done in the arduino/arduino-cli repository after all.

[Bodmer]

Yes, an option would be great.

I have also got a corrupted cache in the past and restarting the IDE used to fix that.

At the moment my work around is to start the IDE with a batch file that clears the cache. But that is also not a clean way to do things. For example:

cd C:\Users\xxxxx\AppData\Local\Programs\Arduino IDE
"Arduino IDE" --clean

Yep, I was investigating CLI options and incorrectly pasted the issue there. Thanks for moving it to the correct place.

[MCF]

In the various discussions around this feature found here and on various forum postings, there is an important and valid use case that is not mentioned - that is the case where compiler warnings are turned up or on for a sketch and then you wish to iteratively clean up the warnings. If there are many warnings across many files it is unlikely that you will want to fix them all at once- better to fix them a few at a time. In this case there is no easy way to force the recompilation of files that have successfully compiled - but with warnings.

I understand the underlying reason that the Arduino IDE is meant to do everything it can automatically (for newer users among others). But I frankly find the reluctance to have an option to do a complete clean of the cache of a sketche's files a bit extreme. I get it - the less a new user has to worry about the better - but being able to rule in or out a caching problem, or the use cxase I mentioned above, would seem to be a good enough thing that it should be available somewhere in the IDE (hide it away in a dark corner, but please make it available :-) ).

FWIW my solution was to turn on the verbose option for compilation (in preferences) then figure out where the sketch's temporary directory was, and then just delete it in between compilations. This is described in more detail in the forum link noted above.

Thanks to everyone who contributes code/effort to keeping the IDE going. Apologies for my lack of any real help other than complaining.

[MCF]

Coincidentally I ran across a related issue in the arduino-builder repo that discusses the use case I noted above. For completeness, here is the issue:

arduino/arduino-builder#301

[thomascallahan]

I just had a similar problem due to renaming some .h and .cpp files in a project. Verify/Compile was still using the old, cached files. Took me a while to notice the old filenames in the error output, and then a bunch of googling trying to find a way to clear the cache (the switch boards, verify, switch back, worked for me).

I'm a relative newcomer to Aruduino, but have been a web developer for 25+ years so this kind of thing isn't exactly new to me. Once I saw the old filenames I immediately knew that there was caching of some kind. I can imagine a true newbie would be extremely confused by this. Having the option to clear the cache seems necessary, and/or having it happen automatically if you rename files in a sketch?

[per1234]

I just had a similar problem due to renaming some .h and .cpp files in a project. Verify/Compile was still using the old, cached files.

This is a known bug: arduino/arduino-ide#970 #1240

This is a case where the development resources would be much better devoted to fixing the bug instead of a adding a cache clear capability to allow the users to do something the build system should be doing automatically.