Incorrect compiler selected by Arduino-cli

[zaphodbe]

Describe the problem

Arduino-cli version 0.27.1 and previous versions. Have both Arduino core and a custom SAMD based core installed. Arduino core has arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-gcc and the SAMD based core has things/tools/arm-none-eabi-gcc/9-2019q4/bin/arm-none-eabi-gcc. On each successive run of Arduino-cli it will randomly select which one of those compilers to use. Similar thing happens for bossac.

I added some debug output to the Arduino-cli code in setup_build_properties.go around line 99 ... here

	for _, tool := range ctx.AllTools {
                println("ctx.AllTools: tool:",tool.Tool.Name,"  ver:",tool.Version.String());
		buildProperties.SetPath("runtime.tools."+tool.Tool.Name+".path", tool.InstallDir)
                buildProperties.SetPath("runtime.tools."+tool.Tool.Name+".path2", tool.InstallDir)
		buildProperties.SetPath("runtime.tools."+tool.Tool.Name+"-"+tool.Version.String()+".path", tool.InstallDir)
	}
	for _, tool := range ctx.RequiredTools {
                println("ctx.RequiredTools: tool:",tool.Tool.Name,"  ver:",tool.Version.String());
		buildProperties.SetPath("runtime.tools."+tool.Tool.Name+".path", tool.InstallDir)
                buildProperties.SetPath("runtime.tools."+tool.Tool.Name+".path3", tool.InstallDir)
		buildProperties.SetPath("runtime.tools."+tool.Tool.Name+"-"+tool.Version.String()+".path", tool.InstallDir)
	}

Basically what appears to be happening is that the AllTools and RequiredTools arrays are created non-deterministically and so depending on the order of entries in those arrays depends on which version of the tool is ultimately selected. With the one from RequiredTools finally taking dominance.

This is the output of Arduino-cli with the println's shown and the output grep'd for the .path variables from 2 consecutive runs...

% arduino-cli -v compile --clean ... --show-properties | grep runtime.tools.arm-none-eabi-gcc
ctx.AllTools: tool: avr-gcc   ver: 7.3.0-atmel3.6.1-arduino7
ctx.AllTools: tool: avrdude   ver: 6.3.0-arduino17
ctx.AllTools: tool: CMSIS-Atmel   ver: 1.2.0
ctx.AllTools: tool: openocd   ver: 0.10.0-arduino7
ctx.AllTools: tool: arduinoOTA   ver: 1.3.0
ctx.AllTools: tool: arduinoOTA   ver: 1.2.1
ctx.AllTools: tool: arm-none-eabi-gcc   ver: 4.8.3-2014q1
ctx.AllTools: tool: bossac   ver: 1.8.0-48-gb176eee
ctx.AllTools: tool: bossac   ver: 1.7.0-arduino3
ctx.AllTools: tool: ctags   ver: 5.8-arduino11
ctx.AllTools: tool: mdns-discovery   ver: 1.0.5
ctx.AllTools: tool: mdns-discovery   ver: 1.0.6
ctx.AllTools: tool: serial-discovery   ver: 1.3.2
ctx.AllTools: tool: serial-monitor   ver: 0.9.1
ctx.AllTools: tool: STM32Tools   ver: 2.1.1
ctx.AllTools: tool: arm-none-eabi-gcc   ver: 9-2019q4
ctx.AllTools: tool: bossac   ver: 1.9.1-thing
ctx.AllTools: tool: CMSIS   ver: 5.4.0
ctx.RequiredTools: tool: openocd   ver: 0.10.0-arduino7
ctx.RequiredTools: tool: arduinoOTA   ver: 1.3.0
ctx.RequiredTools: tool: mdns-discovery   ver: 1.0.6
ctx.RequiredTools: tool: serial-discovery   ver: 1.3.2
ctx.RequiredTools: tool: serial-monitor   ver: 0.9.1
ctx.RequiredTools: tool: arm-none-eabi-gcc   ver: 4.8.3-2014q1
ctx.RequiredTools: tool: CMSIS   ver: 5.4.0
ctx.RequiredTools: tool: CMSIS-Atmel   ver: 1.2.0
ctx.RequiredTools: tool: ctags   ver: 5.8-arduino11
ctx.RequiredTools: tool: bossac   ver: 1.8.0-48-gb176eee
ctx.RequiredTools: tool: STM32Tools   ver: 2.1.1
ctx.RequiredTools: tool: avr-gcc   ver: 7.3.0-atmel3.6.1-arduino7
ctx.RequiredTools: tool: avrdude   ver: 6.3.0-arduino17
compiler.path={runtime.tools.arm-none-eabi-gcc.path}/bin/
runtime.tools.arm-none-eabi-gcc-4.8.3-2014q1.path=~/Library/Arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1
runtime.tools.arm-none-eabi-gcc-9-2019q4.path=~/Library/Arduino15/packages/things/tools/arm-none-eabi-gcc/9-2019q4
runtime.tools.arm-none-eabi-gcc.path=~/Library/Arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1
runtime.tools.arm-none-eabi-gcc.path2=~/Library/Arduino15/packages/things/tools/arm-none-eabi-gcc/9-2019q4
runtime.tools.arm-none-eabi-gcc.path3=~/Library/Arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1

% arduino-cli -v compile --clean ... --show-properties | grep runtime.tools.arm-none-eabi-gcc
ctx.AllTools: tool: arm-none-eabi-gcc   ver: 9-2019q4
ctx.AllTools: tool: bossac   ver: 1.9.1-thing
ctx.AllTools: tool: CMSIS   ver: 5.4.0
ctx.AllTools: tool: STM32Tools   ver: 2.1.1
ctx.AllTools: tool: arduinoOTA   ver: 1.2.1
ctx.AllTools: tool: arduinoOTA   ver: 1.3.0
ctx.AllTools: tool: arm-none-eabi-gcc   ver: 4.8.3-2014q1
ctx.AllTools: tool: avr-gcc   ver: 7.3.0-atmel3.6.1-arduino7
ctx.AllTools: tool: bossac   ver: 1.8.0-48-gb176eee
ctx.AllTools: tool: bossac   ver: 1.7.0-arduino3
ctx.AllTools: tool: CMSIS-Atmel   ver: 1.2.0
ctx.AllTools: tool: avrdude   ver: 6.3.0-arduino17
ctx.AllTools: tool: openocd   ver: 0.10.0-arduino7
ctx.AllTools: tool: ctags   ver: 5.8-arduino11
ctx.AllTools: tool: mdns-discovery   ver: 1.0.6
ctx.AllTools: tool: mdns-discovery   ver: 1.0.5
ctx.AllTools: tool: serial-discovery   ver: 1.3.2
ctx.AllTools: tool: serial-monitor   ver: 0.9.1
ctx.RequiredTools: tool: bossac   ver: 1.9.1-thing
ctx.RequiredTools: tool: avrdude   ver: 6.3.0-arduino17
ctx.RequiredTools: tool: mdns-discovery   ver: 1.0.6
ctx.RequiredTools: tool: serial-discovery   ver: 1.3.2
ctx.RequiredTools: tool: serial-monitor   ver: 0.9.1
ctx.RequiredTools: tool: ctags   ver: 5.8-arduino11
ctx.RequiredTools: tool: arm-none-eabi-gcc   ver: 9-2019q4
ctx.RequiredTools: tool: avr-gcc   ver: 7.3.0-atmel3.6.1-arduino7
ctx.RequiredTools: tool: CMSIS-Atmel   ver: 1.2.0
ctx.RequiredTools: tool: openocd   ver: 0.10.0-arduino7
ctx.RequiredTools: tool: arduinoOTA   ver: 1.3.0
ctx.RequiredTools: tool: CMSIS   ver: 5.4.0
ctx.RequiredTools: tool: STM32Tools   ver: 2.1.1
compiler.path={runtime.tools.arm-none-eabi-gcc.path}/bin/
runtime.tools.arm-none-eabi-gcc-4.8.3-2014q1.path=~/Library/Arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1
runtime.tools.arm-none-eabi-gcc-9-2019q4.path=~/Library/Arduino15/packages/things/tools/arm-none-eabi-gcc/9-2019q4
runtime.tools.arm-none-eabi-gcc.path=~/Library/Arduino15/packages/things/tools/arm-none-eabi-gcc/9-2019q4
runtime.tools.arm-none-eabi-gcc.path2=~/Library/Arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1
runtime.tools.arm-none-eabi-gcc.path3=~/Library/Arduino15/packages/things/tools/arm-none-eabi-gcc/9-2019q4

To reproduce

arduino-cli -v compile --fqbn things:samd: file.ino --show-properties

Expected behavior

Correct compiler and bossac should be selected.

Arduino CLI version

0.27.1

Operating system

macOS

Operating system version

12.5

Additional context

No response

Issue checklist

• I searched for previous reports in the issue tracker
• I verified the problem still occurs when using the nightly build
• My report contains all necessary details

[matthijskooijman]

Thanks for reporting this. I've taken the liberty to fix some formatting in your post (replaced single backticks with triple backticks) and added a link to the code you mentioned.

I have been seeing this same problem with upload tools on a custom-installed (from git) STM32 core, that randomly selects upload tools from multiple different stm32 cores I installed through the board/core manager. I think the problem (at least in my case) is that cores installed into ~/.arduino15 using arduino-cli have a json file describing their dependencies, so arduino-cli knows exactly which version of all tools they need. However, for locally installed cores (in my sketchbook), there is no such dependency mapping, so arduino-cli uses an arbitrary version (in terms of code, I think RequiredTools will empty in this case). The lack of (a way to express) this explicit dependency is a problem, but the non-determinism is a problem on its own (and should probably be the subject of this issue).

One question is of course how to solve this non-determinism, i.e. which version should be preferred? I suspect that the highest version number makes the most sense, even though that is not perfect for all situations.

The code you pointed out indeed seems likely the cause of this problem, and I suspect that would be also be the right place to fix it, probably by just sorting (or iterating) AllTools by increasing version number. I cannot speak for the devs, but I would think this might indeed be the correct fix for this.

[zaphodbe]

The correct answer is that it should only be using the tool and version as specified in the platform.txt and package_..._index.json for that specific board. Such that different versions of compiler or bossac etc can be specified for specific boards or MCUs.

[per1234]

a custom SAMD based core installed

How was it installed?

Did you use arduino-cli core install?

Or did you install it manually to <directories.user>/hardware?

The lack of (a way to express) this explicit dependency is a problem

There is to some extent:

https://arduino.github.io/arduino-cli/dev/platform-specification/#tools

{runtime.tools.TOOL_NAME-TOOL_VERSION.path}

If @zaphodbe uses a reference to the runtime.tools.arm-none-eabi-gcc-9-2019q4.path property in the compilation command patterns, the problem of the arduino:arm-none-eabi-gcc@4.8.3-2014q1 tool being used should not occur.

Unfortunately, the poor design of this property format does still leave room for ambiguity. The flaw is that it does not allow specifying the package name. So if two packages contain an arm-none-eabi-gcc@9-2019q4 tool then the property alone does not give control over which of the two tools will be used by the build system.

it should only be using the tool and version as specified in the platform.txt and package_..._index.json

Please provide the platform.txt and package index files of your custom SAMD boards platform @zaphodbe

[zaphodbe]

To @per1234 questions.

It was installed and updated using the Arduino IDE and I have done a fresh install and problem persists.
Also just for giggles I tried a manual install and got same issue.

To your point I tried specifying a specific version runtime.tools.arm-none-eabi-gcc-9-2019q4.path although this isn't the best workaround will help resolve for the moment.

I did some more digging in the arduino-cli code and have determined that the RequiredTools only uses board specific settings from the boards.txt file and never uses the platform settings from platform.txt as such it defaults to entries from AllTools dependent on the order they were found in.

It seems to me that the dependency order should be boards.txt with highest priority then platform.txt and finally defaults.

Please find attached the platform.txt and package index files (with some redactions)

platform.txt
package_index.json.txt

[matthijskooijman]

Hm, seems my previous comment was quite off, so probably just ignore that (the problem I describe there, for self-installed cores, probably does exist, but reading your issue better, your issue is different since your core seems to be installed using arduino-cli/board manager using a json file, not self-installed).

About your issue - looking at the debug output, I believe that for some reason both runs have a different arm-gcc version in the RequiredTools for some reason, e.g.:

ctx.RequiredTools: tool: arm-none-eabi-gcc ver: 4.8.3-2014q1
ctx.RequiredTools: tool: arm-none-eabi-gcc ver: 9-2019q4

Given the code you highlighted, it seems to me that the problem is not that arm-none-eabi-gcc is in AllTools twice in arbitrary order, but that the version present in RequiredTools is different on different runs. What is also weird, is that RequiredTools has a lot of other tools, like avr-gcc and avrdude, which are not mentioned in the JSON toolDependencies at all (and I suspect that's what RequiredTools is based on, but I might be wrong).

I did some more digging in the arduino-cli code and have determined that the RequiredTools only uses board specific settings from the boards.txt file and never uses the platform settings from platform.txt as such it defaults to entries from AllTools dependent on the order they were found in.

Note sure I understand what you are saying here. AFAIU boards.txt or platform.txt do not reference tool versions at all (except when you explicitly use versioned variables like the workaround you suggested) and I think neither would be a source of RequiredTools at all? Did you maybe mean the package JSON here or so?

[zaphodbe]

@matthijskooijman correct the RequiredTools is created by the FindToolsRequiredForBoard method in package_manager.go by first going through the packages list adding each tool once based on the tool name. So depending on the order from iterating the packages depends on which version gets added to the RequiredTools list. Hence not always picking the correct version.

Then that list is overridden by the board settings. However in testing I never see anything from the board settings and it is not getting the platform settings. To debug I added some output to each of the loops in the FindToolsRequiredForBoard method and find that the platform.ToolDependencies list is empty. This list comes from the platform := board.PlatformRelease which looks like it is pulling Info from the boards.txt hence my previous comment. But still digging in that direction to find out more.

Either way platform.ToolDependencies is empty hence just using the defaults which are not in predictable order.

[cmaglie]

@zaphodbe I've implemented a more deterministic tool-selection algorithm here: #1887

Does it makes things better in your case?