Unexpected behavior with both a packaged platform and an unpackaged platform in user/hardware directory w/ same ID

[obra]

I'm trying to set up and document a workflow for active development of a core, while still being able to build sketches with arduino-cli.

With 0.21 (as well as previous versions, I believe), I'm running into trouble figuring out a reasonable development workflow. Reading through the arduino-cli sketch build documentation, I don't see a way to do what I want or an explanation of what the expected behavior should be.

Because there's no way for arduino-cli to install the required tools for a platform from a core checked out into $(ARDUINO_DIRECTORIES_USER)/hardware, it's been necessary to "prime" the system by installing a prepackaged version of the core with arduino-cli core install $mycore. That will download a copy of the core and resolve compiler and tools dependencies.

After that, I git clone a copy of the HEAD of the core to $(ARDUINO_DIRECTORIES_USER)/hardware

At this point, if I try to use arduino-cli to build a sketch and supply a fqbn, it will use the prepackaged version of the core downloaded with the package manager.

Doc bug: I don't believe there is documentation about what the expected search order is for a core. (Library resolution on the other hand, is now documented incredibly well at https://arduino.github.io/arduino-cli/0.21/sketch-build-process/#dependency-resolution - Whoever wrote those docs has my gratitude. Historically figuring out that dependency resolution was a huge pain. The new docs are just fantastic. Thank you!)

Functionality issue: I believe that if a user has placed a local, unpackaged core in their hardware/ directory, it ought to be used in preference to one downloaded through the boards manager. That said, this would potentially be a breaking change.

I'd be very happy if there were some way to hint to arduino-cli that it should prefer a core in the hardware subdir of the user directory/sketchbook, either in the global config or on the command-line. Alternatively, I'd love it if there were a way to install tools/dependencies for a core checked out into hardware.

Or is there a more 'arduino-ish' way to do this?

Thanks!

[per1234]

Whoever wrote those docs has my gratitude. Historically figuring out that dependency resolution was a huge pain. The new docs are just fantastic. Thank you!

Several people contributed to the effort, but @cmaglie is the original author.

I agree that it is really helpful to have clear documentation of this complex system. Thanks Cristian!

this would potentially be a breaking change.

I don't think it would be very serious. The priorities have never been documented and I think it is not very common to have the same platform installed both places. I mostly see this done by newcomers who accidentally follow both the Boards Manager and manual installation methods because the docs don't make it clear that these are two alternative options (example).

That said, there is a previous report of a change in the priorities causing problems for people here: arduino/Arduino#9237

I'd love it if there were a way to install tools/dependencies for a core checked out into hardware.

I once wished for this also. I believe the ESP8266 platform developers had to go to some lengths to achieve it for their "git" installation method.

The approach of using Boards Manager for the tools installation has served me very well, and is currently in use in the CI systems of all of Arduino's official boards platforms (example). I haven't yet ran into the situation where I needed different tools than those used by a release version of the platform. My plan is to handle this eventuality by creating a development package index with the required tools and a platform specifying them as dependencies, then install that platform via Boards Manager as always.

is there a more 'arduino-ish' way to do this?

The approach taken by Arduino's developers is described here: arduino/Arduino#9237

As you are doing already, Boards Manager is typically used as a convenient way to install the tools.

As you are doing already, the boards platform under development is located under $(ARDUINO_DIRECTORIES_USER)/hardware

The one difference is that you should make the vendor folder unique. In the discussion linked above, that is done by adding a -git suffix to the standard vendor name. For example, the "Arduino AVR Boards" platform would be developed under $(ARDUINO_DIRECTORIES_USER)/hardware/arduino-git/avr. -git is only an arbitrary choice. You are welcome to differentiate the vendor folder name any way you like.

The problem with having a boards platform installed both to $(ARDUINO_DIRECTORIES_DATA)/packages and $(ARDUINO_DIRECTORIES_USER)/hardware is that this allows a name collision of the FQBN. Since the vendor component of the FQBN is determined by the vendor folder name, differentiating the vendor folder name prevents this collision (e.g., arduino:avr:uno vs arduino-git:avr:uno. Once you have unique FQBNs, you can provide Arduino CLI with an explicit directive about which one to use. Unlike the architecture, the vendor name does not often have an effect on the code, so changing the vendor name is unlikely to have side effects.

In a way, this approach is actually better than the development copy overriding the production installation because it means you can easily control which of the two platforms is used by an Arduino CLI command via the FQBN. So you can work with the production platform as needed without having to remember to checkout that tag in the repo every time.

[obra]

@per1234 - Thank you for the incredibly detailed response & pointers to prior art.

For my use case, I can likely make things work with the -git hack, but it sounds like there really is something that ought to get fixed here.

I don't 100% agree that having to resolve a name collision for a FQBN is a 'bad' thing, since it's very similar to how we already treat libraries. (Heck, I can see a world in which it'd be nice to install multiple package-manager-managed versions of one core in parallel.)

Silently choosing the packaged core over a locally defined core feels like the wrong thing.

Possible changes might include:

• Listing all the cores that satisfy that FQBN (including their paths on disk) and identifying which one we chose
  This is what's done with libraries.
• Erroring out if multiple cores satisfy a FQBN
  I'm not a huge fan of "fail when you see ambiguity" and I suspect it's very un-arduinoish. But it'd help make sure that users don't end up in a confusing situation where they keep updating a core and their changes don't take
• Adding support for a "magic" prefix to vendor names that instructs arduino-cli to use a packaged FQBN vs a hardware/ FQBN
  This would, essentially formalize the vendor-git workaround folks are already using. I'm not sure how much it adds, though
• Adding a flag to arduino-cli to specify that hardware/ takes precedence over package-manager installed cores
• Inverting the search order for hardware/ based cores vs package-manager installed cores
  I think that this would result in better default behavior that would help developers working on cores without inconveniencing end users
• Documenting that package-manager installed cores will always silently override locally defined cores of the same name.
  This is the simplest fix for the current situation. I don't love it, but it's certainly an option.

[obra]

One more possibility:

• Adding an arduino-cli core install-deps <corename> that would install a core's dependencies without installing the core itself.
  It doesn't get us the ability to switch back and forth between the production core and the dev core, but it simplifies the dev setup workflow dramatically. I'd thought that I might be able to make it work simply by installing and uninstalling the core, but it looks like the dependency tracking is too good ;)

[obra]

For those playing along at home:

After implementing the -git core naming convention in my build system, it felt pretty ugly and hacky. It was going to mean having to mess with FQBNs all over the place depending on which version of the core happened to be around.

I took a step back and implemented @per1234's development package index idea. It took 5 minutes. The most complicated bit was creating an empty zipfile for the platform. And everything...just works. Here's my development package index: https://github.com/keyboardio/boardsmanager/blob/main/devel/package_kaleidoscope_devel_index.json