ARDUINO_BOARD_MANAGER_ADDITIONAL_URLS not able to take multiple URLs

[vergissberlin]

Bug Report

Current behavior

In a Docker file or even with a export I tried to add multiple URLs (As the variable name suggests) like so
ENV ARDUINO_BOARD_MANAGER_ADDITIONAL_URLS=[https://dl.espressif.com/dl/package_esp32_index.json,https://arduino.esp8266.com/stable/package_esp8266com_index.json]
But in the end in the yaml file I get this

board_manager:
  additional_urls: '[https://dl.espressif.com/dl/package_esp32_index.json,https://arduino.esp8266.com/stable/package_esp8266com_index.json]'

So, the ENV variable is a string and we are not able to add more than one URL.

Expected behavior

board_manager:
  additional_urls: [https://dl.espressif.com/dl/package_esp32_index.json,https://arduino.esp8266.com/stable/package_esp8266com_index.json]

or

board_manager:
  additional_urls:  
    - https://dl.espressif.com/dl/package_esp32_index.json
    - https://arduino.esp8266.com/stable/package_esp8266com_index.json

Environment

• CLI version (output of arduino-cli version): arduino-cli Version: 0.20.2 Commit: 13783819 Date: 2021-12-09T13:32:05Z
• OS and platform: linux

[vergissberlin]

I just figured out, that it is the same if you try to add more than one url with the --additional-urlsflag:

arduino-cli core update-index \
        --additional-urls=https://arduino.esp8266.com/stable/package_esp8266com_index.json,https://dl.espressif.com/dl/package_esp32_index.json

It is also not possible to use it twice like so:

arduino-cli core update-index \
        --additional-urls=https://arduino.esp8266.com/stable/package_esp8266com_index.json\
        --additional-urls=https://dl.espressif.com/dl/package_esp32_index.json

Error message
when I try to compile

Error initializing instance: Invalid additional URL: parse "[https://dl.espressif.com/dl/package_esp32_index.json,https://arduino.esp8266.com/stable/package_esp8266com_index.json]"

[per1234]

Thanks for your report @vergissberlin.

I tried to add multiple URLs (As the variable name suggests) like so
ENV ARDUINO_BOARD_MANAGER_ADDITIONAL_URLS=[https://dl.espressif.com/dl/package_esp32_index.json,https://arduino.esp8266.com/stable/package_esp8266com_index.json]

The required format for this configuration environment variable is space separated:

ENV ARDUINO_BOARD_MANAGER_ADDITIONAL_URLS="https://dl.espressif.com/dl/package_esp32_index.json https://arduino.esp8266.com/stable/package_esp8266com_index.json"

Documentation deficiency

This exposes a deficiency in the Arduino CLI configuration documentation in that we don't completely document the required format for this configuration key.

It is clearly stated for the --additional-urls flag:

--additional-urls strings   Comma-separated list of additional URLs for the Boards Manager.

and shown in an example command for arduino-cli config set:

arduino-cli config set board_manager.additional_urls https://example.com/package_example_index.json https://another-url.com/package_another_index.json

It is somewhat communicated for the board_manager.additional_urls key of the configuration file in that arduino-cli config init produces a file with an empty array:

board_manager:
  additional_urls: []

But the documentation is entirely silent about the format of the ARDUINO_BOARD_MANAGER_ADDITIONAL_URLS environment variable.

Even though I happen to know that this capability is provided by the Viper Go package (which a user is not expected to know), I didn't even find anything about the format in its documentation.

Possible format change

It seems to me that the most intuitive format for the environment variable is the comma separated format already used by the --additional-urls flag:

ENV ARDUINO_BOARD_MANAGER_ADDITIONAL_URLS=https://dl.espressif.com/dl/package_esp32_index.json,https://arduino.esp8266.com/stable/package_esp8266com_index.json

but this currently does not work:

 0 / 427    0.00%Error updating index: Error downloading index 'https://dl.espressif.com/dl/package_esp32_index.json,https://arduino.esp8266.com/stable/package_esp8266com_index.json': Server responded with: 404 Not Found

I see a related report here: spf13/viper#380

Viper does already provide support for comma separated strings:

https://pkg.go.dev/github.com/spf13/viper#readme-decoding-custom-formats

So, after a quick look it doesn't seem too difficult of a change, but of course there is also the consideration of support for people already using the space separated format.

The space separated format is already used by arduino-cli config set

Other discussion

it is the same if you try to add more than one url with the --additional-urls flag:

This should work just fine.

when I try to compile

I suspect this is your problem. If you still have your old invalid value set via the ARDUINO_BOARD_MANAGER_ADDITIONAL_URLS environment variable, then that will be used for any command that doesn't include the --additional-urls flag. So even if you did a valid arduino-cli core update-index --additional-urls command, the arduino-cli compile command that comes after will try to download an index from the bad URL.

[vergissberlin]

Oh wow, thank you for your detailed answer and the respectful tone! As for the second point, you were indeed right. I still had the environment variable set.
As a result of using --additional-urls in the right way, the config dump looks a bit weird, but it seems to e syntactical right, because it works. Looks like so:

    - urls: https://example.com
       https://example.com

Kind regards,
André

[nuno-silva]

Issue is still present as of arduino-cli Version: 0.29.0 Commit: 76251df Date: 2022-11-17T09:21:40Z

1. env var with spaces works (not documented)

~ # export ARDUINO_BOARD_MANAGER_ADDITIONAL_URLS="https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json https://arduino.esp8266.com/stable/package_esp8266com_index.json"
~ # arduino-cli config init
Config file written to: /root/.arduino15/arduino-cli.yaml
~ # head -n5 /root/.arduino15/arduino-cli.yaml
board_manager:
  additional_urls: https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json
    https://arduino.esp8266.com/stable/package_esp8266com_index.json
daemon:
  port: "50051"
~ # arduino-cli core update-index
Downloading index: library_index.tar.bz2 downloaded                                                                                                 
Downloading index: package_index.tar.bz2 downloaded                                                                                                 
Downloading index: package_index.tar.bz2 downloaded                                                                                                 
Downloading index: package_esp32_index.json downloaded                                                                                              
Downloading index: package_esp8266com_index.json downloaded

2. env var with commas does not work

~ # export ARDUINO_BOARD_MANAGER_ADDITIONAL_URLS="https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json,https://arduino.esp8266.com/stable/package_esp8266com_index.json"
~ # arduino-cli config init
Config file written to: /root/.arduino15/arduino-cli.yaml
~ # head -n5 /root/.arduino15/arduino-cli.yaml
board_manager:
  additional_urls: https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json,https://arduino.esp8266.com/stable/package_esp8266com_index.json
daemon:
  port: "50051"
directories:
~ # arduino-cli core update-index
Downloading index: package_index.tar.bz2 downloaded                                                                                                                                                                                            
Downloading index: package_esp8266com_index.json Server responded with: 404 Not Found                                                                                                                                                          
Some indexes could not be updated.

3. --additional-urls with commas (as mentioned in docs) works and produces yaml list

~ # arduino-cli config --additional-urls "https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json,https://arduino.esp8266.com/stable/package_esp8266com_index.json" init
Config file written to: /root/.arduino15/arduino-cli.yaml
~ # head -n5 /root/.arduino15/arduino-cli.yaml
board_manager:
  additional_urls:
  - https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json
  - https://arduino.esp8266.com/stable/package_esp8266com_index.json
daemon:
~ # arduino-cli core update-index
Downloading index: package_index.tar.bz2 downloaded                                                                                                                                                                                            
Downloading index: package_esp32_index.json downloaded                                                                                                                                                                                         
Downloading index: package_esp8266com_index.json downloaded

4. --additional-urls with spaces does not work (expected, since docs mention commas)

~ # arduino-cli config --additional-urls "https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json https://arduino.esp8266.com/stable/package_esp8266com_index.json" init
Config file written to: /root/.arduino15/arduino-cli.yaml
~ # head -n5 /root/.arduino15/arduino-cli.yaml
board_manager:
  additional_urls:
  - https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json
    https://arduino.esp8266.com/stable/package_esp8266com_index.json
daemon:
~ # arduino-cli core update-index
Downloading index: package_index.tar.bz2 downloaded                                                                                                                                                                                            
Downloading index: package_esp8266com_index.json Server responded with: 404 Not Found                                                                                                                                                          
Some indexes could not be updated.