`SettingsService#Merge` and `SettingsService#Write` work incorrectly between `daemon` sessions

[kittaakos]

Describe the problem

Originally from arduino/arduino-ide#2185 (review).

When IDE2 sets the network#proxy to manual proxy settings, then unsets it in another daemon session, it does not work. It works perfectly when setting and then unsetting network#proxy within the same daemon session.

To reproduce

You need two shells.

In shell #1:

./arduino-cli version
arduino-cli  Version: 0.34.0 Commit: 304d48cd Date: 2023-08-23T08:01:50Z

cat ./my-cli-config.yaml

board_manager:
  additional_urls:
  - https://www.pjrc.com/teensy/package_issue1588_index.json
build_cache:
  compilations_before_purge: 10
  ttl: 720h0m0s
daemon:
  port: "50051"
directories:
  builtin:
    libraries: /Users/a.kitta/Library/Arduino15/libraries
  data: /Users/a.kitta/Library/Arduino15
  downloads: /Users/a.kitta/Library/Arduino15/staging
  user: /Users/a.kitta/Documents/Arduino
library:
  enable_unsafe_install: false
locale: en
logging:
  file: ""
  format: text
  level: info
metrics:
  addr: :9090
  enabled: true
output:
  no_color: false
sketch:
  always_export_binaries: false
updater:
  enable_notification: true

./arduino-cli daemon --config-file ./my-cli-config.yaml
Daemon is now listening on 127.0.0.1:50051
{"IP":"127.0.0.1","Port":"50051"}

---

In shell 2:

1. Get all config values

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/settings/v1/settings.proto \
  127.0.0.1:50051 \
  cc.arduino.cli.settings.v1.SettingsService.GetAll | jq -r .jsonData | jq .

{
  "board_manager": {
    "additional_urls": [
      "https://www.pjrc.com/teensy/package_issue1588_index.json"
    ]
  },
  "build_cache": {
    "compilations_before_purge": 10,
    "ttl": "720h0m0s"
  },
  "daemon": {
    "port": "50051"
  },
  "directories": {
    "builtin": {
      "libraries": "/Users/a.kitta/Library/Arduino15/libraries"
    },
    "data": "/Users/a.kitta/Library/Arduino15",
    "downloads": "/Users/a.kitta/Library/Arduino15/staging",
    "user": "/Users/a.kitta/Documents/Arduino"
  },
  "library": {
    "enable_unsafe_install": false
  },
  "locale": "en",
  "logging": {
    "file": "",
    "format": "text",
    "level": "info"
  },
  "metrics": {
    "addr": ":9090",
    "enabled": true
  },
  "network": {
    "user_agent_ext": "daemon"
  },
  "output": {
    "no_color": false
  },
  "sketch": {
    "always_export_binaries": false
  },
  "updater": {
    "enable_notification": true
  }
}

2. Compare with content on disk:

cat ./my-cli-config.yaml

board_manager:
  additional_urls:
   - https://www.pjrc.com/teensy/package_issue1588_index.json
build_cache:
  compilations_before_purge: 10
  ttl: 720h0m0s
daemon:
  port: "50051"
directories:
  builtin:
    libraries: /Users/a.kitta/Library/Arduino15/libraries
  data: /Users/a.kitta/Library/Arduino15
  downloads: /Users/a.kitta/Library/Arduino15/staging
  user: /Users/a.kitta/Documents/Arduino
library:
  enable_unsafe_install: false
locale: en
logging:
  file: ""
  format: text
  level: info
metrics:
  addr: :9090
  enabled: true
output:
  no_color: false
sketch:
  always_export_binaries: false
updater:
  enable_notification: true

3. Update network#proxy config value with manual proxy settings

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/settings/v1/settings.proto \
  -d '{"jsonData": "{\n  \"board_manager\": {\n    \"additional_urls\": [\n      \"https://www.pjrc.com/teensy/package_issue1588_index.json\"\n    ]\n  },\n  \"build_cache\": {\n    \"compilations_before_purge\": 10,\n    \"ttl\": \"720h0m0s\"\n  },\n  \"daemon\": {\n    \"port\": \"50051\"\n  },\n  \"directories\": {\n    \"data\": \"/Users/a.kitta/Library/Arduino15\",\n    \"user\": \"/Users/a.kitta/Documents/Arduino\"\n  },\n  \"library\": {\n    \"enable_unsafe_install\": false\n  },\n  \"locale\": \"en\",\n  \"logging\": {\n    \"file\": \"\",\n    \"format\": \"text\",\n    \"level\": \"info\"\n  },\n  \"metrics\": {\n    \"addr\": \":9090\",\n    \"enabled\": true\n  },\n  \"output\": {\n    \"no_color\": false\n  },\n  \"sketch\": {\n    \"always_export_binaries\": false\n  },\n  \"updater\": {\n    \"enable_notification\": true\n  },\n  \"network\": {\n    \"proxy\": \"http://username:secret@hostename:1234/\"\n  }\n}"}' \
  127.0.0.1:50051 \
  cc.arduino.cli.settings.v1.SettingsService.Merge

{
  
}

4. Write config state to disk

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/settings/v1/settings.proto \
  -d '{"filePath": "/Users/a.kitta/dev/git/arduino-cli/my-cli-config.yaml"}' \
  127.0.0.1:50051 \
  cc.arduino.cli.settings.v1.SettingsService.Write

{
  
}

5. Verify network#proxy, manual proxy settings are configured

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/settings/v1/settings.proto \
  127.0.0.1:50051 \
  cc.arduino.cli.settings.v1.SettingsService.GetAll | jq -r .jsonData | jq .

{
  "board_manager": {
    "additional_urls": [
      "https://www.pjrc.com/teensy/package_issue1588_index.json"
    ]
  },
  "build_cache": {
    "compilations_before_purge": 10,
    "ttl": "720h0m0s"
  },
  "daemon": {
    "port": "50051"
  },
  "directories": {
    "builtin": {
      "libraries": "/Users/a.kitta/Library/Arduino15/libraries"
    },
    "data": "/Users/a.kitta/Library/Arduino15",
    "downloads": "/Users/a.kitta/Library/Arduino15/staging",
    "user": "/Users/a.kitta/Documents/Arduino"
  },
  "library": {
    "enable_unsafe_install": false
  },
  "locale": "en",
  "logging": {
    "file": "",
    "format": "text",
    "level": "info"
  },
  "metrics": {
    "addr": ":9090",
    "enabled": true
  },
  "network": {
    "proxy": "http://username:secret@hostename:1234/",
    "user_agent_ext": "daemon"
  },
  "output": {
    "no_color": false
  },
  "sketch": {
    "always_export_binaries": false
  },
  "updater": {
    "enable_notification": true
  }
}

cat ./my-cli-config.yaml

board_manager:
  additional_urls:
  - https://www.pjrc.com/teensy/package_issue1588_index.json
build_cache:
  compilations_before_purge: 10
  ttl: 720h0m0s
daemon:
  port: "50051"
directories:
  builtin:
    libraries: /Users/a.kitta/Library/Arduino15/libraries
  data: /Users/a.kitta/Library/Arduino15
  downloads: /Users/a.kitta/Library/Arduino15/staging
  user: /Users/a.kitta/Documents/Arduino
library:
  enable_unsafe_install: false
locale: en
logging:
  file: ""
  format: text
  level: info
metrics:
  addr: :9090
  enabled: true
network:
  proxy: http://username:secret@hostename:1234/
  user_agent_ext: daemon
output:
  no_color: false
sketch:
  always_export_binaries: false
updater:
  enable_notification: true

---

Terminate the CLI daemon process in shell 1 and restart it with ./arduino-cli daemon --config-file ./my-cli-config.yaml.
❗ The bug does not happen if you do not restart the daemon process but continue with the grpcurl commands in shell 2.

---

In shell 2:

1. Unset network#proxy manual proxy configuration (using an empty object ({}))

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/settings/v1/settings.proto \
  -d '{"jsonData": "{\n  \"board_manager\": {\n    \"additional_urls\": [\n      \"https://www.pjrc.com/teensy/package_issue1588_index.json\"\n    ]\n  },\n  \"build_cache\": {\n    \"compilations_before_purge\": 10,\n    \"ttl\": \"720h0m0s\"\n  },\n  \"daemon\": {\n    \"port\": \"50051\"\n  },\n  \"directories\": {\n    \"data\": \"/Users/a.kitta/Library/Arduino15\",\n    \"user\": \"/Users/a.kitta/Documents/Arduino\"\n  },\n  \"library\": {\n    \"enable_unsafe_install\": false\n  },\n  \"locale\": \"en\",\n  \"logging\": {\n    \"file\": \"\",\n    \"format\": \"text\",\n    \"level\": \"info\"\n  },\n  \"metrics\": {\n    \"addr\": \":9090\",\n    \"enabled\": true\n  },\n  \"output\": {\n    \"no_color\": false\n  },\n  \"sketch\": {\n    \"always_export_binaries\": false\n  },\n  \"updater\": {\n    \"enable_notification\": true\n  },\n  \"network\": {}\n}"}' \
  127.0.0.1:50051 \
  cc.arduino.cli.settings.v1.SettingsService.Merge

{
  
}

2. Write config state to disk

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/settings/v1/settings.proto \
  -d '{"filePath": "/Users/a.kitta/dev/git/arduino-cli/my-cli-config.yaml"}' \
  127.0.0.1:50051 \
  cc.arduino.cli.settings.v1.SettingsService.Write

{
  
}

3. Verify network#proxy, manual proxy is not set (it is 🐛)

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/settings/v1/settings.proto \
  127.0.0.1:50051 \
  cc.arduino.cli.settings.v1.SettingsService.GetAll | jq -r .jsonData | jq .

{
  "board_manager": {
    "additional_urls": [
      "https://www.pjrc.com/teensy/package_issue1588_index.json"
    ]
  },
  "build_cache": {
    "compilations_before_purge": 10,
    "ttl": "720h0m0s"
  },
  "daemon": {
    "port": "50051"
  },
  "directories": {
    "builtin": {
      "libraries": "/Users/a.kitta/Library/Arduino15/libraries"
    },
    "data": "/Users/a.kitta/Library/Arduino15",
    "downloads": "/Users/a.kitta/Library/Arduino15/staging",
    "user": "/Users/a.kitta/Documents/Arduino"
  },
  "library": {
    "enable_unsafe_install": false
  },
  "locale": "en",
  "logging": {
    "file": "",
    "format": "text",
    "level": "info"
  },
  "metrics": {
    "addr": ":9090",
    "enabled": true
  },
  "network": {
    "proxy": "http://username:secret@hostename:1234/",
    "user_agent_ext": "daemon"
  },
  "output": {
    "no_color": false
  },
  "sketch": {
    "always_export_binaries": false
  },
  "updater": {
    "enable_notification": true
  }
}

3. Verify network#proxy by reading the CLI config from the disk, manual proxy is not set (it is 🐛)

cat ./my-cli-config.yaml

board_manager:
  additional_urls:
  - https://www.pjrc.com/teensy/package_issue1588_index.json
build_cache:
  compilations_before_purge: 10
  ttl: 720h0m0s
daemon:
  port: "50051"
directories:
  builtin:
    libraries: /Users/a.kitta/Library/Arduino15/libraries
  data: /Users/a.kitta/Library/Arduino15
  downloads: /Users/a.kitta/Library/Arduino15/staging
  user: /Users/a.kitta/Documents/Arduino
library:
  enable_unsafe_install: false
locale: en
logging:
  file: ""
  format: text
  level: info
metrics:
  addr: :9090
  enabled: true
network:
  proxy: http://username:secret@hostename:1234/
  user_agent_ext: daemon
output:
  no_color: false
sketch:
  always_export_binaries: false
updater:
  enable_notification: true

Expected behavior

SettingsService#Merge and SettingsService#Write behaves deterministically independently how many daemon session I start.

Arduino CLI version

0.34.0

Operating system

macOS

Operating system version

13.5

Additional context

Downstream ref: arduino/arduino-ide#2184

Issue checklist

• I searched for previous reports in the issue tracker
• I verified the problem still occurs when using the nightly build
• My report contains all necessary details

[MatteoPologruto]

Hello @kittaakos, thanks for reporting this!

We actually fixed this issue in PR #2212, introducing a new command SettingsService#Delete. It was necessary to do it because Viper does not allow to unset a default configuration (which in this case is everything that is loaded from my-cli-config.yaml) by replacing it with an empty object.

To delete the network configuration, I would suggest to follow these steps:

Preliminary steps:

arduino-cli version
arduino-cli  Version: 0.34.2 Commit: 963c1a76 Date: 2023-09-11T10:05:42Z

cat ./my-cli-config.yaml

board_manager:
  additional_urls:
  - https://www.pjrc.com/teensy/package_issue1588_index.json
build_cache:
  compilations_before_purge: 10
  ttl: 720h0m0s
daemon:
  port: "50051"
directories:
  builtin:
    libraries: /home/matteo/.arduino15/libraries
  data: /home/matteo/.arduino15
  downloads: /home/matteo/.arduino15/staging
  user: /home/matteo/Arduino
library:
  enable_unsafe_install: false
locale: en
logging:
  file: ""
  format: text
  level: info
metrics:
  addr: :9090
  enabled: true
network:
  proxy: http://username:secret@hostename:1234/
  user_agent_ext: daemon
output:
  no_color: false
sketch:
  always_export_binaries: false
updater:
  enable_notification: true

Start daemon in shell 1 and verify that network#proxy configuration is set in shell 2

arduino-cli daemon --config-file ./my-cli-config.yaml 
Daemon is now listening on 127.0.0.1:50051
{"IP":"127.0.0.1","Port":"50051"}

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/settings/v1/settings.proto \
  127.0.0.1:50051 \
  cc.arduino.cli.settings.v1.SettingsService.GetAll | jq -r .jsonData | jq .

{
  "board_manager": {
    "additional_urls": [
      "https://www.pjrc.com/teensy/package_issue1588_index.json"
    ]
  },
  "build_cache": {
    "compilations_before_purge": 10,
    "ttl": "720h0m0s"
  },
  "daemon": {
    "port": "50051"
  },
  "directories": {
    "builtin": {
      "libraries": "/home/matteo/.arduino15/libraries"
    },
    "data": "/home/matteo/.arduino15",
    "downloads": "/home/matteo/.arduino15/staging",
    "user": "/home/matteo/Arduino"
  },
  "library": {
    "enable_unsafe_install": false
  },
  "locale": "en",
  "logging": {
    "file": "",
    "format": "text",
    "level": "info"
  },
  "metrics": {
    "addr": ":9090",
    "enabled": true
  },
  "network": {
    "proxy": "http://username:secret@hostename:1234/",
    "user_agent_ext": "daemon"
  },
  "output": {
    "no_color": false
  },
  "sketch": {
    "always_export_binaries": false
  },
  "updater": {
    "enable_notification": true
  }
}

---

In shell 2:

1. Delete network using the DeleteRequest {"key": "network"}

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/settings/v1/settings.proto \
  -d '{"key": "network"}' \
  127.0.0.1:50051 \
  cc.arduino.cli.settings.v1.SettingsService.Delete

2. Write config state to disk

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/settings/v1/settings.proto \
  -d '{"filePath": "/Users/a.kitta/dev/git/arduino-cli/my-cli-config.yaml"}' \
  127.0.0.1:50051 \
  cc.arduino.cli.settings.v1.SettingsService.Write

3. Verify network#proxy, manual proxy is not set

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/settings/v1/settings.proto \
  127.0.0.1:50051 \
  cc.arduino.cli.settings.v1.SettingsService.GetAll | jq -r .jsonData | jq .

{
  "board_manager": {
    "additional_urls": [
      "https://www.pjrc.com/teensy/package_issue1588_index.json"
    ]
  },
  "build_cache": {
    "compilations_before_purge": 10,
    "ttl": "720h0m0s"
  },
  "daemon": {
    "port": "50051"
  },
  "directories": {
    "builtin": {
      "libraries": "/home/matteo/.arduino15/libraries"
    },
    "data": "/home/matteo/.arduino15",
    "downloads": "/home/matteo/.arduino15/staging",
    "user": "/home/matteo/Arduino"
  },
  "library": {
    "enable_unsafe_install": false
  },
  "locale": "en",
  "logging": {
    "file": "",
    "format": "text",
    "level": "info"
  },
  "metrics": {
    "addr": ":9090",
    "enabled": true
  },
  "output": {
    "no_color": false
  },
  "sketch": {
    "always_export_binaries": false
  },
  "updater": {
    "enable_notification": true
  }
}

4. Verify network#proxy by reading the CLI config from the disk, manual proxy is not set

cat ./my-cli-config.yaml

board_manager:
  additional_urls:
  - https://www.pjrc.com/teensy/package_issue1588_index.json
build_cache:
  compilations_before_purge: 10
  ttl: 720h0m0s
daemon:
  port: "50051"
directories:
  builtin:
    libraries: /home/matteo/.arduino15/libraries
  data: /home/matteo/.arduino15
  downloads: /home/matteo/.arduino15/staging
  user: /home/matteo/Arduino
library:
  enable_unsafe_install: false
locale: en
logging:
  file: ""
  format: text
  level: info
metrics:
  addr: :9090
  enabled: true
output:
  no_color: false
sketch:
  always_export_binaries: false
updater:
  enable_notification: true

Please let me know if this process works for you or if you have any doubts!

[kittaakos]

introducing a new command SettingsService#Delete.

IDE2 does not know what has changed, that's why the Merge method has been created. If each CLI client has to keep track of what the config diff is, the merge API is useless.