`arduino_secrets.h` not added to Cloud sketches that contain secrets not defined via the web interface

[per1234]

Describe the problem

Arduino Web Editor has a "Secret tab" feature that makes it easy to share sketches without accidentally exposing sensitive data (e.g., passwords or tokens). Secrets are automatically generated for any identifier in the sketch which starts with SECRET_ and contains all uppercase characters.

The values of these secrets can be defined on Arduino Cloud via a form in the "Secret" tab in Arduino Web Editor.

Secrets are also used in Arduino IoT Cloud Thing sketches to store network credentials. The value of these secrets can be set via the Arduino IoT Cloud web interface.

When you download a sketch containing secrets from Arduino Web Editor (via the ●●● > Download Sketch menu item), the downloaded ZIP file includes a file named arduino_secrets.h that contains #define directives for the secrets.

When you have defined the value of secrets via the Arduino Cloud web interfaces and then pull the sketch via Arduino IDE's Remote Sketchbook, the arduino_secrets.h file is added to the sketch.

🐛 However, if you have not defined values for the secrets via the web interface before pulling, then the sketch will not contain this file.
🐛 The sketch will fail to compile and the average user will find it very difficult to manually add the correct file and code for the missing secrets definitions.

To reproduce

1. Open Arduino Web Editor.
2. Click the New Sketch button.
3. Add the following code:

   char myPassword[] = SECRET_PASSWORD;
   void setup() {}
   void loop() {}

   You should see a tab named "Secret" appear. Do not define the value of SECRET_PASSWORD.
4. Start the Arduino IDE.
5. Open the sketchbook explorer sidebar (folder icon)
6. Click the Remote Sketchbook tab (globe icon)
7. Click the "Sign in" button and sign into your Arduino account if not already.
8. In the Arduino IDE, click the name of the sketch you created in Arduino Web Editor.
9. Click the "Pull" icon (cloud with downward arrow). Wait for the pull to finish.
10. Double click the sketch to open it.
    🐛 The sketch contains only the primary .ino file and ReadMe.adoc.
11. Select Tools > Board > Arduino AVR Boards > Arduino Uno from the Arduino IDE menus.
12. Select Sketch > Verify/Compile from the Arduino IDE menus.
    🐛 The compilation fails:

    Compilation error: 'SECRET_PASSWORD' was not declared in this scope

Expected behavior

When a sketch that uses secrets is pulled, it will always contain:

• An arduino_secrets.h file with #define directives for the secret macros
• An #include directive at the top of the primary sketch file for arduino_secrets.h (include arduino_secrets when needed #438)

Arduino IDE version

Original report

2.0.0-beta.8-snapshot.491fa19

Last verified with

9a6a457

Operating system

Windows

Operating system version

11

Additional context

I can envision two common use cases where secrets are not defined on Arduino Cloud before pulling the sketch:

• The Arduino IoT Cloud web interface is used for the initial setup of a Thing, but the user wants to do all further sketch development in the IDE.
• A copy of a sketch is created in Arduino Web Editor via the ●●● > Save As menu item (secret values are not transferred to the saved copy of the sketch).

---

Additional reports:

• https://forum.arduino.cc/t/cant-run-remote-sketch-locally-with-arduino-ide/1076466
• https://forum.arduino.cc/t/wemos-lolin32-unable-to-verify-ota-the-board-didnt-answer-after-reboot/1087777/5

Issue checklist

• I searched for previous reports in the issue tracker
• I verified the problem still occurs when using the latest nightly build
• My report contains all necessary details

[fstasi]

Addendum: we read secrets from the sketch json file and we create the arduino_secrets.h file manually when secrets exists.

I need to confirm this, but this might require a change on the arduino cloud api so that the "secrets" section in the json is always populated even if secrets are defined but have no assigned value

[kittaakos]

If the value of a secret is not defined, the server is not returning with the sketch.secrets property.

I have two sketches in the Create editor:

alma:

char myPassword[] = SECRET_PASSWORD;
void setup() {}
void loop() {}

No value is defined for the secret

korte:

char myPassword[] = SECRET_PASSWORD;
void setup() {}
void loop() {}

For this sketch, there is a value defined for the secret:

1. List resources in the root:

curl \
  -H "Accept: application/json" \
  -H "Content-Type: application/json" \
  -H "authorization: Bearer $TOKEN" \
  "https://api2.arduino.cc/create/v2/files/d/\$HOME/sketches_v2" | jq .

[
  {
    "children": 2,
    "href": "/create/v2/files/d/8a694e4b83878cc53472bd75ee928053:kittaakos/sketches_v2/alma",
    "modified_at": "2023-01-16T13:46:27.698Z",
    "name": "alma",
    "path": "8a694e4b83878cc53472bd75ee928053:kittaakos/sketches_v2/alma",
    "size": 1576,
    "sketch_info": {
      "id": "7258bf5a-e1ad-45ae-82bc-bfde42071cb7",
      "is_public": false
    },
    "type": "sketch"
  },
  {
    "children": 2,
    "href": "/create/v2/files/d/8a694e4b83878cc53472bd75ee928053:kittaakos/sketches_v2/korte",
    "modified_at": "2023-01-16T13:46:54.140329Z",
    "name": "korte",
    "path": "8a694e4b83878cc53472bd75ee928053:kittaakos/sketches_v2/korte",
    "size": 1576,
    "sketch_info": {
      "id": "8d1c721b-dd40-4ad6-8a95-a725032905f9",
      "is_public": false
    },
    "type": "sketch"
  }
]

2. Get the sketch details, a value for the secret is not defined:

curl \
  -H "Accept: application/json" \
  -H "Content-Type: application/json" \
  -H "authorization: Bearer $TOKEN" \
  "https://api2.arduino.cc/create/v2/sketches/byID/7258bf5a-e1ad-45ae-82bc-bfde42071cb7" | jq .

{
  "board_fqbn": "arduino:avr:ethernet",
  "board_name": "Arduino Ethernet",
  "board_type": "serial",
  "created_at": "2022-10-25T14:02:12.360033Z",
  "href": "/create/v2/files/d/8a694e4b83878cc53472bd75ee928053:kittaakos/sketches_v2/alma",
  "id": "7258bf5a-e1ad-45ae-82bc-bfde42071cb7",
  "is_public": false,
  "libraries": [],
  "modified_at": "2023-01-16T13:46:27.698Z",
  "name": "alma",
  "path": "8a694e4b83878cc53472bd75ee928053:kittaakos/sketches_v2/alma",
  "size": 1723,
  "tutorials": null,
  "types": null,
  "user_id": "b036b0a3-ef0e-44b1-af37-3c756f20fdf0"
}

3. Get the sketch details when the value is defined for the secret:

curl \
  -H "Accept: application/json" \
  -H "Content-Type: application/json" \
  -H "authorization: Bearer $TOKEN" \
  "https://api2.arduino.cc/create/v2/sketches/byID/8d1c721b-dd40-4ad6-8a95-a725032905f9" | jq .

{
  "board_fqbn": "",
  "board_name": "",
  "board_type": "",
  "created_at": "2023-01-16T10:57:24.474541Z",
  "href": "/create/v2/files/d/8a694e4b83878cc53472bd75ee928053:kittaakos/sketches_v2/korte",
  "id": "8d1c721b-dd40-4ad6-8a95-a725032905f9",
  "is_public": false,
  "libraries": [],
  "modified_at": "2023-01-16T13:46:54.140329Z",
  "name": "korte",
  "path": "8a694e4b83878cc53472bd75ee928053:kittaakos/sketches_v2/korte",
  "secrets": [
    {
      "name": "SECRET_PASSWORD",
      "value": "xxx"
    }
  ],
  "size": 1747,
  "tutorials": null,
  "types": null,
  "user_id": "b036b0a3-ef0e-44b1-af37-3c756f20fdf0"
}

The response object contains the secrets if the value is defined. Otherwise, it does not. Without the expected object, I do not know how IDE2 can fix this issue.

[kittaakos]

From the browser's network tab, I can see that the request and response are correct when I leave the secret value empty and the auto-save does the update:

Request:

POST https://api2.arduino.cc/create/v2/sketches/8f7a44c7-98bf-4084-9ad7-5a100cf07ecb

Request payload:

{
  "id": "8f7a44c7-98bf-4084-9ad7-5a100cf07ecb",
  "secrets": {
    "data": [
      {
        "name": "SECRET_PASSWORD",
        "value": ""
      }
    ]
  },
  "libraries": []
}

Response:

{
    "board_fqbn": "",
    "board_name": "",
    "board_type": "",
    "created_at": "2023-02-14T12:35:08.436905Z",
    "href": "/create/v2/files/d/8a694e4b83878cc53472bd75ee928053:kittaakos/sketches_v2/copy_to_cloud",
    "id": "8f7a44c7-98bf-4084-9ad7-5a100cf07ecb",
    "is_public": false,
    "libraries": [],
    "modified_at": "2023-02-14T14:13:47.678774Z",
    "name": "copy_to_cloud",
    "path": "8a694e4b83878cc53472bd75ee928053:kittaakos/sketches_v2/copy_to_cloud",
    "secrets": [
        {
            "name": "SECRET_PASSWORD",
            "value": ""
        }
    ],
    "tutorials": null,
    "types": null,
    "user_id": "b036b0a3-ef0e-44b1-af37-3c756f20fdf0"
}

The problem must be with the GET part.

[kittaakos]

I looked into the server code and did not find anything suspicious. I started the investigation from scratch using https://create.arduino.cc/editor and not curl. The problem is in the frontend part. I sniffed into the XHR/Fetch history in the DevTools' Network tab, and I found the following when my use case was this:

Prerequisite:

• Create a new sketch

Steps:

1. Paste char myPassword[] = SECRET_PASSWORD; to line 4. The Secret tab is visible in the editor after the paste. Switch to the Secret tab.
2. Type x as the value of SECRET_PASSWORD into the <input> field.
3. Delete x from the <input> by pressing the backspace.

---

1. Paste char myPassword[] = SECRET_PASSWORD; to line 4. The Secret tab is visible in the editor after the paste. Switch to the Secret tab.

POST https://api2.arduino.cc/create/v2/sketches/105c241b-30b6-4908-84ae-9b9931e24072

Request payload:

{"id":"105c241b-30b6-4908-84ae-9b9931e24072","libraries":[]}

This is the bug 👆 The editor does not send the secrets.data containing the SECRET_PASSWORD with the default value, which is an empty string (") in the payload.

Response:

{
    "board_fqbn": "arduino:avr:uno",
    "board_name": "Arduino Uno",
    "board_type": "serial",
    "created_at": "2023-02-14T15:46:02.770568Z",
    "href": "/create/v2/files/d/8a694e4b83878cc53472bd75ee928053:kittaakos/sketches_v2/sketch_feb14a",
    "id": "105c241b-30b6-4908-84ae-9b9931e24072",
    "is_public": false,
    "libraries": [],
    "modified_at": "2023-02-14T15:46:30.007601Z",
    "name": "sketch_feb14a",
    "path": "8a694e4b83878cc53472bd75ee928053:kittaakos/sketches_v2/sketch_feb14a",
    "tutorials": null,
    "types": null,
    "user_id": "b036b0a3-ef0e-44b1-af37-3c756f20fdf0"
}

The response is incorrect because the client did not send the secrets.

---

2. Type x as the value of SECRET_PASSWORD into the <input> field.

POST https://api2.arduino.cc/create/v2/sketches/105c241b-30b6-4908-84ae-9b9931e24072

Request payload:

{"id":"105c241b-30b6-4908-84ae-9b9931e24072","secrets":{"data":[{"name":"SECRET_PASSWORD","value":"x"}]},"libraries":[]}

Here the request payload is correct. x is the value of my secret.

Response:

{
    "board_fqbn": "arduino:avr:uno",
    "board_name": "Arduino Uno",
    "board_type": "serial",
    "created_at": "2023-02-14T15:46:02.770568Z",
    "href": "/create/v2/files/d/8a694e4b83878cc53472bd75ee928053:kittaakos/sketches_v2/sketch_feb14a",
    "id": "105c241b-30b6-4908-84ae-9b9931e24072",
    "is_public": false,
    "libraries": [],
    "modified_at": "2023-02-14T15:49:02.276733Z",
    "name": "sketch_feb14a",
    "path": "8a694e4b83878cc53472bd75ee928053:kittaakos/sketches_v2/sketch_feb14a",
    "secrets": [
        {
            "name": "SECRET_PASSWORD",
            "value": "x"
        }
    ],
    "tutorials": null,
    "types": null,
    "user_id": "b036b0a3-ef0e-44b1-af37-3c756f20fdf0"
}

---

3. Delete x from the <input> by pressing the backspace.

POST https://api2.arduino.cc/create/v2/sketches/105c241b-30b6-4908-84ae-9b9931e24072

Request payload:

{"id":"105c241b-30b6-4908-84ae-9b9931e24072","secrets":{"data":[{"name":"SECRET_PASSWORD","value":""}]},"libraries":[]}

This 👆 proves that empty secret values are allowed and correctly handled by the server if the client sends them.

Response:

{
    "board_fqbn": "arduino:avr:uno",
    "board_name": "Arduino Uno",
    "board_type": "serial",
    "created_at": "2023-02-14T15:46:02.770568Z",
    "href": "/create/v2/files/d/8a694e4b83878cc53472bd75ee928053:kittaakos/sketches_v2/sketch_feb14a",
    "id": "105c241b-30b6-4908-84ae-9b9931e24072",
    "is_public": false,
    "libraries": [],
    "modified_at": "2023-02-14T15:50:36.964194Z",
    "name": "sketch_feb14a",
    "path": "8a694e4b83878cc53472bd75ee928053:kittaakos/sketches_v2/sketch_feb14a",
    "secrets": [
        {
            "name": "SECRET_PASSWORD",
            "value": ""
        }
    ],
    "tutorials": null,
    "types": null,
    "user_id": "b036b0a3-ef0e-44b1-af37-3c756f20fdf0"
}

The response contains the empty value for my SECRET_PASSWORD.

[davegarthsimpson]

I will report this in the webide repo