Generated compilation database compiler argument missing extension breaks compilation on Windows.

[Falven]

Bug Report

The generated compilation database has a bug/is missing some compiler commands.
If you are trying to use the compilation database as intellisense configuration as is done in pretty much any IDE, for example, clangd under VSCode, the generated compile_commands.json MUST explicity include the system header include directories otherwise clangd or other intellisense engines cannot find them. This is because Arduino uses different compilers in non-standard locations, with different system includes per compiler, as such the intellisense engines will not know where to find the system includes.

Current behavior

Here is an example file entry for the generated compilation database:

{
  "directory": "c:\\Users\\falve\\.vscode\\extensions\\vsciot-vscode.vscode-arduino-0.4.2\\out\\src",
  "arguments": [
   "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4/bin/arm-none-eabi-g++",
   "-mcpu=cortex-m0plus",
   "-mthumb",
   "-c",
   "-g",
   "-Os",
   "-w",
   "-std=gnu++11",
   "-ffunction-sections",
   "-fdata-sections",
   "-fno-threadsafe-statics",
   "-nostdlib",
   "--param",
   "max-inline-insns-single=500",
   "-fno-rtti",
   "-fno-exceptions",
   "-MMD",
   "-DF_CPU=48000000L",
   "-DARDUINO=10607",
   "-DARDUINO_SAMD_MKRWIFI1010",
   "-DARDUINO_ARCH_SAMD",
   "-DUSE_ARDUINO_MKR_PIN_LAYOUT",
   "-D__SAMD21G18A__",
   "-DUSB_VID=0x2341",
   "-DUSB_PID=0x8054",
   "-DUSBCON",
   "-DUSB_MANUFACTURER=\"Arduino LLC\"",
   "-DUSB_PRODUCT=\"Arduino MKR WiFi 1010\"",
   "-DUSE_BQ24195L_PMIC",
   "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\CMSIS\\4.5.0/CMSIS/Include/",
   "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\CMSIS-Atmel\\1.2.0/CMSIS/Device/ATMEL/",
   "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino/api/deprecated",
   "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino/api/deprecated-avr-comp",
   "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino",
   "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\variants\\mkrwifi1010",
   "c:\\Users\\falve\\OneDrive\\Documents\\Arduino\\libraries\\oprintstream\\build\\sketch\\oprintstream.ino.cpp",
   "-o",
   "c:\\Users\\falve\\OneDrive\\Documents\\Arduino\\libraries\\oprintstream\\build\\sketch\\oprintstream.ino.cpp.o"
  ],
  "file": "c:\\Users\\falve\\OneDrive\\Documents\\Arduino\\libraries\\oprintstream\\build\\sketch\\oprintstream.ino.cpp"
 }

Expected behavior

Here is what the file entry from the compilation database should look like:

{
  "directory": "c:\\Users\\falve\\.vscode\\extensions\\vsciot-vscode.vscode-arduino-0.4.2\\out\\src",
  "arguments": [
   "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4/bin/arm-none-eabi-g++",
   "-mcpu=cortex-m0plus",
   "-mthumb",
   "-c",
   "-g",
   "-Os",
   "-w",
   "-std=gnu++11",
   "-ffunction-sections",
   "-fdata-sections",
   "-fno-threadsafe-statics",
   "-nostdlib",
   "--param",
   "max-inline-insns-single=500",
   "-fno-rtti",
   "-fno-exceptions",
   "-MMD",
   "-DF_CPU=48000000L",
   "-DARDUINO=10607",
   "-DARDUINO_SAMD_MKRWIFI1010",
   "-DARDUINO_ARCH_SAMD",
   "-DUSE_ARDUINO_MKR_PIN_LAYOUT",
   "-D__SAMD21G18A__",
   "-DUSB_VID=0x2341",
   "-DUSB_PID=0x8054",
   "-DUSBCON",
   "-DUSB_MANUFACTURER=\"Arduino LLC\"",
   "-DUSB_PRODUCT=\"Arduino MKR WiFi 1010\"",
   "-DUSE_BQ24195L_PMIC",
   "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\CMSIS\\4.5.0/CMSIS/Include/",
   "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\CMSIS-Atmel\\1.2.0/CMSIS/Device/ATMEL/",
   "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino/api/deprecated",
   "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino/api/deprecated-avr-comp",
   "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino",
   "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\variants\\mkrwifi1010",
   "-isystem C:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\arm-none-eabi\\include\\c++\\7.2.1",
   "-isystem C:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\arm-none-eabi\\include\\c++\\7.2.1\\arm-none-eabi",
   "-isystem C:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\arm-none-eabi\\include\\c++\\7.2.1\\backward",
   "-isystem C:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\lib\\gcc\\arm-none-eabi\\7.2.1\\include",
   "-isystem C:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\lib\\gcc\\arm-none-eabi\\7.2.1\\include-fixed",
   "-isystem C:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\arm-none-eabi\\include",
   "c:\\Users\\falve\\OneDrive\\Documents\\Arduino\\libraries\\oprintstream\\build\\sketch\\oprintstream.ino.cpp",
   "-o",
   "c:\\Users\\falve\\OneDrive\\Documents\\Arduino\\libraries\\oprintstream\\build\\sketch\\oprintstream.ino.cpp.o"
  ],
  "file": "c:\\Users\\falve\\OneDrive\\Documents\\Arduino\\libraries\\oprintstream\\build\\sketch\\oprintstream.ino.cpp"
 }

Environment

$ arduino-cli version
arduino-cli.exe alpha Version: 0.18.1 Commit: b3cf8e19 Date: 2021-04-13T13:08:30Z

Additional context

As an example, look at the compilation database, c_cpp_properties.json, that the Arduino vscode extension generates for it's intellisense configuration. They add on the system include directories.

{
    "version": 4,
    "configurations": [
        {
            "name": "Arduino",
            "compilerPath": "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\bin\\arm-none-eabi-g++",
            "compilerArgs": [
                "-mcpu=cortex-m0plus",
                "-mthumb",
                "-w",
                "-std=gnu++11",
                "-ffunction-sections",
                "-fdata-sections",
                "-fno-threadsafe-statics",
                "-nostdlib",
                "--param",
                "-fno-rtti",
                "-fno-exceptions"
            ],
            "intelliSenseMode": "gcc-x64",
            "includePath": [
                "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\CMSIS\\4.5.0\\CMSIS\\Include\\",
                "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\CMSIS-Atmel\\1.2.0\\CMSIS\\Device\\ATMEL\\",
                "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino\\api\\deprecated",
                "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino\\api\\deprecated-avr-comp",
                "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino",
                "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\variants\\mkrwifi1010",
                "c:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\arm-none-eabi\\include\\c++\\7.2.1",
                "c:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\arm-none-eabi\\include\\c++\\7.2.1\\arm-none-eabi",
                "c:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\arm-none-eabi\\include\\c++\\7.2.1\\backward",
                "c:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\lib\\gcc\\arm-none-eabi\\7.2.1\\include",
                "c:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\lib\\gcc\\arm-none-eabi\\7.2.1\\include-fixed",
                "c:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\arm-none-eabi\\include"
            ],
            "forcedInclude": [
                "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino\\Arduino.h"
            ],
            "cStandard": "c11",
            "cppStandard": "c++11",
            "defines": [
                "F_CPU=48000000L",
                 ...
        }
    ]
}

[matthijskooijman]

Hm, one other solution to this (which I use), is to whitelist the Arduino compilers with clangd, e.g.:

clangd --query-driver="$HOME/.arduino15/packages/*/tools/**/*-gcc,$HOME/.arduino15/packages/*/tools/**/*-g++"

This allows clangd to query the compiler for its internal include paths directly. Maybe a bit hardcoded, but it works well in practice.

As for adding the include paths to the compilation database, I wonder if this is really common practice? The json format lacks a way to distinguish these includes from the actual include paths passed to the compiler, making the command listed in the database not the actual command that was ran. I guess that even if clangd manages to query the compiler, the worst case would be a duplicate system include path, which should be harmless, but still.

Also, I'm not entirely sure how we can do this portably and reliably: arduino-cli does not necessarily know how a compiler commandline looks exactly (it might not even be gcc...). There's a few challenges I can see:

• What command to run to figure out the system includes (and maybe defines)? This probably requires a new recipe, since autogenerating this from the recipe for a given compilation might be fragile (I can imagine that options like -mcpu or so might infliuence the list, so you probably cannot just run the plain gcc ommand).
• Should you run gcc a second time to generate includes (and maybe defines) again for every compiled file? You probably cannot assume that the list is the same for every compiled file (might differ between C and C++ and depending on the platform it could even be different for each file).
• Where in the command listed in the compilation database should the extra include paths be inserted? I can imagine that ordering can be relevant, especially if there are explicit -isystem options in the recipe too.

So I wonder if we can do this reliably at all, or if we should just stick to listing the actual commands that were ran and rely on clangd to figure out the rest with --query-driver...

[matthijskooijman]

I also wonder: You paste a c_cpp_properties.json file from vscode, but how is that file used? Does clangd understand it? Or is it input to vscode (seems like, from a quick web search)? Is your suggestion maybe that arduino-cli should generate a c_cpp_properties.json file in addition to the compile_commands.json maybe? If so, do you know if the format of that file is documented anywhere?

[Falven]

Hm, one other solution to this (which I use), is to whitelist the Arduino compilers with clangd, e.g.:

clangd --query-driver="$HOME/.arduino15/packages/*/tools/**/*-gcc,$HOME/.arduino15/packages/*/tools/**/*-g++"

This allows clangd to query the compiler for its internal include paths directly. Maybe a bit hardcoded, but it works well in practice.

If this works, I agree this would be the optimal solution as the work is being done from the compiler. It's unfortunate that this isn't something easily configurable/you have to know about it.

And I realize that this may be off-topic at this point, but it could potentially help another person trying to use clangd with VSCode and Arduino. I am not able to get this to work under Windows 10 x64. I have my .vscode/settings.json:

"clangd.arguments": [
    "--query-driver=\"%LOCALAPPDATA%\\Arduino15\\packages\\*\\tools\\**\\*-gcc.exe,%LOCALAPPDATA%\\Arduino15\\packages\\*\\tools\\**\\*-g++.exe\""
],

And it does not work. I suspect it may have something to do with the generated compile_commands.json paths mixing forward and backslashes:

{
  "directory": "c:\\Users\\falve\\.vscode\\extensions\\vsciot-vscode.vscode-arduino-0.4.2\\out\\src",
  "arguments": [
   "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4/bin/arm-none-eabi-g++",
   ...

I also tried with the command you pasted:

"clangd.arguments": [
    "--query-driver=\"%LOCALAPPDATA%/.arduino15/packages/*/tools/**/*-gcc,$HOME/.arduino15/packages/*/tools/**/*-g++\""
],

As for adding the include paths to the compilation database, I wonder if this is really common practice? The json format lacks a way to distinguish these includes from the actual include paths passed to the compiler, making the command listed in the database not the actual command that was ran. I guess that even if clangd manages to query the compiler, the worst case would be a duplicate system include path, which should be harmless, but still.

That is true, I do think that this functionality is as you said part of the compiler. That would make more sense.

Also, I'm not entirely sure how we can do this portably and reliably: arduino-cli does not necessarily know how a compiler commandline looks exactly (it might not even be gcc...). There's a few challenges I can see:

• What command to run to figure out the system includes (and maybe defines)? This probably requires a new recipe, since autogenerating this from the recipe for a given compilation might be fragile (I can imagine that options like -mcpu or so might infliuence the list, so you probably cannot just run the plain gcc ommand).
• Should you run gcc a second time to generate includes (and maybe defines) again for every compiled file? You probably cannot assume that the list is the same for every compiled file (might differ between C and C++ and depending on the platform it could even be different for each file).
• Where in the command listed in the compilation database should the extra include paths be inserted? I can imagine that ordering can be relevant, especially if there are explicit -isystem options in the recipe too.

So I wonder if we can do this reliably at all, or if we should just stick to listing the actual commands that were ran and rely on clangd to figure out the rest with --query-driver...

I also wonder: You paste a c_cpp_properties.json file from vscode, but how is that file used? Does clangd understand it? Or is it input to vscode (seems like, from a quick web search)? Is your suggestion maybe that arduino-cli should generate a c_cpp_properties.json file in addition to the compile_commands.json maybe? If so, do you know if the format of that file is documented anywhere?

The c_cpp_properties.json file is generated by the VSCode Arduino extension and is used by the VSCode C/C++ extension for intellisense and formatting. I believe the VSCode C/C++ extension uses fuzzy parsing which gives a lot of false results.

I personally much prefer the clangd extension for VS Code as I am used to it, it has arguably better, more modern intellisense, and also includes clang-format and clang-tidy. the clangd extension for VS Code does not interact with the c_cpp_properties.json file at all. It just uses the clangd CLI and the generated compile_commands.json.

The c_cpp_properties.json file is only generated for the VSCode C/C++ extension by the VSCode Arduino extension.
Whether the CLI should generate that file I think would be a topic of discussion with the folks supporting the VSCode Arduino extension.

[Falven]

I created a barebones VS Code project here to test the driver configuration. In this project you would see that clangd intellisense says Use of undeclared identifier 'std'clang(undeclared_var_use) because it does not know where to find <type_traits>.

And I can see in the clangd output that the --query-driver argument is being passed in:

I[14:00:01.840] argv[0]: c:\Users\falve\AppData\Roaming\Code\User\globalStorage\llvm-vs-code-extensions.vscode-clangd\install\12.0.0\clangd_12.0.0\bin\clangd.exe
I[14:00:01.840] argv[1]: --query-driver="%LOCALAPPDATA%\Arduino15\packages\*\tools\**\*-gcc.exe,%LOCALAPPDATA%\Arduino15\packages\*\tools\**\*-g++.exe"
I[14:00:01.872] Loaded compilation database from c:\Users\falve\OneDrive\Desktop\clangd-arduino\build/compile_commands.json
I[14:00:01.876] Loaded compilation database from c:\Users\falve\OneDrive\Desktop\clangd-arduino\build\compile_commands.json

I added --log=verbose and am getting some more informative output now:
When pasting the full path to the executable, I get the following error message, confirming my suspicions about the format of the generated compile_commands.json path separators.

E[14:14:21.550] System include extraction: C:\Users\falve\AppData\Local\Arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4/bin/arm-none-eabi-g++ does not exist.

Yep, when I correct the last few separators:

"--query-driver=C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\bin\\arm-none-eabi-gcc.exe,C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\bin\\arm-none-eabi-g++.exe"

clangd will output:

V[14:17:51.042] System include extraction: not allowed driver C:\Users\falve\AppData\Local\Arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4/bin/arm-none-eabi-g++

Now the question is: "Is the CLI the one generating these incorrect paths in the Windows compilation database or is it the VSCode Arduino extension after it calls the CLI?"

Oh @matthijskooijman! I think that you can either specify the driver in the compile_commands.json, which would take the manual configuration guesswork away from the developer, or you already are, but because the paths are messed up in Windows, it does not use that compiler as the driver. See the second to last comment from kadircet in this thread.

I have confirmed that calling the CLI directly generates these incorrect paths (It is not the Arduino extension doing this):

$ arduino-cli.exe compile --build-path "./build" --fqbn "arduino:samd:mkrwifi1010" --only-compilation-database --verbose
Using board 'mkrwifi1010' from platform in folder: C:\Users\falve\AppData\Local\Arduino15\packages\arduino\hardware\samd\1.8.11
Using core 'arduino' from platform in folder: C:\Users\falve\AppData\Local\Arduino15\packages\arduino\hardware\samd\1.8.11
Detecting libraries used...
"C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4/bin/arm-none-eabi-g++"

So I believe this is actually a bug with the Arduino CLI rather than a feature request.

Edit 2:

Interestingly, after changing the generated compile_commands.json paths, I get the following error from clangd:

E[17:41:12.228] System include extraction: C:\Users\falve\AppData\Local\Arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4\bin\arm-none-eabi-g++ does not exist.

[Falven]

So to clear something up, @matthijskooijman, the --query-driver command only whitelists potential compilers, but the actual driver commands come from the compilers. So query-driver is useless unless you want to tell it which compiler to use specifically from the same toolchain.

I see now, when I remove the whitelist (which does not work on windows because of path mismatches from the arduino-cli-generated database) the following output:

V[17:58:40.955] Driver produced command: cc1 -cc1 -triple thumbv6m-none-unknown-eabi -fsyntax-only -disable-free -disable-llvm-verifier -discard-value-names -main-file-name clangd-arduino.ino -mrelocation-model static -mframe-pointer=all -fmath-errno -fno-rounding-math -fno-verbose-asm -mconstructor-aliases -nostdsysteminc -target-cpu cortex-m0plus -target-feature +soft-float-abi -target-feature -crc -target-feature -crypto -target-feature -sha2 -target-feature -aes -target-feature -dotprod -target-feature -dsp -target-feature -mve -target-feature -mve.fp -target-feature -fullfp16 -target-feature -ras -target-feature -bf16 -target-feature -sb -target-feature -i8mm -target-feature -lob -target-feature -cdecp0 -target-feature -cdecp1 -target-feature -cdecp2 -target-feature -cdecp3 -target-feature -cdecp4 -target-feature -cdecp5 -target-feature -cdecp6 -target-feature -cdecp7 -target-feature -hwdiv-arm -target-feature -hwdiv -target-feature -fp16fml -target-feature +strict-align -target-abi aapcs -mfloat-abi soft -fallow-half-arguments-and-returns -fno-split-dwarf-inlining -debug-info-kind=limited -dwarf-version=4 -debugger-tuning=gdb -ffunction-sections -fdata-sections -resource-dir c:\Users\falve\AppData\Roaming\Code\User\globalStorage\llvm-vs-code-extensions.vscode-clangd\install\12.0.0\clangd_12.0.0\lib\clang\12.0.0 -D F_CPU=48000000L -D ARDUINO=10607 -D ARDUINO_SAMD_MKRWIFI1010 -D ARDUINO_ARCH_SAMD -D USE_ARDUINO_MKR_PIN_LAYOUT -D __SAMD21G18A__ -D USB_VID=0x2341 -D USB_PID=0x8054 -D USBCON -D USB_MANUFACTURER="Arduino LLC" -D USB_PRODUCT="Arduino MKR WiFi 1010" -D USE_BQ24195L_PMIC -I C:\Users\falve\AppData\Local\Arduino15\packages\arduino\tools\CMSIS\4.5.0\CMSIS\Include\ -I C:\Users\falve\AppData\Local\Arduino15\packages\arduino\tools\CMSIS-Atmel\1.2.0\CMSIS\Device\ATMEL\ -I C:\Users\falve\AppData\Local\Arduino15\packages\arduino\hardware\samd\1.8.11\cores\arduino\api\deprecated -I C:\Users\falve\AppData\Local\Arduino15\packages\arduino\hardware\samd\1.8.11\cores\arduino\api\deprecated-avr-comp -I C:\Users\falve\AppData\Local\Arduino15\packages\arduino\hardware\samd\1.8.11\cores\arduino -I C:\Users\falve\AppData\Local\Arduino15\packages\arduino\hardware\samd\1.8.11\variants\mkrwifi1010 -internal-isystem C:\Users\falve\AppData\Local\Arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4\bin\../lib/clang-runtimes\arm-none-eabi\include\c++\v1 -internal-isystem c:\Users\falve\AppData\Roaming\Code\User\globalStorage\llvm-vs-code-extensions.vscode-clangd\install\12.0.0\clangd_12.0.0\lib\clang\12.0.0\include -internal-isystem C:\Users\falve\AppData\Local\Arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4\bin\../lib/clang-runtimes\arm-none-eabi\include -Os -w -std=gnu++11 -fdeprecated-macro -fdebug-compilation-dir c:\Users\falve\.vscode\extensions\vsciot-vscode.vscode-arduino-0.4.2\out\src -ferror-limit 19 -fno-rtti -fno-signed-char -fgnuc-version=4.2.1 -fno-threadsafe-statics -vectorize-loops -vectorize-slp -faddrsig -x c++-header c:\source\repos\clangd-arduino\clangd-arduino.ino

But nowhere in there are the C++ includes from the toolchain Arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4\arm-none-eabi\include\c++\7.2.1 hence my intellisense not working.

Which makes sense, because if you look at the c_cpp_properties.json file that the VS Code Arduino Extension generates for the C/C++ Extension, you can see that the casing for the C++ includes is different than the top 6:

"includePath": [
    "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\CMSIS\\4.5.0\\CMSIS\\Include\\",
    "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\CMSIS-Atmel\\1.2.0\\CMSIS\\Device\\ATMEL\\",
    "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino\\api\\deprecated",
    "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino\\api\\deprecated-avr-comp",
    "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino",
    "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\variants\\mkrwifi1010",
    "c:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\arm-none-eabi\\include\\c++\\7.2.1",
    "c:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\arm-none-eabi\\include\\c++\\7.2.1\\arm-none-eabi",
    "c:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\arm-none-eabi\\include\\c++\\7.2.1\\backward",
    "c:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\lib\\gcc\\arm-none-eabi\\7.2.1\\include",
    "c:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\lib\\gcc\\arm-none-eabi\\7.2.1\\include-fixed",
    "c:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\arm-none-eabi\\include"
],

It is all lowercase, which leads me to believe that the extension is adding these on to the compiler driver output.

If I add those last include directories as shown below:

 {
  "directory": "c:\\Users\\falve\\.vscode\\extensions\\vsciot-vscode.vscode-arduino-0.4.2\\out\\src",
  "arguments": [
   "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4/bin/arm-none-eabi-g++",
   "-mcpu=cortex-m0plus",
   "-mthumb",
   "-c",
   "-g",
   "-Os",
   "-w",
   "-std=gnu++11",
   "-ffunction-sections",
   "-fdata-sections",
   "-fno-threadsafe-statics",
   "-nostdlib",
   "--param",
   "max-inline-insns-single=500",
   "-fno-rtti",
   "-fno-exceptions",
   "-MMD",
   "-DF_CPU=48000000L",
   "-DARDUINO=10607",
   "-DARDUINO_SAMD_MKRWIFI1010",
   "-DARDUINO_ARCH_SAMD",
   "-DUSE_ARDUINO_MKR_PIN_LAYOUT",
   "-D__SAMD21G18A__",
   "-DUSB_VID=0x2341",
   "-DUSB_PID=0x8054",
   "-DUSBCON",
   "-DUSB_MANUFACTURER=\"Arduino LLC\"",
   "-DUSB_PRODUCT=\"Arduino MKR WiFi 1010\"",
   "-DUSE_BQ24195L_PMIC",
   "-isystemc:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\cmsis\\4.5.0\\cmsis\\include\\",
   "-isystemc:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\cmsis-atmel\\1.2.0\\cmsis\\device\\atmel\\",
   "-isystemc:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino\\api\\deprecated",
   "-isystemc:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino\\api\\deprecated-avr-comp",
   "-isystemc:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino",
   "-isystemc:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\variants\\mkrwifi1010",
   "-isystemc:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\arm-none-eabi\\include\\c++\\7.2.1",
   "-isystemc:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\arm-none-eabi\\include\\c++\\7.2.1\\arm-none-eabi",
   "-isystemc:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\arm-none-eabi\\include\\c++\\7.2.1\\backward",
   "-isystemc:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\lib\\gcc\\arm-none-eabi\\7.2.1\\include",
   "-isystemc:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\lib\\gcc\\arm-none-eabi\\7.2.1\\include-fixed",
   "-isystemc:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\arm-none-eabi\\include",
   "c:\\source\\repos\\clangd-arduino\\build\\sketch\\clangd-arduino.ino.cpp",
   "-o",
   "c:\\source\\repos\\clangd-arduino\\build\\sketch\\clangd-arduino.ino.cpp.o"
  ],
  "file": "c:\\source\\repos\\clangd-arduino\\build\\sketch\\clangd-arduino.ino.cpp"
 },

Then intellisense is happy. Note: all of those includes should be -isystem; they are currently being included with -I in the generated arduino-cli compilation file, which can confuse some intellisense engines or code analyzers (such as clang-tidy) that rely on this metadata to know whether a header is a system header or not. And also the paths need to be fixed on windows...

[matthijskooijman]

@Falven, ok so for the query driver option, the slashes are problematic? Did you try using only backward (or only forward) slashes in query-driver and manually modifying compile_commands.json to match? Does it work then? If so, then I guess the mixed slashes in compile_commands.json prevent clangd from working and should be fixed in arduino-cli.

I think that for executing a command, these mixed slashes are acceptable on Windows, so there has not been any need to fix them so far. The mixed slashes come from the fact that some paths are autofilled (so use \ on Windows) and some come from platform.txt (and typically use '/'). Note that fixing this for command arguments is probably not possible (arduino-cli does not know which arguments are paths and must be changed and which are not and must be left alone), but for the compiler command itself, it is safe to assume that is a path and it should be fixable.

So to clear something up, @matthijskooijman, the --query-driver command only whitelists potential compilers, but the actual driver commands come from the compilers. So query-driver is useless unless you want to tell it which compiler to use specifically from the same toolchain.

I do not understand what you mean here? AFAIU, the idea is that --query-driver should match the compiler command listed in compile_commands.json and then clangd will query the compiler from compile_commands.json for all needed options. It might be that it does this by asking the compiler for the driver used and then querying the driver for options maybe, but all this is handled by clangd and AFAIK you only need to whitelist the outer compiler.

[Falven]

@matthijskooijman I know I gave a lot of information throughout my debugging, so I apologize for that.
I think first we need to come to a common conclusion of what the --query-driver option does.

From the clangd thread I linked, @kadircet says:
"--query-driver option on itself just specifies an allowed list of drivers to execute. The real driver paths are deduced from compile commands."

What I gathered from this, then, is the following: the --query-driver option is not necessary, and only works as a whitelist. If there are multiple compilers in one toolchain, you can tell clangd which one to use from the toolchain. As @kadircet mentioned, The real driver path is deduced from the compile command in the compile_commands.json. In our case, the following line:

...
"arguments": [
   "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\bin\\arm-none-eabi-g++",
...

What we can then deduce is that if you don't provide the --query-driver option, clangd will still query all provided compilers in the compile_commands.json for include paths. Which I believe is correct and is the behavior that I observed in the clangd output I pasted in comments above.

I think the mixed slashes are a potential issue for certain applications. I think they are the underlying issue with clangd, for example, if you are trying to use this --query-driver option with the clangd CLI, it will not consider:

...\7-2017q4\bin\arm-none-eabi-g++

and

...\7-2017q4/bin/arm-none-eabi-g++

to be the same compiler. As can be observed from my previous comments.

[matthijskooijman]

First of, as for terminology, I just realized that the command from compile-commands.json is actually the driver command, which internally executes the compiler (I think I might have had them reversed in my earlier comment). And clangd needs to figure out the compiler command used internally, since that includes all include paths explicitly. Below, I'm using the correct terms.

What we can then deduce is that if you don't provide the --query-driver option, clangd will still query all provided compilers in the compile_commands.json for include paths.

I do not think this is true. I believe that the query-driver option is a security feature, since clangd needs to execute the driver to query it, so it only does that for some known-safe driver paths, plus explicitly whitelisted ones with query-driver. If you do not specify query-driver and you end up with a driver in compile_commands.json that is not (default) whitelisted, I suspect that clangd will just assume you're using some default driver (i.e. /usr/bin/gcc, or maybe clang or so) and query that, or just autogenerate a default compiler commandline and includepaths.

I think the mixed slashes are a potential issue for certain applications. I think they are the underlying issue with clangd, for example, if you are trying to use this --query-driver option with the clangd CLI, it will not consider:

I don't think this is entirely true: From your earlier comment, I see that if you specify all \ in --query-driver and mixed (back)slashes in compile_commands, then clangd complains "not allowed driver". This suggests that clangd does not do any slash normalization of commands, it just checks them against the whitelist verbatim. Seems like a sane approach.

But if you change the --query-driver option to match the mixed (back)slashes from compile_commands.json, then the whitelist passes, and (according to your post) clangd now complains with:

E[14:14:21.550] System include extraction: C:\Users\falve\AppData\Local\Arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4/bin/arm-none-eabi-g++ does not exist.

So that suggests to me that clangd indeed does not like the mixed slashes, but the problem is not in the whitelist, but in the execution of the driver.

So, the proper fix is probably to produce a compile_commands.json with consistent slashes.

Which is why I suggested you test this fix manually, i.e. from my previous comment:

Did you try using only backward (or only forward) slashes in query-driver and manually modifying compile_commands.json to match? Does it work then? If so, then I guess the mixed slashes in compile_commands.json prevent clangd from working and should be fixed in arduino-cli.

[Falven]

Paying attention to slashes, we can see that:

"--query-driver=C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\bin\\arm-none-eabi-gcc.exe,C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\bin\\arm-none-eabi-g++.exe"

outputs

V[14:17:51.042] System include extraction: not allowed driver C:\Users\falve\AppData\Local\Arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4/bin/arm-none-eabi-g++

and

"--query-driver=C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4/bin/arm-none-eabi-gcc,C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4/bin/arm-none-eabi-g++"

outputs

E[08:27:22.735] System include extraction: C:\Users\falve\AppData\Local\Arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4/bin/arm-none-eabi-g++ does not exist.

I think the last output is a good hint. I think the problem then is that the compiler doesn't have these system include paths to extract defined in it's "driver".

"-isystemc:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\arm-none-eabi\\include\\c++\\7.2.1",
"-isystemc:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\arm-none-eabi\\include\\c++\\7.2.1\\arm-none-eabi",
"-isystemc:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\arm-none-eabi\\include\\c++\\7.2.1\\backward",
"-isystemc:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\lib\\gcc\\arm-none-eabi\\7.2.1\\include",
"-isystemc:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\lib\\gcc\\arm-none-eabi\\7.2.1\\include-fixed",
"-isystemc:\\users\\falve\\appdata\\local\\arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\arm-none-eabi\\include",

Edit: Maybe this will be a hint, but I was following this issue which seems to be similar. And the output from the suggested command is the following:

Using built-in specs.
COLLECT_GCC=C:\Users\falve\AppData\Local\Arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4\bin\arm-none-eabi-g++.exe
Target: arm-none-eabi
Configured with: /tmp/jenkins/jenkins-GCC-7-build_toolchain_docker-633_20171130_1512067137/src/gcc/configure --build=x86_64-linux-gnu --host=i686-w64-mingw32 --target=arm-none-eabi --prefix=/tmp/jenkins/jenkins-GCC-7-build_toolchain_docker-633_20171130_1512067137/install-mingw --libexecdir=/tmp/jenkins/jenkins-GCC-7-build_toolchain_docker-633_20171130_1512067137/install-mingw/lib --infodir=/tmp/jenkins/jenkins-GCC-7-build_toolchain_docker-633_20171130_1512067137/install-mingw/share/doc/gcc-arm-none-eabi/info --mandir=/tmp/jenkins/jenkins-GCC-7-build_toolchain_docker-633_20171130_1512067137/install-mingw/share/doc/gcc-arm-none-eabi/man --htmldir=/tmp/jenkins/jenkins-GCC-7-build_toolchain_docker-633_20171130_1512067137/install-mingw/share/doc/gcc-arm-none-eabi/html --pdfdir=/tmp/jenkins/jenkins-GCC-7-build_toolchain_docker-633_20171130_1512067137/install-mingw/share/doc/gcc-arm-none-eabi/pdf --enable-languages=c,c++ --enable-mingw-wildcard --disable-decimal-float --disable-libffi --disable-libgomp --disable-libmudflap --disable-libquadmath --disable-libssp --disable-libstdcxx-pch --disable-nls --disable-shared --disable-threads --disable-tls --with-gnu-as --with-gnu-ld --with-headers=yes --with-newlib --with-python-dir=share/gcc-arm-none-eabi --with-sysroot=/tmp/jenkins/jenkins-GCC-7-build_toolchain_docker-633_20171130_1512067137/install-mingw/arm-none-eabi --with-libiconv-prefix=/tmp/jenkins/jenkins-GCC-7-build_toolchain_docker-633_20171130_1512067137/build-mingw/host-libs/usr --with-gmp=/tmp/jenkins/jenkins-GCC-7-build_toolchain_docker-633_20171130_1512067137/build-mingw/host-libs/usr --with-mpfr=/tmp/jenkins/jenkins-GCC-7-build_toolchain_docker-633_20171130_1512067137/build-mingw/host-libs/usr --with-mpc=/tmp/jenkins/jenkins-GCC-7-build_toolchain_docker-633_20171130_1512067137/build-mingw/host-libs/usr --with-isl=/tmp/jenkins/jenkins-GCC-7-build_toolchain_docker-633_20171130_1512067137/build-mingw/host-libs/usr --with-libelf=/tmp/jenkins/jenkins-GCC-7-build_toolchain_docker-633_20171130_1512067137/build-mingw/host-libs/usr --with-host-libstdcxx='-static-libgcc -Wl,-Bstatic,-lstdc++,-Bdynamic -lm' --with-pkgversion='GNU Tools for Arm Embedded Processors 7-2017-q4-major' --with-multilib-list=rmprofile
Thread model: single
gcc version 7.2.1 20170904 (release) [ARM/embedded-7-branch revision 255204] (GNU Tools for Arm Embedded Processors 7-2017-q4-major) 
COLLECT_GCC_OPTIONS='-E' '-v'
 c:/users/falve/appdata/local/arduino15/packages/arduino/tools/arm-none-eabi-gcc/7-2017q4/bin/../lib/gcc/arm-none-eabi/7.2.1/cc1plus.exe -E -quiet -v -iprefix c:\users\falve\appdata\local\arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4\bin\../lib/gcc/arm-none-eabi/7.2.1/ -isysroot c:\users\falve\appdata\local\arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4\bin\../arm-none-eabi -D__USES_INITFINI__ clangd-arduino.ino
ignoring duplicate directory "c:/users/falve/appdata/local/arduino15/packages/arduino/tools/arm-none-eabi-gcc/7-2017q4/lib/gcc/../../lib/gcc/arm-none-eabi/7.2.1/../../../../arm-none-eabi/include/c++/7.2.1"
ignoring duplicate directory "c:/users/falve/appdata/local/arduino15/packages/arduino/tools/arm-none-eabi-gcc/7-2017q4/lib/gcc/../../lib/gcc/arm-none-eabi/7.2.1/../../../../arm-none-eabi/include/c++/7.2.1/arm-none-eabi"
ignoring duplicate directory "c:/users/falve/appdata/local/arduino15/packages/arduino/tools/arm-none-eabi-gcc/7-2017q4/lib/gcc/../../lib/gcc/arm-none-eabi/7.2.1/../../../../arm-none-eabi/include/c++/7.2.1/backward"
ignoring duplicate directory "c:/users/falve/appdata/local/arduino15/packages/arduino/tools/arm-none-eabi-gcc/7-2017q4/lib/gcc/../../lib/gcc/arm-none-eabi/7.2.1/include"
ignoring nonexistent directory "c:\users\falve\appdata\local\arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4\bin\../arm-none-eabi/tmp/jenkins/jenkins-GCC-7-build_toolchain_docker-633_20171130_1512067137/install-mingw/lib/gcc/arm-none-eabi/7.2.1/../../../../include"
ignoring duplicate directory "c:/users/falve/appdata/local/arduino15/packages/arduino/tools/arm-none-eabi-gcc/7-2017q4/lib/gcc/../../lib/gcc/arm-none-eabi/7.2.1/include-fixed"
ignoring duplicate directory "c:/users/falve/appdata/local/arduino15/packages/arduino/tools/arm-none-eabi-gcc/7-2017q4/lib/gcc/../../lib/gcc/arm-none-eabi/7.2.1/../../../../arm-none-eabi/include"
ignoring nonexistent directory "c:\users\falve\appdata\local\arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4\bin\../arm-none-eabi/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 c:\users\falve\appdata\local\arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4\bin\../lib/gcc/arm-none-eabi/7.2.1/../../../../arm-none-eabi/include/c++/7.2.1
 c:\users\falve\appdata\local\arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4\bin\../lib/gcc/arm-none-eabi/7.2.1/../../../../arm-none-eabi/include/c++/7.2.1/arm-none-eabi
 c:\users\falve\appdata\local\arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4\bin\../lib/gcc/arm-none-eabi/7.2.1/../../../../arm-none-eabi/include/c++/7.2.1/backward
 c:\users\falve\appdata\local\arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4\bin\../lib/gcc/arm-none-eabi/7.2.1/include
 c:\users\falve\appdata\local\arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4\bin\../lib/gcc/arm-none-eabi/7.2.1/include-fixed
 c:\users\falve\appdata\local\arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4\bin\../lib/gcc/arm-none-eabi/7.2.1/../../../../arm-none-eabi/include
End of search list.
COMPILER_PATH=c:/users/falve/appdata/local/arduino15/packages/arduino/tools/arm-none-eabi-gcc/7-2017q4/bin/../lib/gcc/arm-none-eabi/7.2.1/;c:/users/falve/appdata/local/arduino15/packages/arduino/tools/arm-none-eabi-gcc/7-2017q4/bin/../lib/gcc/;c:/users/falve/appdata/local/arduino15/packages/arduino/tools/arm-none-eabi-gcc/7-2017q4/bin/../lib/gcc/arm-none-eabi/7.2.1/../../../../arm-none-eabi/bin/
LIBRARY_PATH=c:/users/falve/appdata/local/arduino15/packages/arduino/tools/arm-none-eabi-gcc/7-2017q4/bin/../lib/gcc/arm-none-eabi/7.2.1/;c:/users/falve/appdata/local/arduino15/packages/arduino/tools/arm-none-eabi-gcc/7-2017q4/bin/../lib/gcc/;c:/users/falve/appdata/local/arduino15/packages/arduino/tools/arm-none-eabi-gcc/7-2017q4/bin/../lib/gcc/arm-none-eabi/7.2.1/../../../../arm-none-eabi/lib/;c:/users/falve/appdata/local/arduino15/packages/arduino/tools/arm-none-eabi-gcc/7-2017q4/bin/../arm-none-eabi/lib/
COLLECT_GCC_OPTIONS='-E' '-v'

It looks like it is ignoring all of the c++ include directories that we are missing.

[matthijskooijman]

I think the last output is a good hint. I think the problem then is that the compiler doesn't have these system include paths to extract defined in it's "driver".

I don't know what you mean by this exactly, but I think it is wrong. What this means is that clangd tries to run the driver to query it for include paths and fails, because it does not support forward slashes in the driver executable path.

It looks like it is ignoring all of the c++ include directories that we are missing.

Do you mean the "ignoring duplicate directory" messages? Those are normal and just indicate that it's going to output these directories just once instead of multiple times. As you can see, they are output after the #include <...> search starts here: line.

As I suggested before:

Did you try using only backward (or only forward) slashes in query-driver and manually modifying compile_commands.json to match? Does it work then?

This is the third time I've given the above suggestion, which I think is the way forward to a fix, but it seems you're ignoring it, so I'm slowly running out of patience here. If my suggestion is unclear, please let me know, but otherwise please try this.

[Falven]

As I suggested before:

Did you try using only backward (or only forward) slashes in query-driver and manually modifying compile_commands.json to match? Does it work then?

This is the third time I've given the above suggestion, which I think is the way forward to a fix, but it seems you're ignoring it, so I'm slowly running out of patience here. If my suggestion is unclear, please let me know, but otherwise please try this.

@matthijskooijman It is not my intention to ignore anything, my apologies.

I have modified the generated compile_commands.json to use windows-style path separators:

[
 {
  "directory": "c:\\Users\\falve\\.vscode\\extensions\\vsciot-vscode.vscode-arduino-0.4.2\\out\\src",
  "arguments": [
   "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\bin\\arm-none-eabi-g++",
   "-mcpu=cortex-m0plus",
   "-mthumb",
   "-c",
   "-g",
   "-Os",
   "-w",
   "-std=gnu++11",
   "-ffunction-sections",
   "-fdata-sections",
   "-fno-threadsafe-statics",
   "-nostdlib",
   "--param",
   "max-inline-insns-single=500",
   "-fno-rtti",
   "-fno-exceptions",
   "-MMD",
   "-DF_CPU=48000000L",
   "-DARDUINO=10607",
   "-DARDUINO_SAMD_MKRWIFI1010",
   "-DARDUINO_ARCH_SAMD",
   "-DUSE_ARDUINO_MKR_PIN_LAYOUT",
   "-D__SAMD21G18A__",
   "-DUSB_VID=0x2341",
   "-DUSB_PID=0x8054",
   "-DUSBCON",
   "-DUSB_MANUFACTURER=\"Arduino LLC\"",
   "-DUSB_PRODUCT=\"Arduino MKR WiFi 1010\"",
   "-DUSE_BQ24195L_PMIC",
   "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\CMSIS\\4.5.0\\CMSIS\\Include\\",
   "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\CMSIS-Atmel\\1.2.0\\CMSIS\\Device\\ATMEL\\",
   "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino\\api\\deprecated",
   "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino\\api\\deprecated-avr-comp",
   "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino",
   "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\variants\\mkrwifi1010",
   "c:\\source\\repos\\clangd-arduino\\build\\sketch\\clangd-arduino.ino.cpp",
   "-o",
   "c:\\source\\repos\\clangd-arduino\\build\\sketch\\clangd-arduino.ino.cpp.o"
  ],
  "file": "c:\\source\\repos\\clangd-arduino\\build\\sketch\\clangd-arduino.ino.cpp"
 },
 {
  "directory": "c:\\Users\\falve\\.vscode\\extensions\\vsciot-vscode.vscode-arduino-0.4.2\\out\\src",
  "arguments": [
   "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\arm-none-eabi-gcc\\7-2017q4\\bin\\arm-none-eabi-g++",
   "-mcpu=cortex-m0plus",
   "-mthumb",
   "-c",
   "-g",
   "-Os",
   "-w",
   "-std=gnu++11",
   "-ffunction-sections",
   "-fdata-sections",
   "-fno-threadsafe-statics",
   "-nostdlib",
   "--param",
   "max-inline-insns-single=500",
   "-fno-rtti",
   "-fno-exceptions",
   "-MMD",
   "-DF_CPU=48000000L",
   "-DARDUINO=10607",
   "-DARDUINO_SAMD_MKRWIFI1010",
   "-DARDUINO_ARCH_SAMD",
   "-DUSE_ARDUINO_MKR_PIN_LAYOUT",
   "-D__SAMD21G18A__",
   "-DUSB_VID=0x2341",
   "-DUSB_PID=0x8054",
   "-DUSBCON",
   "-DUSB_MANUFACTURER=\"Arduino LLC\"",
   "-DUSB_PRODUCT=\"Arduino MKR WiFi 1010\"",
   "-DUSE_BQ24195L_PMIC",
   "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\CMSIS\\4.5.0\\CMSIS\\Include\\",
   "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\CMSIS-Atmel\\1.2.0\\CMSIS\\Device\\ATMEL\\",
   "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino\\api\\deprecated",
   "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino\\api\\deprecated-avr-comp",
   "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\cores\\arduino",
   "-IC:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\variants\\mkrwifi1010",
   "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\variants\\mkrwifi1010\\variant.cpp",
   "-o",
   "c:\\source\\repos\\clangd-arduino\\build\\core\\variant.cpp.o"
  ],
  "file": "C:\\Users\\falve\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\samd\\1.8.11\\variants\\mkrwifi1010\\variant.cpp"
 }
]

I have set the query driver argument as such:

I[11:22:59.419] Working directory: c:\source\repos\clangd-arduino
I[11:22:59.419] argv[0]: c:\Users\falve\AppData\Roaming\Code\User\globalStorage\llvm-vs-code-extensions.vscode-clangd\install\12.0.0\clangd_12.0.0\bin\clangd.exe
I[11:22:59.419] argv[1]: --log=verbose
I[11:22:59.419] argv[2]: --query-driver=C:\Users\falve\AppData\Local\Arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4\bin\arm-none-eabi-gcc,C:\Users\falve\AppData\Local\Arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4\bin\arm-none-eabi-g++

And the relevant output from clangd follows:

I[11:22:59.486] Loaded compilation database from c:\source\repos\clangd-arduino\build/compile_commands.json
V[11:22:59.486] Broadcasting compilation database from c:\source\repos\clangd-arduino
E[11:22:59.487] System include extraction: C:\Users\falve\AppData\Local\Arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4\bin\arm-none-eabi-g++ does not exist.
I[11:22:59.487] ASTWorker building file c:\source\repos\clangd-arduino\clangd-arduino.ino version 1 with command inferred from c:\source\repos\clangd-arduino\build\sketch\clangd-arduino.ino.cpp
[c:\Users\falve\.vscode\extensions\vsciot-vscode.vscode-arduino-0.4.2\out\src]
C:\Users\falve\AppData\Local\Arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4\bin\arm-none-eabi-g++ --target=arm-none-eabi --driver-mode=g++ -mcpu=cortex-m0plus -mthumb -c -g -Os -w -ffunction-sections -fdata-sections -fno-threadsafe-statics -nostdlib --param max-inline-insns-single=500 -fno-rtti -fno-exceptions -DF_CPU=48000000L -DARDUINO=10607 -DARDUINO_SAMD_MKRWIFI1010 -DARDUINO_ARCH_SAMD -DUSE_ARDUINO_MKR_PIN_LAYOUT -D__SAMD21G18A__ -DUSB_VID=0x2341 -DUSB_PID=0x8054 -DUSBCON -DUSB_MANUFACTURER="Arduino LLC" -DUSB_PRODUCT="Arduino MKR WiFi 1010" -DUSE_BQ24195L_PMIC -IC:\Users\falve\AppData\Local\Arduino15\packages\arduino\tools\CMSIS\4.5.0\CMSIS\Include\ -IC:\Users\falve\AppData\Local\Arduino15\packages\arduino\tools\CMSIS-Atmel\1.2.0\CMSIS\Device\ATMEL\ -IC:\Users\falve\AppData\Local\Arduino15\packages\arduino\hardware\samd\1.8.11\cores\arduino\api\deprecated -IC:\Users\falve\AppData\Local\Arduino15\packages\arduino\hardware\samd\1.8.11\cores\arduino\api\deprecated-avr-comp -IC:\Users\falve\AppData\Local\Arduino15\packages\arduino\hardware\samd\1.8.11\cores\arduino -IC:\Users\falve\AppData\Local\Arduino15\packages\arduino\hardware\samd\1.8.11\variants\mkrwifi1010 -x c++-header -std=gnu++11 c:\source\repos\clangd-arduino\clangd-arduino.ino -fsyntax-only -resource-dir=c:\Users\falve\AppData\Roaming\Code\User\globalStorage\llvm-vs-code-extensions.vscode-clangd\install\12.0.0\clangd_12.0.0\lib\clang\12.0.0
V[11:22:59.489] Driver produced command: cc1 -cc1 -triple thumbv6m-none-unknown-eabi -fsyntax-only -disable-free -disable-llvm-verifier -discard-value-names -main-file-name clangd-arduino.ino -mrelocation-model static -mframe-pointer=all -fmath-errno -fno-rounding-math -fno-verbose-asm -mconstructor-aliases -nostdsysteminc -target-cpu cortex-m0plus -target-feature +soft-float-abi -target-feature -crc -target-feature -crypto -target-feature -sha2 -target-feature -aes -target-feature -dotprod -target-feature -dsp -target-feature -mve -target-feature -mve.fp -target-feature -fullfp16 -target-feature -ras -target-feature -bf16 -target-feature -sb -target-feature -i8mm -target-feature -lob -target-feature -cdecp0 -target-feature -cdecp1 -target-feature -cdecp2 -target-feature -cdecp3 -target-feature -cdecp4 -target-feature -cdecp5 -target-feature -cdecp6 -target-feature -cdecp7 -target-feature -hwdiv-arm -target-feature -hwdiv -target-feature -fp16fml -target-feature +strict-align -target-abi aapcs -mfloat-abi soft -fallow-half-arguments-and-returns -fno-split-dwarf-inlining -debug-info-kind=limited -dwarf-version=4 -debugger-tuning=gdb -ffunction-sections -fdata-sections -resource-dir c:\Users\falve\AppData\Roaming\Code\User\globalStorage\llvm-vs-code-extensions.vscode-clangd\install\12.0.0\clangd_12.0.0\lib\clang\12.0.0 -D F_CPU=48000000L -D ARDUINO=10607 -D ARDUINO_SAMD_MKRWIFI1010 -D ARDUINO_ARCH_SAMD -D USE_ARDUINO_MKR_PIN_LAYOUT -D __SAMD21G18A__ -D USB_VID=0x2341 -D USB_PID=0x8054 -D USBCON -D USB_MANUFACTURER="Arduino LLC" -D USB_PRODUCT="Arduino MKR WiFi 1010" -D USE_BQ24195L_PMIC -I C:\Users\falve\AppData\Local\Arduino15\packages\arduino\tools\CMSIS\4.5.0\CMSIS\Include\ -I C:\Users\falve\AppData\Local\Arduino15\packages\arduino\tools\CMSIS-Atmel\1.2.0\CMSIS\Device\ATMEL\ -I C:\Users\falve\AppData\Local\Arduino15\packages\arduino\hardware\samd\1.8.11\cores\arduino\api\deprecated -I C:\Users\falve\AppData\Local\Arduino15\packages\arduino\hardware\samd\1.8.11\cores\arduino\api\deprecated-avr-comp -I C:\Users\falve\AppData\Local\Arduino15\packages\arduino\hardware\samd\1.8.11\cores\arduino -I C:\Users\falve\AppData\Local\Arduino15\packages\arduino\hardware\samd\1.8.11\variants\mkrwifi1010 -internal-isystem C:\Users\falve\AppData\Local\Arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4\bin\../lib/clang-runtimes\arm-none-eabi\include\c++\v1 -internal-isystem c:\Users\falve\AppData\Roaming\Code\User\globalStorage\llvm-vs-code-extensions.vscode-clangd\install\12.0.0\clangd_12.0.0\lib\clang\12.0.0\include -internal-isystem C:\Users\falve\AppData\Local\Arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4\bin\../lib/clang-runtimes\arm-none-eabi\include -Os -w -std=gnu++11 -fdeprecated-macro -fdebug-compilation-dir c:\Users\falve\.vscode\extensions\vsciot-vscode.vscode-arduino-0.4.2\out\src -ferror-limit 19 -fno-rtti -fno-signed-char -fgnuc-version=4.2.1 -fno-threadsafe-statics -vectorize-loops -vectorize-slp -faddrsig -x c++-header c:\source\repos\clangd-arduino\clangd-arduino.ino

[matthijskooijman]

Hm, that surprises me, so just fixing the slashes is not going to help here, it seems.

Also, it seems you're editing your github comments a lot, which sometimes makes it a bit hard to reply, since I often read messages from the e-mail notification, which shows the initial version. Maybe you could use the preview function a bit more, and if you do have significant new findings, just post a new comment? In the initial version of your last comment, the clangd error was different (does not exist rather than not allowed driver), what's up with that?

I wonder if maybe vs is messing up the query driver argument somehow or something else? Maybe you could try running clangd manually directly to rule that out?

[Falven]

Hm, that surprises me, so just fixing the slashes is not going to help here, it seems.

Also, it seems you're editing your github comments a lot, which sometimes makes it a bit hard to reply, since I often read messages from the e-mail notification, which shows the initial version. Maybe you could use the preview function a bit more, and if you do have significant new findings, just post a new comment? In the initial version of your last comment, the clangd error was different (does not exist rather than not allowed driver), what's up with that?

I wonder if maybe vs is messing up the query driver argument somehow or something else? Maybe you could try running clangd manually directly to rule that out?

Yeah, sorry. I made a typo - I had added .exe to the --query-driver path by accident.

Let's try running it manually:

PS C:\source\repos\clangd-arduino> C:\Users\falve\AppData\Roaming\Code\User\globalStorage\llvm-vs-code-extensions.vscode-clangd\install\12.0.0\clangd_12.0.0\bin\clangd.exe --log=verbose --query-driver="C:\Users\falve\AppData\Local\Arduino15\packages\arduino\tooackages\arduino\tools\arm-none-eabi-gcc\7-2017q4\bin\arm-none-eabi-gcc,C:\Users\falve\AppData\Local\Arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4\bin\arm-none-eabi-g++"
clangd is a language server that provides IDE-like features to editors.

It should be used via an editor plugin rather than invoked directly. For more information, see:
        https://clangd.llvm.org/
        https://microsoft.github.io/language-server-protocol/

clangd accepts flags on the commandline, and in the CLANGD_FLAGS environment variable.

I[11:28:22.410] clangd version 12.0.0 (https://github.com/kirillbobyrev/llvm-project 8a844a3a321ea56da21af4d48433ad493b1bcfae)
I[11:28:22.411] PID: 16544
I[11:28:22.411] Working directory: C:\source\repos\clangd-arduino
I[11:28:22.412] argv[0]: C:\Users\falve\AppData\Roaming\Code\User\globalStorage\llvm-vs-code-extensions.vscode-clangd\install\12.0.0\clangd_12.0.0\bin\clangd.exe
I[11:28:22.412] argv[1]: --log=verbose
I[11:28:22.412] argv[2]: --query-driver=C:\Users\falve\AppData\Local\Arduino15\packages\arduino\tooackages\arduino\tools\arm-none-eabi-gcc\7-2017q4\bin\arm-none-eabi-gcc,C:\Users\falve\AppData\Local\Arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4\bin\arm-none-eabi-g++
V[11:28:22.423] User config file is C:\Users\falve\AppData\Local\clangd\config.yaml
I[11:28:22.424] Starting LSP over stdin/stdout

Now I have to figure out how to send it a file open command over stdout?