Build as component fails

[PaulMartinsen]

How does the build system know which paths to add as includes? In particular wifi_provisioning/manager.h is missing from the include list (although it is present in components/wifi_provisioning/include)

I have been following the instructions at esp-idf_component.md to build the library but the build fails at files that include wifi_provisioning/manager.h. This seems to be a recent addition.

It looks like the include for components\wifi_provisioning\include is missing from the build command. Here's the offending build step, reformated for clarity.

ccache C:\Programming\Languages\ESP-IDF\Tools\tools\xtensa-esp32-elf\1.22.0-80-g6c4433a5-5.2.0\xtensa-esp32-elf\bin\xtensa-esp32-elf-g++.exe  -DESP_PLATFORM -DGCC_NOT_5_2_0=0 -DHAVE_CONFIG_H -DIDF_VER=\"v3.3.2\" -DMBEDTLS_CONFIG_FILE=\"mbedtls/esp_config.h\" 
-I../components/variants/esp32 
-I../components/cores/esp32 
-I../components/libraries/ArduinoOTA/src 
-I../components/libraries/AsyncUDP/src 
-I../components/libraries/AzureIoT/src 
-I../components/libraries/BLE/src 
-I../components/libraries/BluetoothSerial/src 
-I../components/libraries/DNSServer/src 
-I../components/libraries/EEPROM/src 
-I../components/libraries/ESP32/src 
-I../components/libraries/ESPmDNS/src 
-I../components/libraries/FFat/src 
-I../components/libraries/FS/src 
-I../components/libraries/HTTPClient/src 
-I../components/libraries/HTTPUpdate/src 
-I../components/libraries/NetBIOS/src 
-I../components/libraries/Preferences/src 
-I../components/libraries/SD_MMC/src 
-I../components/libraries/SD/src 
-I../components/libraries/SimpleBLE/src 
-I../components/libraries/SPIFFS/src 
-I../components/libraries/SPI/src 
-I../components/libraries/Ticker/src 
-I../components/libraries/Update/src 
-I../components/libraries/WebServer/src 
-I../components/libraries/WiFiClientSecure/src 
-I../components/libraries/WiFi/src 
-I../components/libraries/Wire/src 
-I../components/cores/esp32/libb64 
-Iconfig 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/spi_flash/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/mbedtls/port/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/mbedtls/mbedtls/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/esp32/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/driver/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/esp_ringbuf/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/tcpip_adapter/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/lwip/include/apps 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/lwip/include/apps/sntp 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/lwip/lwip/src/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/lwip/port/esp32/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/lwip/port/esp32/include/arch 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/lwip/include_compat 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/vfs/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/esp_event/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/log/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/efuse/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/efuse/esp32/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/newlib/platform_include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/newlib/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/freertos/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/app_trace/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/heap/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/soc/esp32/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/soc/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/mdns/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/console 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/ethernet/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/esp_adc_cal/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/fatfs/src 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/wear_levelling/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/sdmmc/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/nvs_flash/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/app_update/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/bootloader_support/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/spiffs/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/openssl/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/bt/include 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/bt/bluedroid/api/include/api 
-IC:/Programming/Languages/ESP-IDF/Release-3-3-2/components/bt/common/osi/include 
-mlongcalls   -mfix-esp32-psram-cache-issue -mlongcalls -Os -ffunction-sections -fdata-sections -fstrict-volatile-bitfields -nostdlib -Wall -Werror=all -Wno-error=unused-function -Wno-error=unused-but-set-variable -Wno-error=unused-variable -Wno-error=deprecated-declarations -Wextra -Wno-unused-parameter -Wno-sign-compare -fstack-protector -ggdb -std=gnu++11 -fno-rtti -fexceptions -MD -MT esp-idf/components/CMakeFiles/idf_component_components.dir/libraries/WiFiClientSecure/src/ssl_client.cpp.obj -MF esp-idf\components\CMakeFiles\idf_component_components.dir\libraries\WiFiClientSecure\src\ssl_client.cpp.obj.d -o esp-idf/components/CMakeFiles/idf_component_components.dir/libraries/WiFiClientSecure/src/ssl_client.cpp.obj -c ../components/libraries/WiFiClientSecure/src/ssl_client.cpp

In file included from ../components/libraries/WiFi/src/WiFiSTA.h:28:0,
                 from ../components/libraries/WiFi/src/WiFi.h:32,
                 from ../components/libraries/WiFiClientSecure/src/ssl_client.cpp:20:
../components/libraries/WiFi/src/WiFiGeneric.h:31:39: fatal error: wifi_provisioning/manager.h: No such file or directory

[Ben79543]

Hi Paul,
Did you find a way?
Fyi: https://github.com/espressif/arduino-esp32/issues/1531

[Ben79543]

I tried a dirty hack and copied /wifi_provisioning in the wifi src directory.
That error is gone. But I then get another one.

C:/esp/blink/components/arduino/libraries/WiFi/src/ETH.cpp: In member function 'IPAddress ETHClass::dnsIP(uint8_t)': C:/esp/blink/components/arduino/libraries/WiFi/src/ETH.cpp:196:37: error: conversion from 'const ip_addr_t* {aka const ip_addr*}' to non-scalar type 'ip_addr_t {aka ip_addr}' requested ip_addr_t dns_ip = dns_getserver(dns_no);
I found what I thought was a solution(?) for this one: #3760

But then I get a consequence of that fix too because it is lwip that fails next.

C:/esp/esp-idf/components/tcpip_adapter/tcpip_adapter_lwip.c:769:17: error: incompatible types when assigning to type 'ip_addr_t {aka struct ip_addr}' from type 'const ip_addr_t * {aka const struct ip_addr *}' dns->ip = dns_getserver(type); ^

C:/esp/esp-idf/components/tcpip_adapter/tcpip_adapter_lwip.c:769:17: error: incompatible types when assigning to type 'ip_addr_t {aka struct ip_addr}' from type 'const ip_addr_t * {aka const struct ip_addr *}' dns->ip = dns_getserver(type);

That's too many errors... must be wrong somewhere...

[Ben79543]

I tried upgrading esp-idf to 3.3.
Now it compiles without errors after all the hacks above....
Now need to try in arduino... With the .a in the right place...

[Ben79543]

... and that did not compile in arduino. :-(

-------------------------Edit:
Ok I finally got it to work with this info:

• in Arduino I have esp boards 1.04 in borad manager.
• I installed IDF 3.2.
• I did a git checkout to d3e562907 ***//EDIT: There were still some errors, In the end I used IDF v3.2.3 and it worked
• Then I applied couple of hacks (as decribed above): Build error: conversion from 'const ip_addr_t {aka const ip_addr}' to non-scalar type 'ip_addr_t {aka ip_addr}' requested #3760 and also copying the /wifi_provisioning in the wifi src directory.
• This allows to get the "make" command succeed.
• But to get Arduino to compile correctly copy all the built .a files EXCEPT the ones in "bootloader" into lib directory of your arduino ide. That is the critical part.

That is how I finally got an arduino sketch to successfully compile.
Took me 5 full days. :-(

[PaulMartinsen]

Just coming back to this after being distracted by another project. Thanks so much for all your work Ben. I really appreciate it.

I followed along but got stuck when I tried to copy wifi_provisioning/manager.h from tagged branch release/v3.2: it wasn't there! I realize I was on the wrong branch as I write this. Should have been release/v3.2.3.

Somewhere in there I went down the rabbit hole trying to figure out how the make files were working. I found if I add wifi_provisioning to line 208 of CMakeLists.txt then the wifi provisioning path gets added to the include list so I don't have to copy across the files anymore. So that line becomes:
set(COMPONENT_REQUIRES spi_flash mbedtls mdns ethernet esp_adc_cal wifi_provisioning)

That looks like the key to telling the build system which components in the ESP32's components folder are required for the build. The ESP32 documentation talks about it in Preset Component Variables.

Then with your fixes for the return type of dns_getserver in libraries/WiFi/src/ETH.cpp and libraries/WiFi/src/WiFiSTA.cpp I was able to build against ESP-IDF release 3.3.2.

I turned the hello_world_main.c into an Arduino sketch with setup() and loop() functions and am able to build that with idf.py build. I haven't gotten as far as moving the library files into the Arduino directories and building with the Arduino IDE. I'll try to test that this weekend but I expect to get the same result you did. Once I verify that works, I'll try to figure out exactly which esp-idf commit the arduino-esp32 library is using, update this issue & consider doing a PR.

Thanks again for your help Ben. I was really stuck on this.

[PaulMartinsen]

Turns out the change to the return type of dns_getserver is not in the esp-idf branch referenced in this commit:
1977370e6fc069e93ffd8818798fbfda27ae7d99
IDF release/v3.3 (#3339)

• IDF release/v3.3 46b12a560
• fix build
• IDF release/v3.3 367c3c09c

Which refers to this esp-idf commit. Which is not the latest commit on the release/v3.3 branch.

So provided you check out the specific esp-idf commit, arduino-esp32 (the latest as of 2020-05-16) will build provided wifi_provisioning is added to CMake.txt as described above.

[PaulMartinsen]

I created #4003 to implement this, with thanks to @Ben79543 for all his help.

[ryanzmi]

Urr when is this fix going to merge into master?
I'm using arduino-esp32 origin/master and IDF release/v3.3 367c3c09c
idf.py build did fail at the file Paul indicated.
After this wifi_provisioning is added to CMake, this works again.
Which I think it's a good fix.

[me-no-dev]

you guys want to try the latest version? the esp32s2 branch compiles against IDF master. There is also a 4.0 branch that compiles with IDF 4.0.

[Ben79543]

hi @me-no-dev
I actually tried to compile arduino as a component (branch idf-release/v4.0) on idf v4.
I described it here: https://esp32.com/viewtopic.php?f=13&t=15564

I got it to compile indeed with idf.py. Then I copied the .a librairies back to arduino. Which was my goal from start (to get low power ble working).
A simple blink sketch would compile, but another simple sketch with BLE commands would not.

Not sure if it has to do with the compile librairies or how I copied them back to arduino...

[me-no-dev]

your best bet to compile the libs is to use the lib-builder :) it's what compiles the libs for Arduino currently (and even creates the pull request for the new libs) https://github.com/espressif/esp32-arduino-lib-builder

[me-no-dev]

uhmm lib builder 4.0 branch that is