External PSRAM not supported?

[jgamble-simple]

Hardware:

Board: Custom board based on ESP32-WROVER-E
Core Installation version: 1.0.6
IDE name: Platform.io
PSRAM enabled: yes
Upload Speed: 230400
Computer OS: Windows 10 (Using wsl for uploading/platform.io)

Description:

Calling heap_caps_malloc_extmem_enable(0) causes a crash about 20% of the time when trying to connect to wifi, I have trimmed back my application to the bare min required to crash it. If I remove the heap_caps_malloc_extmem_enable call then it works properly........

Is this supposed to work? Or this crash expected?

Compiler Flags defined in platformio.ini:

build_unflags = -Os
build_flags = 
   -DCORE_DEBUG_LEVEL=5
   -DBOARD_HAS_PSRAM=1
   -DCONFIG_SPIRAM_CACHE_WORKAROUND=1
   -mfix-esp32-psram-cache-issue
   -O3 -ffast-math
   -Wl,-Map,output.map

    -DCONFIG_ARDUINO_ISR_IRAM=1

Sketch:

#include <WiFi.h>
#include "esp_wifi.h"
#include "esp_heap_caps.h"

void setup() {
  heap_caps_malloc_extmem_enable(0);

  WiFi.begin("", ""); // Credentials removed
}

void loop() {
}

Debug Messages:

ets Jul 29 2019 12:21:46

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:1044
load:0x40078000,len:10124
load:0x40080400,len:5828
entry 0x400806a8
[D][esp32-hal-psram.c:47] psramInit(): PSRAM enabled
[D][WiFiGeneric.cpp:374] [D][WiFiGeneric.cpp:374] _eventCallback(): Event: 2 - STA_START
_eventCallback(): Event: 0 - WIFI_READY
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Core 0 register dump:
PC      : 0x4008d508  PS      : 0x00060034  A0      : 0x8008e14c  A1      : 0x3ffbf790
A2      : 0x3f800de4  A3      : 0xb33fffff  A4      : 0x3ffbf7cc  A5      : 0xbad00bad
A6      : 0x00000001  A7      : 0x00000001  A8      : 0x00000000  A9      : 0x00000000
A10     : 0x00000000  A11     : 0x00000000  A12     : 0x40062229  A13     : 0x3ffbf8b0
A14     : 0x00e0a000  A15     : 0x3ffbfec8  SAR     : 0x00000018  EXCCAUSE: 0x00000007
EXCVADDR: 0x00000000  LBEG    : 0x4008b190  LEND    : 0x4008b1ac  LCOUNT  : 0x00000000
Core 0 was running in ISR context:
EPC1    : 0x40062229  EPC2    : 0x00000000  EPC3    : 0x00000000  EPC4    : 0x4008d508

ELF file SHA256: 0000000000000000

Backtrace: 0x4008d508:0x3ffbf790 0x4008e149:0x3ffbf7c0 0x4008cd95:0x3ffbf7f0 0x40083bd6:0x3ffbf810 0x40081522:0x3ffbf830 0x4008143f:0x3ffbf860 0x40081917:0x3ffbf880 0x400891c5:0x3ffbf8b0 0x40062226:0x3ffb3f80 0x4009243f:0x3ffb3fa0 0x4009247a:0x3ffb3fd0 0x40092506:0x3ffb4000 0x40092975:0x3ffb4020 0x40089c78:0x3ffb4040 0x40089d0a:0x3ffb4060 0x401353f7:0x3ffb4080 0x4013577d:0x3ffb40a0 0x401335ea:0x3ffb40c0 0x40133b02:0x3ffb4140 0x40132f07:0x3ffb41b0 0x400df10a:0x3ffb41f0 0x400dd158:0x3ffb4210 0x400eb689:0x3ffb4230 0x400de76b:0x3ffb42b0 0x4010a497:0x3ffb42d0 0x401086aa:0x3ffb42f0 0x401097cc:0x3ffb4310 0x400fe3e7:0x3ffb43a0 0x40100666:0x3ffb43d0 0x4008d1ca:0x3ffb4400
  #0  0x4008d508:0x3ffbf790 in uxPortCompareSetExtram at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c:452
  #1  0x4008e149:0x3ffbf7c0 in vPortCPUAcquireMutexIntsDisabledExtram at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/tasks.c:3507
      (inlined by) vPortCPUAcquireMutexIntsDisabled at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/portmux_impl.h:95
      (inlined by) vTaskEnterCritical at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/tasks.c:4201
  #2  0x4008cd95:0x3ffbf7f0 in xQueueGenericSendFromISR at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/queue.c:2038
  #3  0x40083bd6:0x3ffbf810 in queue_send_from_isr_wrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/esp_adapter.c:324
  #4  0x40081522:0x3ffbf830 in pp_post at ??:?
  #5  0x4008143f:0x3ffbf860 in lmacProcessRxSucData at ??:?
  #6  0x40081917:0x3ffbf880 in wDev_ProcessFiq at ??:?
  #7  0x400891c5:0x3ffbf8b0 in _xt_lowint1 at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/xtensa_vectors.S:1154
  #8  0x40062226:0x3ffb3f80 in ?? ??:0
  #9  0x4009243f:0x3ffb3fa0 in esp_rom_spiflash_read_status at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/spi_flash/spi_flash_rom_patch.c:662
  #10 0x4009247a:0x3ffb3fd0 in esp_rom_spiflash_wait_idle at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/spi_flash/spi_flash_rom_patch.c:662
  #11 0x40092506:0x3ffb4000 in esp_rom_spiflash_erase_sector_internal at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/spi_flash/spi_flash_rom_patch.c:662
  #12 0x40092975:0x3ffb4020 in esp_rom_spiflash_erase_sector at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/spi_flash/spi_flash_rom_patch.c:662
  #13 0x40089c78:0x3ffb4040 in spi_flash_erase_range at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/spi_flash/flash_ops.c:247
  #14 0x40089d0a:0x3ffb4060 in spi_flash_erase_sector at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/spi_flash/flash_ops.c:215
  #15 0x401353f7:0x3ffb4080 in nvs::Page::erase() at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/nvs_flash/src/nvs_page.cpp:876
  #16 0x4013577d:0x3ffb40a0 in nvs::PageManager::requestNewPage() at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/nvs_flash/src/nvs_pagemanager.cpp:184
  #17 0x401335ea:0x3ffb40c0 in nvs::Storage::writeMultiPageBlob(unsigned char, char const*, void const*, unsigned int, nvs::VerOffset) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/nvs_flash/src/nvs_storage.cpp:579
  #18 0x40133b02:0x3ffb4140 in nvs::Storage::writeItem(unsigned char, nvs::ItemType, char const*, void const*, unsigned int) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/nvs_flash/src/nvs_storage.cpp:579
  #19 0x40132f07:0x3ffb41b0 in nvs_set_blob at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/nvs_flash/src/nvs_api.cpp:547
  #20 0x400df10a:0x3ffb41f0 in wifi_nvs_set at ??:?
  #21 0x400dd158:0x3ffb4210 in wifi_station_save_ap_channel at ??:?
  #22 0x400eb689:0x3ffb4230 in cnx_auth_done at ??:?
  #23 0x400de76b:0x3ffb42b0 in ieee80211_auth_done at ??:?
  #24 0x4010a497:0x3ffb42d0 in wpa_neg_complete at ??:?
  #25 0x401086aa:0x3ffb42f0 in wpa_parse_kde_ies at ??:?
  #26 0x401097cc:0x3ffb4310 in eapol_txcb at ??:?
  #27 0x400fe3e7:0x3ffb43a0 in ppProcTxDone at ??:?
  #28 0x40100666:0x3ffb43d0 in ppTask at ??:?
  #29 0x4008d1ca:0x3ffb4400 in vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c:355 (discriminator 1)

Rebooting...
ets Jul 29 2019 12:21:46

[igrr]

I think the root cause is that behavior of some IDF drivers depends on CONIFG_SPIRAM_USE_MALLOC Kconfig option. For example, if this option is enabled, Wi-Fi driver will use a slightly more complex code to allocate queues in internal RAM: https://github.com/espressif/esp-idf/blob/3e370c4296247b349aa3b9a0076c05b9946d47dc/components/esp_wifi/esp32/esp_adapter.c#L125-L153.
The authors of that code assumed that if CONFIG_SPIRAM_USE_MALLOC is disabled, it means that malloc only returns memory from internal RAM.

I think the solution for this problem is to enable CONIFG_SPIRAM_USE_MALLOC by default — which is done in arduino-esp32 2.0.0 and later releases. There are some downsides to that, like increased heap usage, but there is work in progress to fix this issue.

[VojtechBartoska]

@SuGlider PTAL on this related issue.

[SuGlider]

@jgamble-simple

Unfortunatelly 2.0.0 is not availble to PlaformIO at this moment. But it can be used with Arduino IDE.

I created a repository with a new Arduino Core 2.0.0 that can be tested.
It is part of modifications related to heap and binary allocation (#5630), and it also has CONIFG_SPIRAM_USE_MALLOC enabled.

https://github.com/espressif/arduino-esp32/tree/mem-optimized

In order to install it as a separated board for testing, please follow the instructions from
https://docs.espressif.com/projects/arduino-esp32/en/latest/installing.html
The only difference here would be to clone mem-optimized branch instead.

git clone -b mem-optimized https://github.com/espressif/arduino-esp32 esp32

In case it works fine for a number of users, we will commit it for the next ESP32 Arduino release.
Please test it with your project and let me know.

[jgamble-simple]

@SuGlider

Actually, I figured out how to get arduino version 2.0.0 for the esp-idf by modifying my ini file:

platform = https://github.com/platformio/platform-espressif32.git#feature/arduino-upstream
platform_packages =
  framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#2.0.0
board = esp-wrover-kit
framework = arduino

So far it works with a couple tweaks to the existing code.....

[mrdc]

So far it works

@igrr
@jgamble-simple

Hi!

I've tried your solution and heap haven't increased compared to pre-2.0.0 version :(
CONFIG_SPIRAM_USE_MALLOC=y is in sdkconfig by default in 2.0.0+, but getFreeHeap shows 30k free after boot. Even less than with 1.0.5 version, which showed 65k free for the same firmware.

[jgamble-simple]

You still have to call heap_caps_malloc_extmem_enable(0); which will enable the external heap for a malloc call.

The default is to make all malloc calls use internal heap. The argument is the smallest allocation that would still be allocated on internal heap.

Also I am not sure what the getFreeHeap function returns, so I am not sure if that is supposed to reflect a larger heap.

[SuGlider]

@jgamble-simple
getFreeHeap returns actually free DRAM and IRAM. Only DRAM accounts for heap.
In order to get an accurate free heap amount, please use heap_caps_get_free_size(MALLOC_CAP_8BIT)

For more information:
hhttps://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/system/mem_alloc.html#memory-capabilities

[mrdc]

In order to get an accurate free heap amount, please use heap_caps_get_free_size(MALLOC_CAP_8BIT)

heap_caps_get_free_size(MALLOC_CAP_8BIT) shows 4394040.

It looks like ssl_client.cpp (WiFiClientSecure) uses getFreeHeap (getHeapSize shows 219044) :

[ssl_client.cpp:56] start_ssl_client(): Free internal heap before TLS 177812
...
[ssl_client.cpp:240] start_ssl_client(): Free internal heap after TLS 138528

and it fails when FreeHeap is low. Does it mean that WiFiClientSecure (from arduino-esp32 2.0) uses only internal RAM and not PSRAM?

[lbernstone]

and it fails when FreeHeap is low. Does it mean that WiFiClientSecure (from arduino-esp32 2.0) uses only iternal RAM and not PSRAM?

WiFiClientSecure is using the hardware accelerated encryption functions. AFAIK, there is no way to use the PSRAM to feed/collect the hardware accelerator. There is a way to use DMA with the HW AES, so it might be possible to keep this off the heap, but that would be beyond my skills, and would likely need some changes in IDF.

[igrr]

Note that mbedTLS memory allocation policy can be configured, currently it is fixed to "internal RAM": https://github.com/espressif/esp32-arduino-lib-builder/blob/51a3ba2bcd1eaad61ff9d67de5cc93b5baeeb9f8/sdkconfig.esp32#L1152-L1155

[lbernstone]

Very interesting. I may have to play around with that at some point.
https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/kconfig.html#config-mbedtls-mem-alloc-mode

[mrdc]

Note that mbedTLS memory allocation policy can be configured, currently it is fixed to "internal RAM"

I've just checked it (framework-arduinoespressif32 0.0.0+sha.44c1198):
CONFIG_MBEDTLS_EXTERNAL_MEM_ALLOC=y

# CONFIG_MBEDTLS_INTERNAL_MEM_ALLOC=y
CONFIG_MBEDTLS_EXTERNAL_MEM_ALLOC=y
# CONFIG_MBEDTLS_DEFAULT_MEM_ALLOC is not set
# CONFIG_MBEDTLS_CUSTOM_MEM_ALLOC is not set
CONFIG_MBEDTLS_SSL_MAX_CONTENT_LEN=16384
# CONFIG_MBEDTLS_ASYMMETRIC_CONTENT_LEN is not set
# CONFIG_MBEDTLS_DYNAMIC_BUFFER is not set
# CONFIG_MBEDTLS_DEBUG is not set

[ 26130][V][ssl_client.cpp:59] start_ssl_client(): Free internal heap before TLS 171639
...
[ 27635][V][ssl_client.cpp:294] start_ssl_client(): Free internal heap after TLS 132607

and
CONFIG_MBEDTLS_INTERNAL_MEM_ALLOC=y

CONFIG_MBEDTLS_INTERNAL_MEM_ALLOC=y
# CONFIG_MBEDTLS_EXTERNAL_MEM_ALLOC=y
# CONFIG_MBEDTLS_DEFAULT_MEM_ALLOC is not set
# CONFIG_MBEDTLS_CUSTOM_MEM_ALLOC is not set
CONFIG_MBEDTLS_SSL_MAX_CONTENT_LEN=16384
# CONFIG_MBEDTLS_ASYMMETRIC_CONTENT_LEN is not set
# CONFIG_MBEDTLS_DYNAMIC_BUFFER is not set
# CONFIG_MBEDTLS_DEBUG is not set

[ 26141][V][ssl_client.cpp:59] start_ssl_client(): Free internal heap before TLS 171639
...
[ 27642][V][ssl_client.cpp:294] start_ssl_client(): Free internal heap after TLS 132607

Gives the same heap size...

[igrr]

@mrdc Could you please clarify where have you specified CONFIG_MBEDTLS_EXTERNAL_MEM_ALLOC=y? Did you rebuild arduino-esp32 libraries using arduino-lib-builder?

[mrdc]

Could you please clarify where have you specified CONFIG_MBEDTLS_EXTERNAL_MEM_ALLOC=y?

Sure, here you go: \.platformio\packages\framework-arduinoespressif32@src-2567d2800cf655927e3f141014aee93c\tools\sdk\esp32\sdkconfig

Did you rebuild arduino-esp32 libraries using arduino-lib-builder?

Not yet, have to check first how to do it :)