Skip to content

Task watchdog getting triggered when opening OTA partition #3775

Closed
@dflogeras

Description

@dflogeras

Hardware:

Board: ESP32 Dev Module
Core Installation/update date: 1.0.4
IDE name: Arduino IDE
Flash Frequency: 80Mhz
PSRAM enabled: no
Upload Speed: 921600
Computer OS: Linux

Description:

We are noticing an issue on some modules (so far 2/4) whereby calling esp_ota_begin() can stall the ipc0 task for too long, and cause a watchdog reset.

In case it matters, we are compiling with CPU freq 80MHz, and using the minimal SPIFFS w OTA partition scheme.

Initially, we thought that the arduino LoopTask was causing the watchdog timing out, so we moved the calls to esp_ota_get_next_update_partition() and esp_ota_begin() into their own thread to let the LoopTask run freely. This does not solve it, as the task that watchdogs is actually ipc0 as shown:

E (13210) task_wdt: Task watchdog got triggered. The following tasks did not re:
E (13241) task_wdt: - IDLE0 (CPU 0)
E (13374) task_wdt: Tasks currently running:
E (13374) task_wdt: CPU 0: ipc0
E (13374) task_wdt: CPU 1: IDLE1
E (13374) task_wdt: Aborting.
abort() was called at PC 0x400dbbb7 on core 0

Putting a JTAG on and inspecting the ipc0 task, its callstack is as follows:

[Switching to thread 9 (Thread 1073446484)]
#0 0x40083df2 in spi_flash_op_block_func (arg=0x0) at /home/runner/work/esp32-a
rduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/spi_flash/cache_
utils.c:82
82 in /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder
/esp-idf/components/spi_flash/cache_utils.c
(gdb) where
#0 0x40083df2 in spi_flash_op_block_func (arg=0x0) at /home/runner/work/esp32-a
rduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/spi_flash/cache_
utils.c:82
#1 0x40082c6a in ipc_task (arg=0x0) at /home/runner/work/esp32-arduino-lib-buil
der/esp32-arduino-lib-builder/esp-idf/components/esp32/ipc.c:62
#2 0x40091270 in vPortTaskWrapper (pxCode=0x40082c08 <ipc_task>, pvParameters=0
x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp
-idf/components/freertos/port.c:143

Working boards seem to still take over 4 seconds to return from esp_ota_begin(). To test the theory that we have some boards that may have slower flash than the others, we built our own SDK and changed the task watchdog timer from 5s to 10s, and that works around the issue. The misbehaving boards actually take about 5.2 seconds to call esp_ota_begin(). The problem also seems to be affected negatively when BLE is initialized (as in the minimal example below).

I'd really prefer not to have to build our own SDK from a bunch of master branches. Is there any way to get the arduino released SDK to raise the watchdog timeout limit? If not, is there anyway to check out EXACTLY the sources used to re-create a numbered release? I only know about the tools in https://github.com/espressif/esp32-arduino-lib-builder which don't seem to be versioned.

Sketch: (leave the backquotes for code formatting)

#include <esp_ota_ops.h>
#include <BLEServer.h>
#include <BLEDevice.h>
#include <BLE2902.h>

BLEServer* ble_server( nullptr );
BLEService* ble_service( nullptr );
BLECharacteristic* tx_characteristic( nullptr );
BLECharacteristic* rx_characteristic( nullptr );
BLECharacteristic* proto_characteristic( nullptr );

static const std::string service_uuid( "6e400001-b5a3-f393-e0a9-e50e24dcca9e" );
static const std::string characteristic_rx_uuid( "6e400002-b5a3-f393-e0a9-e50e24dcca9e" );
static const std::string characteristic_tx_uuid( "6e400003-b5a3-f393-e0a9-e50e24dcca9e" );

static const std::string characteristic_protocol_uuid( "6e400004-b5a3-f393-e0a9-e50e24dcca9e" );


static const esp_partition_t* partition;
static esp_ota_handle_t handle;
SemaphoreHandle_t open_sem;

void open_task( void* arg ) {

  while( 1 ) {

    xSemaphoreTake( open_sem, portMAX_DELAY );

    Serial.println( "opening" );
    const int64_t tic( esp_timer_get_time() );

    partition = esp_ota_get_next_update_partition( 0x0 );
    esp_ota_begin( partition, OTA_SIZE_UNKNOWN, &handle );
    Serial.println( "done opening" );
    const int64_t toc( esp_timer_get_time() );
    Serial.println( (unsigned int)(toc-tic ));
  }
}

void setup() {
  // put your setup code here, to run once:

  Serial.begin( 115200 );

  open_sem = xSemaphoreCreateBinary();
  xTaskCreate( open_task, "ota", 4096, nullptr, 5, NULL );

  BLEDevice::init( "tester" );

  ble_server = BLEDevice::createServer();

  ble_service = ble_server->createService( service_uuid );

  tx_characteristic = ble_service->createCharacteristic( characteristic_tx_uuid,
                                                         BLECharacteristic::PROPERTY_NOTIFY );

  tx_characteristic->addDescriptor(new BLE2902());

  proto_characteristic = ble_service->createCharacteristic( characteristic_protocol_uuid,
                                                            BLECharacteristic::PROPERTY_READ );

  rx_characteristic = ble_service->createCharacteristic( characteristic_rx_uuid,
                                                         BLECharacteristic::PROPERTY_WRITE );


  ble_service->start();

  proto_characteristic->setValue( "2.0" );
  proto_characteristic->notify();

  ble_server->getAdvertising()->start();

  xSemaphoreGive( open_sem );
}

void loop() {
  // put your main code here, to run repeatedly:

  Serial.println( "here" );
  delay( 250 );
}

Debug Messages:

Nothing related to SPI flash, just BLE initialization:

[V][BLEDevice.cpp:76] createServer(): >> createServer
[V][BLEServer.cpp:281] registerApp(): >> registerApp - 0
[V][FreeRTOS.cpp:189] take(): Semaphore taking: name: RegisterAppEvt (0x3ffde95p
[V][FreeRTOS.cpp:198] take(): Semaphore taken:  name: RegisterAppEvt (0x3ffde95p
[V][FreeRTOS.cpp:63] wait(): >> wait: Semaphore waiting: name: RegisterAppEvt (p
[D][BLEDevice.cpp:102] gattServerEventHandler(): gattServerEventHandler [esp_gan
[V][BLEUtils.cpp:1530] dumpGattServerEvent(): GATT ServerEvent: Unknown
[V][BLEUtils.cpp:1720] dumpGattServerEvent(): dumpGattServerEvent: *** NOT CODE*
[V][BLEServer.cpp:144] handleGATTServerEvent(): >> handleGATTServerEvent: Unknon
[V][FreeRTOS.cpp:143] give(): Semaphore giving: name: RegisterAppEvt (0x3ffde95p
[V][BLEServer.cpp:271] handleGATTServerEvent(): << handleGATTServerEvent
[V][FreeRTOS.cpp:77] wait(): << wait: Semaphore released: name: RegisterAppEvt >
[V][BLEServer.cpp:285] registerApp(): << registerApp
[V][BLEDevice.cpp:83] createServer(): << createServer
[V][BLEServer.cpp:67] createService(): >> createService - 6e400001-b5a3-f393-e0e
[V][FreeRTOS.cpp:189] take(): Semaphore taking: name: CreateEvt (0x3ffdfe28), oe
[V][FreeRTOS.cpp:198] take(): Semaphore taken:  name: CreateEvt (0x3ffdfe28), oe
[V][BLEService.cpp:60] executeCreate(): >> executeCreate() - Creating service (e
[V][FreeRTOS.cpp:189] take(): Semaphore taking: name: CreateEvt (0x3ffdff20), oe
[V][FreeRTOS.cpp:198] take(): Semaphore taken:  name: CreateEvt (0x3ffdff20), oe
[V][FreeRTOS.cpp:63] wait(): >> wait: Semaphore waiting: name: CreateEvt (0x3ffe
[D][BLEDevice.cpp:102] gattServerEventHandler(): gattServerEventHandler [esp_gan
[V][BLEUtils.cpp:1530] dumpGattServerEvent(): GATT ServerEvent: Unknown         
[V][BLEUtils.cpp:1720] dumpGattServerEvent(): dumpGattServerEvent: *** NOT CODE*
[V][BLEServer.cpp:144] handleGATTServerEvent(): >> handleGATTServerEvent: Unknon
[V][FreeRTOS.cpp:143] give(): Semaphore giving: name: CreateEvt (0x3ffdfe28), oe
[V][BLEService.cpp:194] setHandle(): >> setHandle - Handle=0x28, service UUID=6)
[V][BLEService.cpp:200] setHandle(): << setHandle                               
[V][FreeRTOS.cpp:143] give(): Semaphore giving: name: CreateEvt (0x3ffdff20), oe
[V][BLEServer.cpp:271] handleGATTServerEvent(): << handleGATTServerEvent        
[V][FreeRTOS.cpp:77] wait(): << wait: Semaphore released: name: CreateEvt (0x3f>
[V][BLEService.cpp:76] executeCreate(): << executeCreate                        
[V][FreeRTOS.cpp:63] wait(): >> wait: Semaphore waiting: name: CreateEvt (0x3ffe
[V][FreeRTOS.cpp:77] wait(): << wait: Semaphore released: name: CreateEvt (0x3f>
[V][BLEServer.cpp:83] createService(): << createService                         
[V][BLEService.cpp:222] addCharacteristic(): >> addCharacteristic()             
[D][BLEService.cpp:225] addCharacteristic(): Adding characteristic: uuid=6e40008
[V][BLEService.cpp:237] addCharacteristic(): << addCharacteristic()             
[V][BLECharacteristic.cpp:68] addDescriptor(): >> addDescriptor(): Adding UUID: 
[V][BLECharacteristic.cpp:70] addDescriptor(): << addDescriptor()               
[V][BLEService.cpp:222] addCharacteristic(): >> addCharacteristic()             
[D][BLEService.cpp:225] addCharacteristic(): Adding characteristic: uuid=6e40008
[V][BLEService.cpp:237] addCharacteristic(): << addCharacteristic()             
[V][BLEService.cpp:222] addCharacteristic(): >> addCharacteristic()             
[D][BLEService.cpp:225] addCharacteristic(): Adding characteristic: uuid=6e40008
[V][BLEService.cpp:237] addCharacteristic(): << addCharacteristic()             
[V][BLEService.cpp:134] start(): >> start(): Starting service (esp_ble_gatts_st8
[V][BLECharacteristic.cpp:79] executeCreate(): >> executeCreate()               
[D][BLECharacteristic.cpp:90] executeCreate(): Registering characteristic (esp_8
[V][FreeRTOS.cpp:189] take(): Semaphore taking: name: CreateEvt (0x3ffded3c), oe
[V][FreeRTOS.cpp:198] take(): Semaphore taken:  name: CreateEvt (0x3ffded3c), oe
[V][FreeRTOS.cpp:63] wait(): >> wait: Semaphore waiting: name: CreateEvt (0x3ffe
[D][BLEDevice.cpp:102] gattServerEventHandler(): gattServerEventHandler [esp_gan
[V][BLEUtils.cpp:1530] dumpGattServerEvent(): GATT ServerEvent: Unknown         
[V][BLEUtils.cpp:1720] dumpGattServerEvent(): dumpGattServerEvent: *** NOT CODE*
[V][BLEServer.cpp:144] handleGATTServerEvent(): >> handleGATTServerEvent: Unknon
[V][BLECharacteristic.cpp:591] setHandle(): >> setHandle: handle=0x2a, charactee
[V][BLECharacteristic.cpp:593] setHandle(): << setHandle                        
[V][BLECharacteristic.cpp:198] handleGATTServerEvent(): >> handleGATTServerEvenn
[V][FreeRTOS.cpp:143] give(): Semaphore giving: name: CreateEvt (0x3ffded3c), oe
[V][BLECharacteristic.cpp:452] handleGATTServerEvent(): << handleGATTServerEvent
[V][FreeRTOS.cpp:77] wait(): << wait: Semaphore released: name: CreateEvt (0x3f>
[V][BLECharacteristic.cpp:198] handleGATTServerEvent(): >> handleGATTServerEvenn
[V][BLEDescriptor.cpp:57] executeCreate(): >> executeCreate(): UUID: 00002902-0f
[V][BLECharacteristic.cpp:452] handleGATTServerEvent(): << handleGATTServerEvent
[V][FreeRTOS.cpp:189] take(): Semaphore taking: name: CreateEvt (0x3ffdf25c), oe
[V][BLECharacteristic.cpp:198] handleGATTServerEvent(): >> handleGATTServerEvenn
[V][FreeRTOS.cpp:198] take(): Semaphore taken:  name: CreateEvt (0x3ffdf25c), oe
[V][BLECharacteristic.cpp:452] handleGATTServerEvent(): << handleGATTServerEvent
[V][FreeRTOS.cpp:63] wait(): >> wait: Semaphore waiting: name: CreateEvt (0x3ffe
[V][BLEServer.cpp:271] handleGATTServerEvent(): << handleGATTServerEvent        
[D][BLEDevice.cpp:102] gattServerEventHandler(): gattServerEventHandler [esp_gan
[V][BLEUtils.cpp:1530] dumpGattServerEvent(): GATT ServerEvent: Unknown         
[V][BLEUtils.cpp:1720] dumpGattServerEvent(): dumpGattServerEvent: *** NOT CODE*
[V][BLEServer.cpp:144] handleGATTServerEvent(): >> handleGATTServerEvent: Unknon
[V][BLECharacteristic.cpp:198] handleGATTServerEvent(): >> handleGATTServerEvenn
[V][BLEDescriptor.cpp:220] setHandle(): >> setHandle(0x2b): Setting descriptor b
[V][BLEDescriptor.cpp:222] setHandle(): << setHandle()                          
[V][FreeRTOS.cpp:143] give(): Semaphore giving: name: CreateEvt (0x3ffdf25c), oe
[V][BLECharacteristic.cpp:452] handleGATTServerEvent(): << handleGATTServerEvent
[V][FreeRTOS.cpp:77] wait(): << wait: Semaphore released: name: CreateEvt (0x3f>
[V][BLECharacteristic.cpp:198] handleGATTServerEvent(): >> handleGATTServerEvenn
[V][BLEDescriptor.cpp:81] executeCreate(): << executeCreate                     
[V][BLECharacteristic.cpp:452] handleGATTServerEvent(): << handleGATTServerEvent
[V][BLECharacteristic.cpp:116] executeCreate(): << executeCreate                
[V][BLECharacteristic.cpp:198] handleGATTServerEvent(): >> handleGATTServerEvenn
[V][BLECharacteristic.cpp:79] executeCreate(): >> executeCreate()               
[V][BLECharacteristic.cpp:452] handleGATTServerEvent(): << handleGATTServerEvent
[D][BLECharacteristic.cpp:90] executeCreate(): Registering characteristic (esp_8
[V][BLEServer.cpp:271] handleGATTServerEvent(): << handleGATTServerEvent        
[V][FreeRTOS.cpp:189] take(): Semaphore taking: name: CreateEvt (0x3ffe0464), oe
[V][FreeRTOS.cpp:198] take(): Semaphore taken:  name: CreateEvt (0x3ffe0464), oe
[V][FreeRTOS.cpp:63] wait(): >> wait: Semaphore waiting: name: CreateEvt (0x3ffe
[D][BLEDevice.cpp:102] gattServerEventHandler(): gattServerEventHandler [esp_gan
[V][BLEUtils.cpp:1530] dumpGattServerEvent(): GATT ServerEvent: Unknown         
[V][BLEUtils.cpp:1720] dumpGattServerEvent(): dumpGattServerEvent: *** NOT CODE*
[V][BLEServer.cpp:144] handleGATTServerEvent(): >> handleGATTServerEvent: Unknon
[V][BLECharacteristic.cpp:591] setHandle(): >> setHandle: handle=0x2d, charactee
[V][BLECharacteristic.cpp:593] setHandle(): << setHandle                        
[V][BLECharacteristic.cpp:198] handleGATTServerEvent(): >> handleGATTServerEvenn
[V][BLECharacteristic.cpp:452] handleGATTServerEvent(): << handleGATTServerEvent
[V][BLECharacteristic.cpp:198] handleGATTServerEvent(): >> handleGATTServerEvenn
[V][FreeRTOS.cpp:143] give(): Semaphore giving: name: CreateEvt (0x3ffe0464), oe
[V][BLECharacteristic.cpp:452] handleGATTServerEvent(): << handleGATTServerEvent
[V][FreeRTOS.cpp:77] wait(): << wait: Semaphore released: name: CreateEvt (0x3f>
[V][BLECharacteristic.cpp:198] handleGATTServerEvent(): >> handleGATTServerEvenn
[V][BLECharacteristic.cpp:116] executeCreate(): << executeCreate                
[V][BLECharacteristic.cpp:452] handleGATTServerEvent(): << handleGATTServerEvent
[V][BLECharacteristic.cpp:79] executeCreate(): >> executeCreate()               
[V][BLEServer.cpp:271] handleGATTServerEvent(): << handleGATTServerEvent        
[D][BLECharacteristic.cpp:90] executeCreate(): Registering characteristic (esp_8
[V][FreeRTOS.cpp:189] take(): Semaphore taking: name: CreateEvt (0x3ffe0790), oe
[V][FreeRTOS.cpp:198] take(): Semaphore taken:  name: CreateEvt (0x3ffe0790), oe
[V][FreeRTOS.cpp:63] wait(): >> wait: Semaphore waiting: name: CreateEvt (0x3ffe
[D][BLEDevice.cpp:102] gattServerEventHandler(): gattServerEventHandler [esp_gan
[V][BLEUtils.cpp:1530] dumpGattServerEvent(): GATT ServerEvent: Unknown         
[V][BLEUtils.cpp:1720] dumpGattServerEvent(): dumpGattServerEvent: *** NOT CODE*
[V][BLEServer.cpp:144] handleGATTServerEvent(): >> handleGATTServerEvent: Unknon
[V][BLECharacteristic.cpp:591] setHandle(): >> setHandle: handle=0x2f, charactee
[V][BLECharacteristic.cpp:593] setHandle(): << setHandle                        
[V][BLECharacteristic.cpp:198] handleGATTServerEvent(): >> handleGATTServerEvenn
[V][BLECharacteristic.cpp:452] handleGATTServerEvent(): << handleGATTServerEvent
[V][BLECharacteristic.cpp:198] handleGATTServerEvent(): >> handleGATTServerEvenn
[V][BLECharacteristic.cpp:452] handleGATTServerEvent(): << handleGATTServerEvent
[V][BLECharacteristic.cpp:198] handleGATTServerEvent(): >> handleGATTServerEvenn
[V][FreeRTOS.cpp:143] give(): Semaphore giving: name: CreateEvt (0x3ffe0790), oe
[V][BLECharacteristic.cpp:452] handleGATTServerEvent(): << handleGATTServerEvent
[V][FreeRTOS.cpp:77] wait(): << wait: Semaphore released: name: CreateEvt (0x3f>
[V][BLEServer.cpp:271] handleGATTServerEvent(): << handleGATTServerEvent        
[V][BLECharacteristic.cpp:116] executeCreate(): << executeCreate                
[V][FreeRTOS.cpp:189] take(): Semaphore taking: name: StartEvt (0x3ffdeb54), owt
[V][FreeRTOS.cpp:198] take(): Semaphore taken:  name: StartEvt (0x3ffdeb54), owt
[V][FreeRTOS.cpp:63] wait(): >> wait: Semaphore waiting: name: StartEvt (0x3ffdt
[D][BLEDevice.cpp:102] gattServerEventHandler(): gattServerEventHandler [esp_gan
[V][BLEUtils.cpp:1530] dumpGattServerEvent(): GATT ServerEvent: Unknown         
[V][BLEUtils.cpp:1720] dumpGattServerEvent(): dumpGattServerEvent: *** NOT CODE*
[V][BLEServer.cpp:144] handleGATTServerEvent(): >> handleGATTServerEvent: Unknon
[V][FreeRTOS.cpp:143] give(): Semaphore giving: name: StartEvt (0x3ffdeb54), owt
[V][BLECharacteristic.cpp:198] handleGATTServerEvent(): >> handleGATTServerEvenn
[V][FreeRTOS.cpp:77] wait(): << wait: Semaphore released: name: StartEvt (0x3ff>
[V][BLECharacteristic.cpp:452] handleGATTServerEvent(): << handleGATTServerEvent
[V][BLEService.cpp:159] start(): << start()                                     
[V][BLECharacteristic.cpp:198] handleGATTServerEvent(): >> handleGATTServerEvenn
[V][BLECharacteristic.cpp:646] setValue(): >> setValue: length=3, data=322e30, e
[V][BLECharacteristic.cpp:452] handleGATTServerEvent(): << handleGATTServerEvent
[V][FreeRTOS.cpp:189] take(): Semaphore taking: name: SetValue (0x3ffe0524), ow>
[V][BLECharacteristic.cpp:198] handleGATTServerEvent(): >> handleGATTServerEvenn
[V][FreeRTOS.cpp:198] take(): Semaphore taken:  name: SetValue (0x3ffe0524), ow>
[V][BLECharacteristic.cpp:452] handleGATTServerEvent(): << handleGATTServerEvent
[V][FreeRTOS.cpp:143] give(): Semaphore giving: name: SetValue (0x3ffe0524), ow>
[V][BLEServer.cpp:271] handleGATTServerEvent(): << handleGATTServerEvent        
[V][BLECharacteristic.cpp:655] setValue(): << setValue                          
[V][BLECharacteristic.cpp:477] notify(): >> notify: length: 3                   
[D][BLECharacteristic.cpp:782] onNotify(): BLECharacteristicCallbacks           
[D][BLECharacteristic.cpp:783] onNotify(): BLECharacteristicCallbacks           
[V][GeneralUtils.cpp:296] hexDump():      00 01 02 03 04 05 06 07 08 09 0a 0b 0f
[V][GeneralUtils.cpp:297] hexDump():      -- -- -- -- -- -- -- -- -- -- -- -- --
[V][GeneralUtils.cpp:323] hexDump(): 0000 32 2e 30                             0
[V][BLECharacteristic.cpp:487] notify(): << notify: No connected clients.       
[D][BLECharacteristic.cpp:794] onStatus(): BLECharacteristicCallbacks           
[D][BLECharacteristic.cpp:795] onStatus(): BLECharacteristicCallbacks           
[I][BLEDevice.cpp:569] getAdvertising(): create advertising                     
[D][BLEDevice.cpp:571] getAdvertising(): get advertising                        
[V][BLEAdvertising.cpp:181] start(): >> start: customAdvData: 0, customScanResp0
[D][BLEAdvertising.cpp:200] start(): - no services advertised                   
[V][BLEAdvertising.cpp:241] start(): << start

Metadata

Metadata

Assignees

No one assigned

    Labels

    Status: StaleIssue is stale stage (outdated/stuck)

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions