[Feature Req] Auto-detect SPIFFS/partition layout

[TD-er]

For ESPEasy I have a lot of builds defined.
The number of builds can be more than halved by not having separate builds for 1M, 2M, 4M etc.
Lately I also added test builds for 4M2M layout (4M flash, 2M SPIFFS) and when rolled out to other build definitions the number of build environments will increase even more.

Also the chance of loading the wrong image on a node does increase with all these options.

Would it be possible to have the sketch detect the flash size and detect the location of the SPIFFS, instead of hard-coded in the build?

Then I can add a number of best practice offsets for different setups and chose the best one based on the found hardware when a SPIFFS has to be created (or LittleFS) and when there is one already present just use that one.

This will remove all the flash size related builds and make it just about selecting the processor + set of plugins you want.

If this is already possible, where can I find information on how to set it?

I guess the core libraries also need some fixed offsets for things like WiFi settings and the EPROM area. So these should then also be auto-probed and offer some function call to query their locations.

[d-a-v]

Very interesting

Flash size: Esp.h: uint32_t EspClass::getFlashChipRealSize(void)
Locations: the logic in python boards.txt builder can be duplicated
SPIFFS/LittleFS: a quick scan for FS signatures can be done in the possible locations computed by the above logic.

WiFi settings addresses is independently computed by nonos-sdk-fw's by a similar way. There's nothing to do here. The only symbols that we use and defined by flash chip size / menu configuration are put to tools/sdk/ld/eagle.flash.*:

PROVIDE ( _FS_start = 0x402BB000 );
PROVIDE ( _FS_end = 0x402FB000 );
PROVIDE ( _FS_page = 0x100 );
PROVIDE ( _FS_block = 0x1000 );
PROVIDE ( _EEPROM_start = 0x402fb000 );

[TD-er]

Those WiFi settings addresses, are they computed at compile time, or run time?
If at run time, then we don't need to set the flash size at compile time for those.
If at compile time, then we still have to differentiate between flash sizes, but may be a bit more flexible in sketch/SPIFFS/LittleFS layout.

[d-a-v]

The logic in the python script is there to respect rf-cal | sdk-wifi-settings region relatively to the flash chip size to leave it untouched.
nonos-sdk computes the address and uses it at run time.

[earlephilhower]

SPIFFS does not have anything like a FSINFO block. It also happily mounts things which only nominally look like a SPIFFS FS. And there is no sane way of telling where a SPIFFS FS ends.

I would say the only sane and proper thing here is the partition layout stuff EPS32 and other chips already use.

[devyte]

@TD-er what you're asking doesn't make sense to me.

• The FS end address ia always at the end of the flash size (minus some space for wifi sdk params).
• The user has to select the FS size desired (can't be automatic)
• The FS start address is automatically calculated from flash size and FS size
• For specific boards, the flash size is known (e.g.: wemos d1 r2 is 4MB, done).
• For the generic board menu, the flash size has to be set by the user, because that is the whole point of it.

Given this, and previous comments, I'm closing this. If we're missing the point, please clarify.

[TD-er]

Given this, and previous comments, I'm closing this. If we're missing the point, please clarify.

The main problem is that the location of SPIFFS is defined at compile time, but it would be great if we can override this at run time.
As I understand from @earlephilhower we cannot currently probe for SPIFFS location, so another thing that is needed then is some indicator somewhere to get a hint on the SPIFFS location.
This could be a single byte, as it is only needed to know the number of blocks from the end, or some other enum-type.

My point is, that it would be a great feature to have the SPIFFS defined not only at compile time, but also at run time.
This way I can have a single build, which would support a setup with different flash sizes and different SPIFFS sizes.
For example the 4M/1M sketch/SPIFFS ratio for backwards compatibility and 4M/2M for newer setups.
But also the same binary for 2M/512k or any other weird combination.

[d-a-v]

Flash size and other flash-relative configuration in the menu is necessary for:

• where to store EEPROM (that can actually be calculated at run time)
• where FS ends (that can actually be calculated at run time)
• where FS starts (that can be stored in a specific location, read at runtime)

[JAndrassy]

how does SPIFFS know where FS starts?

[earlephilhower]

&__FS_start is taken at compile time.

[devyte]

@JAndrassy saif:

how does SPIFFS know where FS starts?

I said:

The FS start address is automatically calculated from flash size and FS size

There are pre-calculated ld scripts based on the menu options (board variant, FS size). The symbols are defined in there and get linked during build.
As @d-a-v said, the logic for the symbol calculations is in the boards.txt.py script.

[JAndrassy]

@JAndrassy saif:

how does SPIFFS know where FS starts?

I said:

The FS start address is automatically calculated from flash size and FS size

There are pre-calculated ld scripts based on the menu options (board variant, FS size). The symbols are defined in there and get linked during build.
As @d-a-v said, the logic for the symbol calculations is in the boards.txt.py script.

yes, yes, it clicked in my head a little later :-)

[d-a-v]

This symbol could be replaced by another one computed at run time from a fixed-address configuration structure.

[d-a-v]

Actually after internal dicussion, it is today possible for a user to instantiate manually the FS with different start/end addresses.

Arduino/cores/esp8266/spiffs_api.cpp

Lines 138 to 143 in 6e51ef0

	FS SPIFFS = FS(FSImplPtr(new spiffs_impl::SPIFFSImpl(
	FS_PHYS_ADDR,
	FS_PHYS_SIZE,
	FS_PHYS_PAGE,
	FS_PHYS_BLOCK,
	SPIFFS_MAX_OPEN_FILES)));

Arduino/libraries/LittleFS/src/LittleFS.cpp

Line 192 in 6e51ef0

FS LittleFS = FS(FSImplPtr(new littlefs_impl::LittleFSImpl(FS_PHYS_ADDR, FS_PHYS_SIZE, FS_PHYS_PAGE, FS_PHYS_BLOCK, FS_MAX_OPEN_FILES)));

So if we have an internal core API offering the same calculation as what we have in the python script, we could dynamically process EEPROM_START.

Then a pre-compiled sketch would not anymore depend on flash size (the FS size or start address and page size would be stored into a table indexed by flash size).

@TD-er would you be ready to translate the python logic to the core api ?

[TD-er]

@TD-er would you be ready to translate the python logic to the core api ?

What is the current Python code used for generating the Eagle files?
Is there any difference in numbers between LittleFS and SPIFFS?
Is there a byte where I can store information about the filesystem? Or should I use the EPROM for it?