Migrate core from NONOS SDK to FreeRTOS SDK

[devyte]

Problem Description

We recently encountered issues with the latest pre3 NONOS SDK that forced us to revert to 2.2.1, ref #5784. The issues in the NONOS sdk, and their potential solutions, are tracked there.

This issue is meant to track solving a higher-level dilemma, which is what to do about the NONOS SDK moving forward.
Espressif has explained that the NONOS SDK will be maintained for the time being, but likely not enhanced. Their plan seems to be to EOL it, as emphasis will be kept on the FreeRTOS SDK and its migration to be similar to esp-idf.

There are then several paths forward possible:

1. Migrate to NONOS SDK v3.x

Pros:

• Maintains status quo in our core as NONOS (continuity in the architecture)
• Should address the current SDK issue dilemma (sleep bug vs. stability)
• Keeps results of recent efforts to free available heap and other high profile improvements
• Potential to eventually have full control if the closed libs ever get reverse-engineered
• Buys us time (we can always change our minds and migrate to FreeRTOS later)

Cons:

• Not straightforward to migrate due to the introduction of partitions
• NONOS sdk is meant for EOL anyways, so direction is destined to be a dead end in the (possible far) future, unless the closed libs get reverse-engineered far enough

2. Migrate to a full FreeRTOS SDK (currently at v3.1)

Pros:

• Espressif seems to want to move in this direction, so support would be better than for NONOS sdk
• Availability of an underlying OS (tasks, threads, IPC, etc)
• Likely allows a better architecture design (2 contexts that are real contexts)

Cons:

• Most likely free available heap will be far less than with NONOS sdk
• Huge design effort: this SDK aims to be similar to esp-idf, but esp-idf is not well-suited for the ESP due to RAM constraints. That means that some architecture parts need to have a different design. (E.g.: too many OS tasks, and each requires its own stack and msg queue, lots of C-C++ interface points, etc)
• Huge migration effort: will require a full rewrite of a whole lot of the core code
• Probably less control over some parts, such as lwip/ssl

3. Migrate to a limited FreeRTOS SDK (no preemptive)

Pros

• May get us off NONOS SDK the fastest (or may not)

Cons:

• No guarantees of stability (in fact, the chances of a whole lot of new bugs and overall functional breakage is rather high)
• Huge effort
• We'll be on FreeRTOS, but it will be a gutted version of the OS (not real-time, cooperative, etc)
• Migrating to a full FreeRTOS architecture still means a complete rewrite of a lot of code (doing this limited migration could potentially be wasted effort)
• Likely less available heap

@d-a-v @earlephilhower @igrr and other maintainers: floor is open for discussion.

After thinking about this at length, my own preferences are as follows:

• Pursue option 1: migrate to SDK v3.x first. We can choose at a later time, once we have a stable core, to migrate to FreeRTOS, or continue on NONOS.
• Avoid option 3 at all costs. I consider it very risky with very little return, and I think that migrating to a limited version of the FreeRTOS will buy us the worst of all worlds, and at a high price at that.

[devyte]

If we are to move directly to the FreeRTOS SDK, there are some decisions to make. For starters:

1. Where will our home be?
   a. in our own repo, and import the FreeRTOS SDK repo to pick the libs/sources we need/want, like we currently do
   b. as a component in the FreeRTOS SDK repo
2. Which toolchain will we use?
   a. Build our own, like we currently do
   b. The toolchain provided by the FreeRTOS SDK repo, which comes from an Espressif server
3. Which upstream libs will we use (lwip, bearssl, freertos)?
   a. Pull and build our own versions from upstream, like we currently do
   b. Use those in the FreeRTOS SDK repo

[TD-er]

If Espressif is actually putting NonOS to EOL, is there still any reason for them to keep their core libraries closed?
And what does that say about the roadmap they plan to keep producing the ESP8266/ESP8285?
After all the ESP32 is way more powerful and has a lot more memory and its price is already quite acceptable for mass deployment.
Does the current install base of ESP82xx CPUs need a newer core platform if it is making some serious development next to impossible due to lack of free memory?
For example, does FreeRTOS support SSL and is thus taking so much heap, or is the lower free heap also effectively preventing SSL support?

And to play advocate of the devil, if it takes (really) a lot of effort of both the esp8266-Arduino team and the users to convert to an Arduino platform based on FreeRTOS, is there still demand for an Arduino implementation? Why would users then still want to move to Arduino/C++ if there is already a very fast growing MicroPython community?
After all, for beginners is Python much easier to learn than C++/Arduino.
So if a change will take a lot of time to implement and make it stable, including rendering lots of examples outdated, won't that move people away to other platforms?

I think those are just about as important questions in this discussion. (I take it as a discussion ;) )

[devyte]

@TD-er The following are my own personal answers and opinions:

Is there still any reason for them to keep their core libraries closed?

Yes. I don't agree with the libs being closed, but I don't see Espressif opening that code.

What does that say about the roadmap they plan to keep producing the ESP8266/ESP8285

They will keep producing it for a long time to come, that is not in question. The ESP32 is more powerful, even the simpler versions, but it is still higher cost than the ESP, and it has its own list of issues. Migrating to the ESP32 may be an option for many users, but it is not an option for a whole lot of commercial users. In those cases, migration means complete redevelopment.
In addition, the formal stance by Espressif, per my own understanding from reading their comments, is that the nonos sdk is being given low priority for now, and that users are being encouraged to migrate to FreeRTOS. There has been mention of EOL for the SDK, but there has also been mention of of a long support time still to come.
In contrast, the FreeRTOS sdk is being given high priority for fixes and enhancements. It is from this that I read the situation as Espressif trying to consolidate their development/maintenance efforts behind the FreeRTOS sdk.

does FreeRTOS support SSL

That is like asking whether the nonos sdk supports ssl. The support is provided by our Arduino core due to integration of bearssl, and not by the nonos sdk. In the case of FreeRTOS, there is ssl support as a component, but if we migrate to FreeRTOS sdk we'd evaluate whether we use that (wolfssl/mbedtls), or port our own bearssl integration.

is there still demand for an Arduino implementation?

yes, in fact support for a FreeRTOS-based core was requested a long time ago.

Why would users then still want to move to Arduino/C++ if there is already a very fast growing MicroPython community?

The MicroPython fw offers very little available heap for the user app. The last time I checked, it was something like 20KB less than our core. i think they still have a long way to go in that regard.

So if a change will take a lot of time to implement and make it stable ... won't that move people away to other platforms?

That may be, which is why I'm of the opinion we should achieve a stable nonos core version first, with all big issues resolved, and then freeze there. Once that happens, we should start development on the core based on FreeRTOS SDK

I'd like to clarify that this issue is not meant for a general discussion about the future of the ESP, or ESP vs. ESP32, or C++ vs. Py, etc. This issue is meant for discussing how we should proceed with our core development to assure that we have a viable path forward before the nonos sdk is EOL'd, should that happen. That means we should decide what to do and proceed accordingly ASAP.
We could stick with NONOS and reverse engineer the closed libs and achieve development freedom. Or we could migrate to FreeRTOS sdk and assure support. Or something else.

[hreintke]

The positive on using FreeRTOS is that compatibility with esp32 is improved/better possible.
Maybe even code sharing between the two frameworks, but application migration should benefit.

[TerryE]

HI, I am the lead Lua core developer on the NodeMCU firmware project. My purpose for posting here, very much as a guest, is just to note that we are going through very similar discussions, though our drivers are somewhat different in that the Lua VM (like the Python VM) imposes some heavy runtime constraints on our implementation options, so I am monitoring your discussions and I invite you to do likewise with ours.

As far as I can see at the moment (up to V3.0) the ESP8266 non-OS and RTOS variants use pretty much the same set of closed-source hardware abstraction libraries. It's just that RTOS has an extra layer over these to provide the normal RTOS services (tasks, threads, IPC, ...) and it uses a completely different make / build architecture.

Like the JS-based nodeRED, our Lua VM employs an intrinsically non-blocking, task-based, single threaded paradigm, so RTOS offers us little functional benefit, and indeed could give our less experienced module contributers plenty of scope to screw things up.

Pre 3.0 this was all moot since the ESP8266 simply didn't have enough RAM resource to run RTOS + Lua VM + sensible apps, but one of he things that Espressif has added to 3.0 is a variant of our non-aligned address exception handler (which they had initially ported to RTOS) and this allows pretty much all const data to be moved into flash. (This increased our available RAM by almost 30Kb when we first introduced it in 2016.) Now that the SDK libraries also use this, our builds have freed an extra 12Kb RAM, which might open the option of us moving to RTOS for us in the future. (Our main driver here is that our ESP32 variant is RTOS based, and this fork is causing us real maintenance overheads and problems.)

[earlephilhower]

Howdy, @TerryE . Thanks for the info!

....but one of [t]he things that Espressif has added to 3.0 is a variant of our non-aligned address exception handler...Now that the SDK libraries also use this, our builds have freed an extra 12Kb RAM....

And hence the big slowdown seen in loops/second in 2.2.x and pre3.0, potentially.

The exception handler is neat from a usability perspective, but with an 8-10x performance penalty when it hits vs. inline using the 32b-only accesses to PMEM space it is not so neat for performance.

Did we see a 12KB bump in heap free when going from 2.2.x to pre3, @d-a-v? I don't think I remember quite such a leap...

[d-a-v]

Moving from sdk2.2.1 to sdk-pre3 with an empty sketch or a simple web-server shows:

• + 1200 bytes in flash
• - 120 bytes in heap

We were not using real sdk-v3, but an intermediate version #5521.

[devyte]

Hello @TerryE , it has been a while. Welcome to the discussion here.
My initial dabblings with the ESP were with the Lua firmware, I actually learned the Lua language as a result, and I fully understand the development approach and timing restrictions you have.
After I migrated here, I've been (trying) to keep up with the discussions in your Lua repo, so I'm aware that we're in the same boat regarding the future of the NONOS sdk.

I saw that you pushed to migrate to post-v3 sdk. We migrated to the last version of the sdk prior to v3, because the partitions introduced in v3 are giving us headaches. At this point, we're deciding whether it's worth the effort to push through to a post-v3, or to migrate directly to an implementation based on some form of the latest FreeRTOS sdk v3.x.

could give our less experienced module contributers plenty of scope to screw things up

In addition to the long migration effort, this is one of the sticking points being discussed. The question has come up whether we really need a full FreeRTOS.

And here's what I find interesting: your current approach is event-driven, while ours is dual-context (similar to cofunctions), but in both cases there are timing restrictions. In your case, each task must be very short, while in our case the second context (aka CONT) must yield to the primary context (aka SYS) often enough.

My personal opinion, based on my experience with the Lua firmware, and with the Arduino firmware here, is that, if designed correctly, a firmware architecture implemented over the full FreeRTOS sdk would relax a whole lot of restrictions. My thinking is simple: Have two threads: one that handles the wifi stack and maybe TCP/IP communication, and the other running the user application.
In our case, the user application would be the CONT context.
I didn't get deep enough into the C-side of your Lua firmware, but I think that in your case the user application would be the Lua VM.

[TerryE]

And hence the big slowdown seen in loops/second in 2.2.x and pre3.0, potentially.

We don't see a big slowdown. That may be because the Lua VM is already a pretty fat Flash-based app. I also put in some instrumentation into our handler to collect exception hotspots - the old Pareto principle applies, and looked at these:

• The bootROM memxxx and strxxx routines are flash aware and will do word-aligned access so long as the data starts are word-aligned. Use -O2 rather than -Os and the xtensa GCC does this automatically.
• I added some inline asm access macros for some frequently accessed byte fields for structures often held in Flash. These removed the exception at a cost of an extra instruction per access.

Plus a few other tweaks and recodes. So in our case the exception handler didn't get called that much.

IMO, it's also important to avoid the temptation of dropping the icache from 32k to 16k, as we found that this did crater runtime performance.

I've only dabbled with the Arduino IDE using Atmel chips and not ESP ones, but I must admit that I don't see how this would mate well with a RTOS environment, but that is your business not mine. 😊