Android NDK r25b changes will break developers using r22b or older

[alex-pinkus]

Discussion continued from #102332.

Background

Changes that were merged in #102332, slated for 1.66.0, will update the CI build scripts so that all Android targets¹ use the r25b release² of the Android NDK to build Rust binaries, moving from r15c. The r25b release is the newest version of the NDK at time of writing (Oct 2022), and is a Long Term Support release (LTS)³. r15c predates LTS designation, is over five years old, and is past the end of its support life. Rust's Android platform support document⁴, merged on 24 September 2022, indicates support for the most recent LTS release of the NDK, so the changes in #102332 are in line with that document.

Currently, Rust developers who wish to use an android-ndk toolchain newer than r22b must use build-time hacks to do so. This is because the Rust standard library uses libgcc for its unwinder implementation on Android, but libgcc is not included in new versions of the NDK. When using r23b or newer in a Rust library, a developer will see ld: error: unable to find library -lgcc. #85806 added logic to select between libgcc or libunwind, but this logic runs when building Rust itself, not when compiling the downstream application, so it is mainly useful with the nightly flag -Z build-std. Since older versions of the NDK do not have libunwind, and newer versions do not have libgcc, support for r23 (and newer) is mutually exclusive from support for r22 (and older), unless a developer applies a manual workaround⁵. This workaround, for building r25b, with a binary release that uses libgcc⁶, involves a fake libgcc.a containing INPUT(-lunwind). One common form of this workaround⁷ is used in the ring crate and referenced from setup scripts for flutter_rust_bridge; a slightly modified form can be found in cargo-apk.

Problem

When the updated CI script lands in a stable release, the Rust binary releases for Android targets will compute has_unwind=true, and link against libunwind instead of libgcc, causing every project that builds using NDK version r22b or older to fail with the message error: cannot find -lunwind. Because building with r23b or newer requires either nightly Rust or a build-script workaround, it is likely that most Android projects using Rust today are using r22b or older and will therefore be broken by this change. One comment notes that Firefox will be broken⁸; a quick GitHub search also uncovers libraries like openssl-src among others that build using older NDK versions on CI.

Depending on the version that a project must be upgraded from, going to r23b can be a non-trivial effort. The process for obtaining a toolchain changed in r19⁹, so a developer must rewrite their CI compilation scripts. One commenter noted seeing compilation issues in a codebase containing C++ headers¹⁰, although they were able to resolve the issues. These are not reasons to never upgrade, but they indicate that a surprise NDK upgrade may be unwelcome work requiring more than just a version number bump.

Options

Currently, an undetermined (but probably large) subset of the Android Rust ecosystem will find itself broken on December 15th, when Rust version 1.66.0 comes out. What should be done about it? Some ideas, in no particular order:

1. Roll back to version r15c by reverting the change

Reverting the change is the simplest option, and buys an indefinite amount of time to define a more graceful deprecation strategy. Doing so, however, prolongs the usage of a build tool that's well past end of life, and means that any features introduced in the last 5 years aren't available to Rust developers.

2. Downgrade to a newer toolchain that is older than r23

With a net-new change to use r21e (newest compatible previous LTS version, released January 2021) or r22b (released March 2021)¹¹, the binaries produced by CI could take advantage of a significantly newer toolchain and retain compatibility with older toolchains downstream. A Rust binary compiled using r22b can be linked into an application using a toolchain as old as r10e (and likely older, but I didn't have an older NDK at hand).

However, both r21e and r22b are still considered obsolete and unsupported. Upgrading to one of these versions means that breakage will still occur later, when we do decide to upgrade.

3. Warn users about breakage over a longer time period

On the r25b pull request, @jonhoo mentioned that a blog post would help alert users to the change. @jonhoo compared this to the change in minimum linux-gnu versions that happened in August¹² and the accompanying announcement¹³. That change was pre-warned over several releases; in order to take a similar action here, the r25b PR would need to be reverted. While this impacts just the Android targets, rather than tier-1 linux targets as the glibc bump did, the impact on those developers is much more severe. By raising awareness of the issue and defining a timeline for the change, the blog post would help to reduce impact and limit unpleasant surprises.

4. Make no change, but still warn users about the looming breakage

Even if no action is taken to delay this change, a blog post will be a prudent way to alert developers that the change is happening in six weeks. If it is safe to recommend the INPUT(-lunwind) workaround for broad use, the blog post could recommend immediate migration to r23 or newer before the change hits stable.

---

Conclusion

As time marches on, developer tools evolve, and long-term support versions fade into unsupported obsolescence. It's natural and expected for Rust to drop support for older releases of the Android NDK, as with any other toolchain or environment that has passed end of life.

However, the current implementation will require a big-bang migration to r23+ for any project wishing to adopt 1.66.0. Some developers may have adopted r23, if they are attentive to Rust community workarounds, or brave enough to run their CI only on nightly. It's likely that many have not. In the interest of preserving existing project compatibility, it may be prudent to attempt a graceful migration to libunwind wherever possible, which can be done after still upgrading to a version that's newer than the current one. By giving developers ample warning that allows them to adopt today's workaround, the Rust project can keep the ecosystem current without causing sudden surprises.

Footnotes

1. Android targets: arm-linux-androideabi, armv7-linux-androideabi, thumbv7neon-linux-androideabi-ndk, i686-linux-android-ndk, aarch64-linux-android, x86_64-linux-android ↩

2. Android NDK r25 revision history. ↩

3. NDK release process, detailing the implications of LTS releases. ↩

4. Platform support for *-linux-android and *-linux-androideabi and PR: Add a platform support document for Android #101780 ↩

5. libunwind availability in the Android NDK: ↩

6. Apparent first attestation of the INPUT(-lunwind) workaround ↩

7. echo "INPUT(-lunwind)" search results on GitHub ↩

8. Comment from @glandium indicating Firefox breakage ↩

9. Standalone toolchain (deprecated) documentation ↩

10. Comment reporting compilation errors caused by the upgrade ↩

11. In order to use r22b, the logic in library/unwind/build.rs will also need to prefer libgcc over libunwind, since r22b includes libunwind.a within the C++ STL in arm-linux-androideabi only, for reasons that are not clear. ↩

12. https://github.com/rust-lang/rust/pull/95026 ↩

13. Increasing the glibc and Linux kernel requirements ↩

[Mark-Simulacrum]

Nominating for compiler discussion.

I would probably be somewhat in favor of fully rolling back the bump on soon to be beta 1.66 (we branch today), and landing a bump (edit: on master) only to r21/r22 so we retain compatibility for slightly longer.

We should in parallel issue a blog post with our strategy for this bump, and ideally a strategy for future bumps. It looks like Android's LTS releases are only supported until the next one is issued (once a year roughly) which means there's not really an overlap period in which we can stick with a supported copy until user's migrate. This situation definitely reminds me of our similar challenges on BSDs.

[tmandry]

I see a couple issues here:

Difficulty in moving between ndk versions

I think we should strive to match the experience of the ndk itself here. In particular if there is no ndk release with both libgcc and libunwind, the Rust project shouldn't go out of its way to ease the upgrade experience beyond giving people enough warning. (If someone wants to contribute such a soft transition path and it's not a lot of maintenance overhead, that's fine.)

Still, matching the ndk experience requires being in line with the expected upgrade cadence of ndk releases. I think this is ultimately up to the Android target maintainers.

Tying rustup releases to ndk and target API levels

It doesn't seem ideal to have your version of Rust coupled to your target API level, especially on a frequently updating platform such as Android. (Unless you can mix code targeting different API levels in the same binary, in which case it might be fine?) I expect to see similar issues with Fuchsia, where the goal is to update the platform even more frequently.

Thinking out loud a bit, this makes me wonder if we should just point people toward -Z build-std and stabilize the functionality in some capacity. This gives people maximum flexibility to mix and match ndk versions, set target API levels, and manage upgrade paths on their own. On my machine it's not that much overhead (+~17s to initial build time), but I would like to be conscious of the impact to people who are building on slower laptops.

[alex-pinkus]

On point 1: Totally agree that the Rust project isn't the right place to ease the android-ndk upgrade experience in general. Is there clear direction somewhere on what "enough warning" means? With the glibc version bump, the blog post was published almost two months in advance.

On point 2: In the general case, at least, rustup does fairly well with staying uncoupled from ndk versions. As I mentioned, a distribution of rust that uses ndk r22b will build fine against an application using as old as r10e, and possibly older. This weird situation is more a problem of the android-ndk than it is a problem of anything Rust has done. (As a side note, with target API levels, I understand the story to be a bit more complex, with the current min of android-14 changing to android-19 with this version bump, which restricts the possible targets the application can declare, but users can declare a lower min or target if their ndk supports it, even if that causes issues for the stdlib).

Rust is fairly unique here because, if you combine points 1 and 2, every project that uses Rust has a dependency by definition that will break them when it upgrades. C++ doesn't have this problem because their stdlib gets distributed as part of the ndk. Theoretically, a random C++ library might also have this problem if they link against libgcc, but no one library is going to have the kind of reach across C++ that the Rust stdlib has across Rust. It's also not uncommon in my experience for Android C++ projects to download their dependencies from source anyway (fresco is a good example) and build them directly.

Pointing people toward a stabilized version of -Z build-std definitely seems like a preferred solution here. Initial builds are going to have other far greater bottlenecks, and as an added bonus, Android developers are often also binary-size-conscious which build-std would help with. It's not clear to me whether you're suggesting that as a prerequisite for the ndk-bump or just as a useful tool in the future. Is build-std far enough along that some form of it could be stabilized in the right timeframe?

[apiraino]

I think there is enough meat here to fire up an MCP FCP. Similar situation were handled that way. An MCP was suggested in the first place by @jyn514 in this comment.

Anyway, this will be discussed tomorrow. Between the previous discussion on Zulip and this issue comments, I think there is already great context provided.