Android backtraces don't work since gimli is used as the symbolizer

[inferrna]

Got this non-informative backtrace in logcat:

12-16 22:26:30.581 12603 12619 I RustStdoutStderr: thread '<unnamed>' panicked at 'called 'Result::unwrap()' on an 'Err' value: GlutinCreationError(NoAvailablePixelFormat)', src/main.rs:74:71
12-16 22:26:30.581 12603 12619 I RustStdoutStderr: stack backtrace:
12-16 22:26:30.581 12603 12619 I RustStdoutStderr:    0: 0xdd581fee - <unknown>
12-16 22:26:30.581 12603 12619 I RustStdoutStderr:    1: 0xdd5966c6 - <unknown>
.....
12-16 22:26:30.584 12603 12619 I RustStdoutStderr:   25: 0xe957757e - <unknown>

Then, while trying

cargo apk gdb
got this

Remote communication error.  Target disconnected.: Connection reset by peer.
(gdb) quit

Found in log

12-16 22:22:10.125 31207 31207 E adbd    : failed to connect to socket 'localfilesystem:/data/data/rust.conrod_android_skeleton/debug_socket': Connection refused
12-16 22:22:10.378 31207 31207 E adbd    : failed to connect to socket 'localfilesystem:/data/data/rust.conrod_android_skeleton/debug_socket': Connection refused

The app conrod_android_skeleton was set as "Debugging app" in settings (also same result was when it wasn't set).
So, it obviously is a problem on my side. But I'd like to have an option for attaching all debug symbols right into binary.

[msiglreith]

Hi, thanks for the report!
Do you have a repository for testing it locally by chance?

[inferrna]

https://github.com/inferrna/conrod-android-skeleton

it's my tries to actualise this https://github.com/jbg/conrod-android-skeleton

[msiglreith]

Hmm the missing backtrace symbolization seems to be a limitation of the backtrace library right now, but I will look a bit more around. Apparently, the rust compiler switched from libbacktrace to gimli this year and android support seems to be lacking right now: rust-lang/backtrace-rs#351

[dvc94ch]

apparently there is support in backtrace-rs now. according to rust-lang/backtrace-rs#442 some changes to the AndroidManifest.xml are required. those should probably be added in debug builds. however I still can't get it to work

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
          xmlns:tools="http://schemas.android.com/tools"
          package="com.myapp">

    <application
            tools:replace="android:extractNativeLibs"
            android:extractNativeLibs="true" />

</manifest>

[rib]

Just to mention that I'm seeing the same issue and also tried setting android:extractNativeLibs="true" as was mentioned as a workaround under rust-lang/backtrace-rs#442 but that didn't seem to help.

I've tried with ndk 22b and 23b so far and I'm building for arm64-v8a.

As far as I could tell the library had debug symbols (checked with nm and readelf). I also double checked symbols weren't being stripped by extracting the library from the apk and checked that file had symbols.

[rib]

I'm not so sure now that backtrace-rs is really expected to support symbolizing a backtrace on Android (see rust-lang/backtrace-rs#442 (comment)) - I think it just tries to support getting the library offset for each stack frame.

Actually in my case I'm seeing that even the offsets aren't valid - they are far to large - even if I use the legacy apk packaging to ensure the shared libraries are separate files.

What I've realized I can do as a workaround though is to check for the backtrace that's automatically printed by Android itself because that at least has correct addresses / offsets. I then wrote a python script that just scrapes the offsets which can be piped to addr2line in the NDK:

#!/usr/bin/python3
import sys
import re

for line in sys.stdin:
    search = re.search('#[0-9]+ +pc +([0-9A-Fa-f]+) +', line)
    if search != None:
        print(search.group(1))

So then I can copy paste a backtrace into a file and get a backtrace with filenames, demangled functions and line numbers via:

cat backtrace.txt | py ./android-backtrace.py | 'path/to/toolchains/llvm/prebuilt/windows-x86_64/bin/aarch64-linux-android-addr2line.exe' -e ./app/src/main/jniLibs/arm64-v8a/libmain.so -f -C

A notable advantage of this approach is that it doesn't require packaged / uploaded libraries to have symbols (so long as you can reference a copy of the library with symbols when symbolizing) and doesn't require the use of extractNativeLibs.

For example with this backtrace from adb:

2022-05-10 04:54:22.928 29166-29166/? A/DEBUG:     #00 pc 0000000000021d94  /system/lib64/libc.so (abort+124)
2022-05-10 04:54:22.928 29166-29166/? A/DEBUG:     rust-mobile/ndk#1 pc 000000000199ad18  /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so
2022-05-10 04:54:22.928 29166-29166/? A/DEBUG:     rust-mobile/ndk#2 pc 0000000001997b1c  /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so
2022-05-10 04:54:22.928 29166-29166/? A/DEBUG:     rust-mobile/ndk#3 pc 00000000019979c0  /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so
2022-05-10 04:54:22.928 29166-29166/? A/DEBUG:     rust-mobile/ndk#4 pc 00000000019976e8  /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so
2022-05-10 04:54:22.928 29166-29166/? A/DEBUG:     rust-mobile/ndk#5 pc 0000000001995a88  /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so
2022-05-10 04:54:22.928 29166-29166/? A/DEBUG:     rust-mobile/ndk#6 pc 0000000001997420  /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so
2022-05-10 04:54:22.928 29166-29166/? A/DEBUG:     rust-mobile/ndk#7 pc 000000000014b604  /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so
2022-05-10 04:54:22.928 29166-29166/? A/DEBUG:     rust-mobile/ndk#8 pc 000000000014b6a8  /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so
2022-05-10 04:54:22.928 29166-29166/? A/DEBUG:     rust-mobile/ndk#9 pc 0000000000950224  /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so
2022-05-10 04:54:22.928 29166-29166/? A/DEBUG:     rust-mobile/ndk#10 pc 0000000000b69ec0  /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so
2022-05-10 04:54:22.928 29166-29166/? A/DEBUG:     rust-mobile/ndk#11 pc 0000000000a1f490  /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so
2022-05-10 04:54:22.928 29166-29166/? A/DEBUG:     rust-mobile/ndk#12 pc 0000000000b64a24  /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so
2022-05-10 04:54:22.928 29166-29166/? A/DEBUG:     rust-mobile/ndk#13 pc 0000000000b599f0  /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so
2022-05-10 04:54:22.928 29166-29166/? A/DEBUG:     rust-mobile/ndk#14 pc 0000000000b594d8  /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so
2022-05-10 04:54:22.928 29166-29166/? A/DEBUG:     rust-mobile/ndk#15 pc 0000000000b64774  /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so
2022-05-10 04:54:22.928 29166-29166/? A/DEBUG:     rust-mobile/ndk#16 pc 000000000099f968  /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so
2022-05-10 04:54:22.928 29166-29166/? A/DEBUG:     rust-mobile/ndk#17 pc 0000000001617d68  /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so
2022-05-10 04:54:22.928 29166-29166/? A/DEBUG:     rust-mobile/ndk#18 pc 000000000014fa14  /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so
2022-05-10 04:54:22.928 29166-29166/? A/DEBUG:     rust-mobile/ndk#19 pc 000000000014be74  /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so (android_main+144)
2022-05-10 04:54:22.928 29166-29166/? A/DEBUG:     rust-mobile/ndk#20 pc 0000000001966740  /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so (_rust_glue_entry+348)
2022-05-10 04:54:22.928 29166-29166/? A/DEBUG:     rust-mobile/ndk#21 pc 0000000001976654  /data/app/co.realfit.agdkbevy-g7hRl_WmsH9ztNPfd-IkzA==/lib/arm64/libmain.so

The above script will pull out:

0000000000021d94
000000000199ad18
0000000001997b1c
00000000019979c0
00000000019976e8
0000000001995a88
0000000001997420
000000000014b604
000000000014b6a8
0000000000950224
0000000000b69ec0
0000000000a1f490
0000000000b64a24
0000000000b599f0
0000000000b594d8
0000000000b64774
000000000099f968
0000000001617d68
000000000014fa14
000000000014be74
0000000001966740
0000000001976654

Which can be piped to addr2line to get:

??
??:0
std::sys::unix::abort_internal::hf2e00ab422f03a90
/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c//library/std/src/sys/unix/mod.rs:259
rust_panic
/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c//library/std/src/panicking.rs:748
std::panicking::rust_panic_with_hook::hcdc16bc1d6a90d9f
/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c//library/std/src/panicking.rs:716
std::panicking::begin_panic_handler::_$u7b$$u7b$closure$u7d$$u7d$::h80c5bec89bedcc17
/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c//library/std/src/panicking.rs:588
std::sys_common::backtrace::__rust_end_short_backtrace::hbdfac0e0f8306365
/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c//library/std/src/sys_common/backtrace.rs:138
rust_begin_unwind
/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c//library/std/src/panicking.rs:584
core::panicking::panic_fmt::h28a757e2227ef488
/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c//library/core/src/panicking.rs:143
core::result::unwrap_failed::hc5bc744b126af212
/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c//library/core/src/result.rs:1749
core::result::Result$LT$T$C$E$GT$::unwrap::hdee8c853a0eb3fec
/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/result.rs:1065
bevy_render::renderer::initialize_renderer::_$u7b$$u7b$closure$u7d$$u7d$::h0cd0ec9c415b7d46
C:\Users\Robert\src\bevy\crates\bevy_render\src\renderer/mod.rs:224
_$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::hceb4b9af4b98638b
/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/future/mod.rs:84
futures_lite::future::block_on::_$u7b$$u7b$closure$u7d$$u7d$::hcea9b5092dbbef10
C:\Users\Robert\.cargo\registry\src\github.com-1ecc6299db9ec823\futures-lite-1.12.0\src/future.rs:89
std::thread::local::LocalKey$LT$T$GT$::try_with::h5f1204b6f3f53c68
/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/thread/local.rs:413
std::thread::local::LocalKey$LT$T$GT$::with::hce01bea0642e5b3f
/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/thread/local.rs:389
futures_lite::future::block_on::hb3d920eab81bb4be
C:\Users\Robert\.cargo\registry\src\github.com-1ecc6299db9ec823\futures-lite-1.12.0\src/future.rs:79
_$LT$bevy_render..RenderPlugin$u20$as$u20$bevy_app..plugin..Plugin$GT$::build::h6259e21f62b5446d
C:\Users\Robert\src\bevy\crates\bevy_render\src/lib.rs:140
bevy_app::plugin_group::PluginGroupBuilder::finish::hc623480a2b4343d0
C:\Users\Robert\src\bevy\crates\bevy_app\src/plugin_group.rs:112
bevy_app::app::App::add_plugins::h860e64c350394fbd
C:\Users\Robert\src\bevy\crates\bevy_app\src/app.rs:799
android_main
C:\Users\Robert\src\agdk-rust\examples\agdk-bevy/src/lib.rs:38
_rust_glue_entry
C:\Users\Robert\src\agdk-rust\game-activity\src/lib.rs:735
android_app_entry
C:\Users\Robert\src\agdk-rust\game-activity/csrc/game-activity/native_app_glue/android_native_app_glue.c:208

[dvc94ch]

Interesting how you managed to symbolize backtraces from logcat crash reports. I think that getting rust backtraces is a different issue. Many years ago before gimli became the default, backtraces did work, and they should work again.

[rib]

Yeah, at the very least it would be good if backtrace-rs could determine valid library offsets - ideally without requiring legacy apk packing. I guess I could live without the on-device symbolizing considering the implicit need to package libraries with symbols which in turn makes packaging and uploading slower for testing.

Presumably for some versions of Android it can work with the extractNativeLibs workaround, or else maybe there has been a regression somewhere in backtrace-rs / gimli because for all the backtraces i've seen people posting for Android issues they all look to have bogus offsets.

Maybe a small thing but I think it's more awkward to scrape the system-provided backtrace from adb than it would be to scrape a Rust backtrace in situations where you're filtering the adb log for a single process (just because the system backtrace gets dumped by another system process).

[AuTsing]

Is there any solution?

[peterdk]

Having the same issue here. No filled traces for the Rust code. (Android 13, NDK 25, ARM64)

[MarijnS95]

This should be "fixed" for panic_log by rust-lang/backtrace-rs#415 (and obviated by rust-lang/backtrace-rs#656 since backtrace 0.3.74 / Rust 1.82), and if you use Sentry / findshlibs, also gimli-rs/findshlibs#65.

But if you use xbuild, we don't actually set __ANDROID_API__ to enable these workarounds, resulting in <unknown> symbolication. That should be fixed with rust-mobile/xbuild#209 (but as said should not affect Rust >=1.82).

Please test this out and let me know! @AuTsing and @peterdk, your reports are more recent than those workarounds, could you let us know if you were using xbuild at that time?

Note that when this issue was opened, those workarounds were not yet available (they are from 2021, this issue is from 2020). The target/platform level should have always been passed in cargo-apk.