[Bug] Unity App freezes for several seconds to minutes when an exception occurs with Crashlytics running

[Noshire]

[REQUIRED] Please fill in the following fields:

• Unity editor version: 2021.3.1f1
• Firebase Unity SDK version: 10.2.0
• Source you installed the SDK: .unitypackage
• Problematic Firebase Component: Crashlytics
• Other Firebase Components in use: Analytics
• Additional SDKs you are using: None
• Platform you are using the Unity editor on: Windows
• Platform you are targeting: iOS, Android
• Scripting Runtime: IL2CPP
• Pre-built SDK from the website

[REQUIRED] Please describe the issue here:

Crashlytics increases the time a Unity app freezes when handling an uncaught exception by several orders of magnitude.
We are struggling with this in a production app where users often quit the app because they assume it's stuck forever.

We managed to reproduce it in isolation (crashlytics-issue-repro project), and have used this isolated case to set up 3 test cases:

• Synchronous, 50 Method Callstack: A void method that calls itself 50 times recursively and then throws an Exception
• Asynchronous, 50 Method Callstack: An async Task method that calls itself 50 times recursively using await and then throws an Exception
• Synchronous, 50 Background Threads: A void method that starts 50 worker threads that idle for 1 second and immediately throws an Exception (to find out whether uploading all thread info to Crashlytics could also cause overhead)

We've ran them with Crashlytics not initialized yet ("Crashlytics OFF"), then initialized Crashlytics and ran them again ("Crashlytics ON"). Test results running on a Huawei Y6 2018 / ATU-L21 with Android 8.0.0:

	Crashlytics OFF	Crashlytics ON
Synchronous, 50 Method Callstack	00:00.015	00:05.131
Asynchronous, 50 Method Callstack	00:03.607	28:29.103
Synchronous, 50 Background Threads	00:00.109	00:05.205

Exceptions that didn't cause any perceivable stutter without Crashlytics suddenly cause a >5s freeze when Crashlytics is enabled. The most troublesome was the asynchronous callstack, though, which already incurs a 3.6s freeze without Crashlytics, but suddenly jumps to nearly half an hour of freeze with Crashlytics enabled.

Steps to reproduce:

1. Open the crashlytics-issue-repro project in Unity 2021.3.1f1
2. Import your own google-services.json
3. Ensure scripting backend is set to IL2CPP
4. Build and run on any Android phone. The issue is much more visible on slightly older phones. I've used a Huawei Y6 2018 / ATU-L21.
5. After the app starts, press the "Freeze for 3 Seconds" button as a calibration test. This simply invokes Thread.Sleep(3000), so the resulting text should show a freeze of around 3 seconds.
6. Press the three "Throw" buttons in sequence. "Throw (Simple Stack)" and "Throw (Many Threads)" should be very fast, while "Throw (Async Stack)" could lead to a short freeze.
7. Press the "Initialize Firebase" button. Crashlytics is now running.
8. Press the three "Throw" buttons in sequence. "Throw (Simple Stack)" and "Throw (Many Threads)" should now take several seconds (depending on device), while "Throw (Async Stack)" should lead to a very long freeze (almost 30 minutes on the Huawei device).

Downloadable Repro Project: https://github.com/Noshire/crashlytics-issue-repro

Code Snippets

All code can be found in the linked repository. This snippet, for example, demonstrates how we fabricate the async callstack that throws an exception deep down, which leads to a 28 minute freeze with Crashlytics enabled:

        /// <summary>
        ///     Fabricates a deep async callchain and then throws
        /// </summary>
        public async Task ThrowAsync()
        {
            await EnlargeAsyncStacktraceAndFinallyThrow(50);
        }

        [MethodImpl(MethodImplOptions.NoInlining)]
        private static async Task EnlargeAsyncStacktraceAndFinallyThrow(int count)
        {
            if (count > 0)
                // ReSharper disable once TailRecursiveCall
                await EnlargeAsyncStacktraceAndFinallyThrow(count - 1);
            else
                throw new Exception("This is a test exception.");
        }

[paulinon]

Hi @Noshire,

Could you verify if this behavior occurs when using an Android Virtual Device with similar specs? Also, could you specify any other devices that freeze for several minutes?

Additionally, I noticed that you're freezing the main thread with your implementation. Have you considered using coroutines for your project? These pause your execution and eventually return control to Unity.

[Noshire]

Thanks for the quick reply!

Could you verify if this behavior occurs when using an Android Virtual Device with similar specs?

I will run it through a few virtual devices on Monday and report back here.

Also, could you specify any other devices that freeze for several minutes?

The behaviour is noticeable on pretty much any Android device. The magnitude of the freeze seems to be vaguely related to the device's overall performance. Here's an extended table with the 2 devices I was able to test thus far:

	Huawei Y6 2018 Android 8		Samsung Galaxy S10+ Android 12
	Crashlytics OFF	Crashlytics ON	Crashlytics OFF	Crashlytics ON
Synchronous, 50 Method Callstack	00:00.015	00:05.131	00:00.005	00:00.933
Asynchronous, 50 Method Callstack	00:03.607	28:29.103	00:00.764	05:27.317
Synchronous, 1 Method Stack, 50 Background Threads	00:00.109	00:05.205	00:00.008	00:00.935

Additionally, I noticed that you're freezing the main thread with your implementation.

Can you specify where you saw that? If you're referring to the "Freeze for 3 Seconds" button, that's intended as a 'sanity check' to verify that the repro project actually reports a close-to-3-seconds freeze when actually freezing the app for 3 seconds. The "Asynchronous, 50 Method Callstack" routine uses single-threaded async-await which runs synchronously though.

This is well-represented in the test results: using any of the buttons without Crashlytics enabled results in a very small frame time. With Crashlytics enabled, the frametime for executing these test methods would actually remain unchanged, but what I'm measuring is the huge frame-freeze that happens immediately after execution has finished - when Crashlytics catches the uncaught exception and processes it. This is the basic reporting method used in the repro project:

private async void ThrowAndMeasure(Func<Task> action)
{
    Stopwatch sw = Stopwatch.StartNew();

    try
    {
        await action();
    }
    catch (Exception e)
    {
        Debug.LogException(e);
    }

    sw.Stop();
    TimeSpan frameFreeze = await _measurer.MeasureNextFrameFreeze();
    TimeSpan totalTime = sw.Elapsed + frameFreeze;
    TxtStatus.text = $"Exception Thrown (Duration: {totalTime})";
}

If you run this on an Android device and output the value of sw.Elapsed individually, you will see that sw.Elapsed (the time used for running the test) is always very small; it's frameFreeze (the measurement of the next big frame spike) that scales by orders of magnitude with Crashlytics enabled.

Thanks for diving in - this is currently a big impediment for us, so we're glad this is getting actively resolved 🙏

[Noshire]

I've tested the APK on 2 physical and 3 virtual devices. The outcome is consistent: if Firebase Crashlytics is enabled:

• an unhandled exception thrown inside a moderately deep regular/sync callstack will freeze the app for ~10-350x as long as it normally would.
• an unhandled exception thrown inside a moderately deep async callstack will freeze the app for ~250-450x as long as it normally would.

As an additional input: the same freezes occur regardless of whether the device is connected to the internet - leading me to believe that the issue must be somewhere in Crashlytics' local processing of the stack trace, not in the upload.

Since I can't get the APK zipped to <25MB, I've instead uploaded it on nextcloud here:
https://nc.gbanga.ch/s/EKXPYceabX6oTFR

Here's the breakdown:

		Synchronous 50 Method Stack	Asynchronous 50 Method Stack	Synchronous 1 Method Stack, 50 Background Threads
[PHYSICAL] Huawei Y6 2018 / ATU-L21	Crashlytics OFF	00:00.015	00:03.607	00:00.109
	Crashlytics ON	00:05.131	28:29.103	00:05.205
[PHYSICAL] Samsung Galaxy S10+	Crashlytics OFF	00:00.005	00:00.764	00:00.008
	Crashlytics ON	00:00.933	05:27.317	00:00.935
[VIRTUAL] Pixel 2 API 30, 4GB Allocated RAM	Crashlytics OFF	00:00.008	00:03.077	00:00.024
	Crashlytics ON	00:01.888	12:43.396	00:02.189
[VIRTUAL] Pixel 6 API 30, 4GB Allocated RAM	Crashlytics OFF	00:00.101	00:03.627	00:00.125
	Crashlytics ON	00:01.989	21:20.248	00:02.148
[VIRTUAL] Nexus 5 API 30, 4GB Allocated RAM	Crashlytics OFF	00:00.169	00:03.053	00:00.111
	Crashlytics ON	00:01.917	17:36.471	00:02.171

@paulinon hope this helps. Let me know if any further info is required. Thanks!

[paulinon]

Thanks for the additional input, @Noshire. I was able to replicate this issue using my physical devices. Let me relay this to the team so that we could identify any underlying causes. I'll be marking this as a bug for now.

Here are my results after testing:

		Synchronous 50 Method Stack	Asynchronous 50 Method Stack	Synchronous 1 Method Stack, 50 Background Threads
POCO X3 GT	Crashlytics OFF	00:00:00.0641006	00:00:00.8509732	00:00:00.0120490
	Crashlytics ON	00:00:00.0509361	00:00:10.9412961	00:00:00.0300745
Vivo 1906	Crashlytics OFF	00:00:00.0043904	00:00:02.467878760	00:00:00.05001617
	Crashlytics ON	00:00:00.2284354	00:00:47:4598317	00:00:00.1391159