Firebase Realtime Database Observers Stop After NSPOSIXErrorDomain Code=89 Error on iOS (Xcode 15.3, Firebase SDK 11.3.0)

[pilsen]

Description

I'm developing an iOS app with SwiftUI using Xcode 15.3 and Firebase SDK 11.3.0. The app relies on Firebase Realtime Database to monitor foodEntries with observers for real-time updates.

Implementation Overview:

Observers Setup: Listens to foodEntries node for .childAdded and .childChanged events.
Connection Monitoring: Uses .info/connected reference and NWPathMonitor to track network status.
Issue:

Approximately 1 in 10 launches, either on app start or when returning from background, I encounter the following console error:

2024-10-10 17:49:08.588502+0200 MyApp[22614:2112636] [[FirebaseDatabase]] 11.3.0 - [FirebaseDatabase][I-RDB083016] Error sending web socket data: Error Domain=NSPOSIXErrorDomain Code=89 "Operation canceled" UserInfo={NSDescription=Operation canceled}.

Consequences:

Observers Stop Working: The observers' callbacks are no longer triggered, halting real-time updates.
Connection Status Untracked: The .info/connected reference doesn’t update, preventing detection of connection issues.
Partial Connectivity: Data writes may still work, but observers fail post-error.

Temporary Workaround:

Sending the app to the background and bringing it back to the foreground resets the Firebase connection, temporarily restoring observer functionality.

What I've Tried:

Reinitializing Observers: Removing and re-adding observers on connection issues.
Network Monitoring: Implemented NWPathMonitor to handle reconnections.
Error Handling: Attempted retries on data operation failures.
App Check Adjustments: Enforced and removed App Check on the server side without success (currently testing removal of setAppCheckProviderFactory & AppCheck.appCheck() on client side).

Additional Observation:

I've noticed that the issue is easier to reproduce on a 3G network compared to a stable Wi-Fi connection, though this is just a guess based on 3-4 days of testing.

Relevant Configuration:

FirebaseApp.configure()
FirebaseConfiguration.shared.setLoggerLevel(.debug)
Database.database().isPersistenceEnabled = true

class AppCheckProviderFactory: NSObject, AppCheckProviderFactory {
    func createProvider(with app: FirebaseApp) -> AppCheckProvider? {
        if #available(iOS 14.0, *) {
            return AppAttestProvider(app: app)
        } else {
            return DeviceCheckProvider(app: app)
        }
    }
}

let providerFactory = AppCheckProviderFactory()
AppCheck.setAppCheckProviderFactory(providerFactory)
_ = AppCheck.appCheck()

Problem:

It seems that the Firebase initialization code might be causing the connection issue, as .info/connected remains unchanged and observers stop receiving updates after the error.

Questions:

Cause Identification: Why does the NSPOSIXErrorDomain Code=89 "Operation canceled" error occur, and how does it affect Firebase's WebSocket connections?
Robust Error Handling: What are best practices to handle such network-related errors to maintain continuous observer functionality?
Network Monitoring Integration: How can I effectively integrate network monitoring to trigger Firebase reconnections without manual app state resets?
Alternative Solutions: Are there other approaches or Firebase configurations that can prevent this issue?

Additional Information:

Offline Persistence: Enabled.
Environment: Issue occurs on both simulators and physical devices with stable internet.
App Lifecycle Management: Observers are set up when the app becomes active and removed when it resigns active.

Any guidance or examples on resolving this Firebase connection error and ensuring reliable observer functionality would be greatly appreciated!

Reproducing the issue

import FirebaseDatabase
import Network
import Combine

class UserData: ObservableObject {
    private let monitor = NWPathMonitor()
    private let networkQueue = DispatchQueue(label: "NetworkMonitor")
    private var cancellables = Set<AnyCancellable>()
    
    @Published var isAppFirebaseConnected: Bool = false
    @Published var isAppNetworkConnected: Bool = false
    
    init() {
        setupFirebaseMonitoring()
        startNetworkMonitoring()
    }
    
    private func setupFirebaseMonitoring() {
        let db = Database.database().reference()
        let connectedRef = db.child(".info/connected")
        
        connectedRef.observe(.value) { [weak self] snapshot in
            if let connected = snapshot.value as? Bool {
                DispatchQueue.main.async {
                    self?.isAppFirebaseConnected = connected
                    if connected {
                        print("Firebase is connected.")
                        // Initialize observers here
                        self?.initObservers()
                    } else {
                        print("Firebase is disconnected.")
                        self?.removeObservers()
                    }
                }
            }
        }
    }
    
    private func startNetworkMonitoring() {
        monitor.pathUpdateHandler = { [weak self] path in
            let isConnected = path.status == .satisfied
            DispatchQueue.main.async {
                self?.isAppNetworkConnected = isConnected
                print("Network connection status: \(isConnected ? "Connected" : "Disconnected")")
                
                if isConnected && self?.isAppFirebaseConnected == false {
                    // Attempt to reconnect Firebase if needed
                    self?.reconnectFirebase()
                }
            }
        }
        monitor.start(queue: networkQueue)
    }
    
    private func initObservers() {
        let db = Database.database().reference()
        db.child("foodEntries").observe(.childAdded) { snapshot in
            // Handle new food entry
            print("New food entry added: \(snapshot.key)")
        }
        
        db.child("foodEntries").observe(.childChanged) { snapshot in
            // Handle updated food entry
            print("Food entry updated: \(snapshot.key)")
        }
    }
    
    private func removeObservers() {
        let db = Database.database().reference()
        db.child("foodEntries").removeAllObservers()
        print("Removed all Firebase observers.")
    }
    
    private func reconnectFirebase() {
        print("Attempting to reconnect Firebase...")
        Database.database().goOffline()
        DispatchQueue.main.asyncAfter(deadline: .now() + 1.0) {
            Database.database().goOnline()
        }
    }
}

Firebase SDK Version

11.3.0

Xcode Version

15.3

Installation Method

Swift Package Manager

Firebase Product(s)

App Check, Database

Targeted Platforms

iOS

Relevant Log Output

2024-10-10 17:49:08.588502+0200 MyApp[22614:2112636] [[FirebaseDatabase]] 11.3.0 - [FirebaseDatabase][I-RDB083016] Error sending web socket data: Error Domain=NSPOSIXErrorDomain Code=89 "Operation canceled" UserInfo={NSDescription=Operation canceled}.

If using Swift Package Manager, the project's Package.resolved

Expand Package.resolved snippet

Replace this line with the contents of your Package.resolved.

If using CocoaPods, the project's Podfile.lock

Expand Podfile.lock snippet

Replace this line with the contents of your Podfile.lock!

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[paulb777]

@pilsen Thanks for the detailed report. We'll investigate.

In the meantime, using version 10.26.0 or earlier may be a workaround.

[pilsen]

@pilsen Thanks for the detailed report. We'll investigate.

In the meantime, using version 10.26.0 or earlier may be a workaround.

I’ve noticed that the issue no longer occurs after disabling App Check enforcement and removing setAppCheckProviderFactory and AppCheck.appCheck() from the client side. While this might be coincidental, it suggests that the problem could be related to App Check token refreshes when the app launches or returns from the background. Just my thoughts!

[ncooke3]

Hi @pilsen, one subtle issue with the shared configuration code is that the intended AppCheck provider may not be set in time. In the worst case, App Check token retrievals that are attempted at startup may have indiscriminate behavior depending on if the setAppCheckProviderFactory API is completed before the token fetch. To avoid potential issues, the AppCheck provider factory should be set before configure (e.g. https://firebase.google.com/docs/app-check/ios/custom-provider#initialize).

- FirebaseApp.configure()
- FirebaseConfiguration.shared.setLoggerLevel(.debug)
- Database.database().isPersistenceEnabled = true

class AppCheckProviderFactory: NSObject, AppCheckProviderFactory {
    func createProvider(with app: FirebaseApp) -> AppCheckProvider? {
        if #available(iOS 14.0, *) {
            return AppAttestProvider(app: app)
        } else {
            return DeviceCheckProvider(app: app)
        }
     }
}

let providerFactory = AppCheckProviderFactory()
AppCheck.setAppCheckProviderFactory(providerFactory)
_ = AppCheck.appCheck()

+ FirebaseApp.configure()
+ FirebaseConfiguration.shared.setLoggerLevel(.debug)
+ Database.database().isPersistenceEnabled = true

I’ve noticed that the issue no longer occurs after disabling App Check enforcement and removing setAppCheckProviderFactory and AppCheck.appCheck() from the client side. While this might be coincidental, it suggests that the problem could be related to App Check token refreshes when the app launches or returns from the background. Just my thoughts!

I'm not yet sure if applying the above diff will fix the issue, but it seems possible related to your App Check findings above.

[ncooke3]

Also, is Firebase Auth also being used in this app?

[pilsen]

Also, is Firebase Auth also being used in this app?

Thanks for your insights, @ncooke3! I'm currently testing by placing AppCheck.setAppCheckProviderFactory(providerFactory) before FirebaseApp.configure(). And yes, Firebase Auth is also being used in the app. Does that increase the likelihood that placing AppCheck after configure caused these issues, as you mentioned?

[pilsen]

Also, is Firebase Auth also being used in this app?

Unfortunately, changing the App Check placement by moving AppCheck.setAppCheckProviderFactory(providerFactory) before FirebaseApp.configure() didn’t resolve the issue. I’m still observing error 89 again.

That being said, there's still a possibility that the error could occur even without App Check, though I haven’t seen it happen since removing App Check. I’ll keep monitoring to see if the issue reappears without App Check.

[ncooke3]

Hi @pilsen, thank you for the responses. I have so far been unable to reproduce. I'm using the code snippets provided above, as well as the Network Link Conditioner to replicate "3G" and "Very Bad Network" performance.

A couple questions to help potentially further our understanding:

• When the issue is reproduced and the problematic log is printed, are there any other suspicious looking RTDB logs before or after it? (e.g. "I-RDB082002 Closing realtime connection.")
• Dropping a breakpoint here should pause when the problematic log is printed. It could be helpful to share what the call stack looks like at this point. It would also be interesting to know what the string is at that time (e.g. po string in the debugger while the execution is paused on the breakpoint).

I created a branch of the 11.3 release branch with an added one-time retry when the error in question is encountered: https://github.com/firebase/firebase-ios-sdk/tree/fix-13877

If you're able to give it a try, please let me know if it makes a difference or not.

[ncooke3]

Also, is a VPN being used?