Redo HTLC Handling Pipeline to do more checks async

[TheBlueMatt]

We only ever half-completed our transition to a batched HTLC processing pipeline - we currently forward in batches, but fail-backwards in-line. This is particularly nasty as it also means we do a ton of checks during update_add_htlc processing to decide if we're gonna fail an HTLC or pass it along. If we instead don't even decode the onion until we hit process_pending_htlc_forwards we can remove a ton of code in channel around PendingHTLCStatus matching and handling, not to mention making our timing-attack resistance substantially more robust in one go.

This will be a backwards compatibility nightmare, so we'll have to do it in stages. Basically add a new PendingHTLCStatus::HandleOnion which just stores the onion itself, implement the handling side of that, and let it bake in a release or two, then go start storing HTLCs with it, then remove parsing for the old states after a few more releases.

See eg #1199 (comment) for timing oracles that are hard to solve without this, plus the TODO at the top of fail_htlc_backwards_internal talking about how our back-fails are timing-vulnerable.

[valentinewallace]

This issue is partially addressed by #2845 and #2933.

As a follow-up to the latter PR, there is one further timing oracle remaining:

As of #2933, if we successfully decode an HTLC onion in process_pending_update_add_htlcs but then fail later in process_pending_htlc_forwards, we'll queue the failure in ChannelManager::forward_htlcs and generate a new PendingHTLCsForwardable event. However, if we instead fail to decode the onion in process_pending_update_add_htlcs, we'll send an update_fail_htlc directly with no intermediate Forwardable event.

These two paths should be unified, meaning if we fail during onion decoding we should still push the failure to self.forward_htlcs and push a Forwardable event to match the other failure path, rather than failing immediately.