Rust does not comply with IEEE 754 floats: arithmetic can produce signaling NaN

[Muon]

The compiler optimizes x * 1.0 to x, which is incorrect whenever x is a signaling NaN. IEEE 754 mandates that the result is a quiet NaN.

Example (rustc 1.66, with -C opt-level=2): https://godbolt.org/z/5qa5Go17M

This is almost certainly an LLVM bug. It's been filed over there under llvm/llvm-project#43070. I don't know which optimization pass is responsible for it.

[CAD97]

Interestingly enough, MSVC does the same transformation even with /fp:strict.

I've access to the actual text of IEEE 754-2019 through my current university (I suppose I could've grabbed 754-2008 instead, but I grabbed the latest revision), so here's the actual relevant quotes:

§6.2 ¶3: Signaling NaNs shall be reserved operands that signal the invalid operation exception (see §7.2) for every general-computational and signaling-computational operation except for the conversions described in §5.12.

§6.2 ¶2: Under default exception handling, any operation signaling an invalid operation exception and for which a floating-point result is to be delivered, except as stated otherwise, shall deliver a quiet NaN. For non-default treatment, see §8.

§7.2 ¶2: For operations producing results in floating-point format, the default result of an operation that signals the invalid operation exception shall be a quiet NaN that should provide some diagnostic information (see §6.2). These operations are:
a) any general-computational operation on a signaling NaN (see §6.2), except for some conversions (see §5.12)

§5.12 is about conversions between floating-point data and character sequences, so is not relevant. Multiplication is a "formatOf general-computational operation".

Thus, I agree that the standard mandates that signalingNaN * 1.0 should (raise an invalid operation exception and) evaluate to a quiet NaN, and that the transformation to just return signalingNaN is incorrect.

If we relax the language specification from "default floating point environment" to "implementation defined floating point environment" and allow the language to arbitrarily change the exception handling mode, then we can justify the transformation, by first setting the exception handling mode for invalid operation to substitute(signalingNaN), doing signalingNaN * 1.0 (signaling an invalid operation exception, handling that by substituting signalingNaN as the result), and then setting the exception handling mode back to whatever it was previously (probably mayRaiseFlag in this world).

FWIW, alive2 times out for float and considers the transformation valid for half. From LLVM LangRef:

The default LLVM floating-point environment assumes that floating-point instructions do not have side effects. Results assume the round-to-nearest rounding mode. No floating-point exception state is maintained in this environment. Therefore, there is no attempt to create or preserve invalid operation (SNaN) or division-by-zero exceptions [emphasis mine].

The benefit of this exception-free assumption is that floating-point operations may be speculated freely without any other fast-math relaxations to the floating-point model.

I tried using the (experimental?) constrained floating point intrinsics, but I don't think alive2 supports the strictfp attribute. As far as I can tell, we need !"fpexcept.strict" in order to avoid removing operations which may raise a floating point exception. But even with !"fpexcept.ignore", that's just defined as "may assume that the exception status flags will not be read and that floating-point exceptions will be masked", which I don't think is strong enough to assume signaling NaNs don't exist (invalid operation exceptions won't be raised), and the change from signaling to quiet should need to be preserved under those semantics.

Of note from the Rust docs, though, is that a bunch of f32 methods say roughly

This follows the IEEE 754-2008 semantics for operation, except for handling of signaling NaNs; this function handles all NaNs the same way [...]

I think there's a fair chance that the Rust opsem will make no distinction between signaling and quiet NaNs. I can somewhat justify a perverse reading of IEEE 754-1985 that allows all binary encoded NaNs to be quiet; 754-2008 and 754-2019 of course mandate a specific interpretation of the mantissa MSB as indicating quiet (set) or signaling (cleared), but Rust does support targets such as MIPS which do not agree with the 2008 definition.

[CAD97]

I would cc T-opsem and/or A-mir-formality but I don't think either have a pingable role.

For lack of a better classification,

@rustbot modify labels +T-lang

[CAD97]

cc also

• Our floating point semantics were a mess unsafe-code-guidelines#237
• Document guarantees (or lack thereof) regarding sign, quietness, and payload of NaNs #73328
• i686 floating point behavior does not agree with unit tests in debug mode #73288
• LLVM (thus, Rust) floating point support support is incomplete #10186

[workingjubilee]

As much as I am usually in favor of pedantic adherence of IEEE754, it is extremely dubious that we would want to guarantee that this transformation would not occur, as Rust does not meaningfully support floating point exceptions and thus does not support the utility of this exception to the identity.

[thomcc]

I think the argument would be that sNaN vs qNaN is still a meaningful difference even if exceptions don't exist.

[workingjubilee]

Plausible.

[CAD97]

Yes, that's exactly it. It's not observable just via f32::total_cmp like the sign bit, but it's actually prescribed by the spec, unlike the sign bit¹.

It's very likely that the strictest bound we'll get for Rust is that floating point operations returning NaN return a nondeterministic² quiet NaN. This is (almost³) compliant with the IEEE spec in the mayRaiseFlag⁴ exception handling mode, so long as you allow implementer's discretion to choose quiet NaN payloads as permitting nondeterminism⁵.

We're not concerned about the rest of the NaN payload here (though that's also its own unresolved question), just the signaling bit. Unlike the rest of the NaN payload an even sign, the IEEE standard does explicitly require that no operation produce a NaN with the signalling bit unquiet; all operations must produce quiet NaNs.

Wasm is brought up as a target with nondeterministic NaNs, but it actually does only ever produce quiet NaNs. The spec states that "There is no observable difference between quiet and signalling NaNs," but this is only in respect to the fact that exception flags are not exposed in wasm. Separately, wasm defines the concept of an arithmetic NaN, which exactly matches the IEEE definition of quiet NaNs. It then requires all arithmetic operations to produce arithmetic NaNs (and if all NaN inputs are canonical NaNs, for the output to be a canonical NaN⁶).

Speaking on behalf of myself only, I would place slim odds favoring LLVM and subsequently Rust deciding to treat all NaNs as being quiet, willfully ignoring the signaling bit and violating the spec in a minor way (producing sNaN from arithmetic operations) in the name of performance. This seems to me the simplest way to reconcile the current behavior⁷ with the IEEE spec.

LLVM's documentation on the constrained floating-point operations states that mixing the unconstrained and constrained operations is not allowed, and that when in strictfp mode you can get the normal behavior by setting the appropriate metadata arguments. This isn't quite accurate in practice: by my testing, the strict intrinsics don't (currently) perform this reduction from x * 1.0 to x. It would be an interesting experiment to apply strictfp annotation everywhere and use the default-modality constrained floating-point operation intrinsics instead of the builtins, to determine the current overhead of actually asking LLVM for the correct behavior. I have my suspicions that it'll be higher than necessary just for sNaN quieting, unfortunately, as strictfp function calls have no way to be weakened down from fully strict.

More recent discussion on the LLVM discourse forum: https://discourse.llvm.org/t/semantics-of-nan/66729

Footnotes

1. Which makes it somewhat odd imho that the floating point total order sorts quiet/signalling NaN next to each other but positive/negative to opposite extremes. You'd think the bit prescribed by the standard would be more important than the one left fully unspecified. ↩

2. As with freeze undef: demonic nondeterminism each time an operation producing NaN is evaluated, but using the result of a single operation is consistent with itself. This forbids duplicating floating point operations. ↩

3. Technically LLVM optimizations are also allowed to raise floating point exceptions spuriously, even when no floating point operations are executed, so that's nonconforming, as per the IEEE spec, status flags are raised without an exception being signaled only at the user's request. ↩

4. This is the mode which allows a floating point exception to nondeterministically raise or not raise exception flags. ↩

5. Brought up in LLVM discussion is that you could NaN-box the instruction pointer to implement the error introspection suggested by the spec, which is effectively nondeterministic at any higher level than true bare metal machine code. This has convinced me that allowing nondeterminism is a reasonable interpretation of the IEEE spec. The spec similarly does not specify the sign of a NaN result. ↩

6. This terminology is quite unfortunate. To the IEEE spec, all binary floats are canonical. To wasm, a canonical NaN is one with the first payload bit set (making the NaN quiet) and no other bits. In other places, a canonical NaN is required to have an unset sign bit, but wasm allows both positive and negative NaNs to be canonical. ↩

7. w.r.t. normalizing sNaN to qNaN, anyway. I have no proposal for the exception flags better than stating that entering Rust code immediately sets all fp exception flags to a demonically nondeterministic value. ↩

[Muon]

Floating-point operations returning a nondeterministically chosen quiet NaN is perfectly compliant. Also bear in mind that Rust itself does not implement the full scope (in particular, flags) of IEEE 754, it only claims that floating-point arithmetic operations follow it.

I do not imagine there being a significant performance hit to disabling that "optimization". The only thing multiplication by 1 does is quiet a signaling NaN. Same with x + -0.0, which also gets miscompiled to x. Luckily, it doesn't transform x + 0.0 to x as well; that one would be a much more egregious error, since -0 + +0 = +0.

Which makes it somewhat odd imho that the floating point total order sorts quiet/signalling NaN next to each other but positive/negative to opposite extremes. You'd think the bit prescribed by the standard would be more important than the one left fully unspecified.

That's so totalOrder can be implemented efficiently! The ordering you get from it is the same as the natural order from interpreting the float as a sign-magnitude integer (assuming you're using the recommended [but not mandated] choice of signaling bit).

[CAD97]

Floating-point operations returning a nondeterministically chosen quiet NaN is perfectly compliant.

Yes, but it's been expressed a couple times that it'd be nice to have deterministic results.

I do not imagine there being a significant performance hit to disabling that "optimization". The only thing multiplication by 1 does is quiet a signaling NaN.

The reason I expect it to be more expensive is not due to what we actually want to disable, but what will get prevented in the crossfire. strictfp in LLVM is a heavy and experimental hammer meant for supporting floating point environment access; that it also suppresses the contraction which fails to quiet the NaN even when ignoring floating point exceptions is more due to disabling basically all optimizations around floating point operations rather than actually respecting IEEE semantics better.

recommended [but not mandated] choice of signaling bit

Huh yeah, you're right; I somehow got it in my head that the latter standard revisions prescribed the interpretation of the signaling bit rather than just providing a recommendation for interchange.

This means the (questionable) interpretation where your binary format for floats contains no encodings for signaling NaN ("should"), but your operations do "support" signaling NaN ("shall") should they be given one (impossible) is still (arguably) somewhat remotely viable.

Providing such an implementation of IEEE floating point would I think be an accurate description of the current behavior (modulo FFI visibility of floating point exception flags).

[Muon]

Yes, but it's been expressed a couple times that it'd be nice to have deterministic results.

It's not possible to guarantee that a specific qNaN is produced everywhere, even just on the current tier 1 platforms. In particular, the default qNaN on x86 always has the sign bit set (0xFFC00000 for f32), but it's always cleared on aarch64 (0x7FC00000 for f32).

This means the (questionable) interpretation where your binary format for floats contains no encodings for signaling NaN ("should"), but your operations do "support" signaling NaN ("shall") should they be given one (impossible) is still (arguably) somewhat remotely viable.

Firstly, we shouldn't consider ignoring sNaNs, because they are in fact something that exists and are in fact supported by the targets. This miscompilation causes the following to behave differently between debug and release: https://godbolt.org/z/e7qvYEPhc EDIT: simpler version: https://godbolt.org/z/j9h44d979. Secondly, encoding support for them is mandated by IEEE 754-2019, section 3.4:

The representation r of the floating-point datum, and value v of the floating-point datum represented, are inferred from the constituent fields as follows:
a) If E = 2ʷ − 1 and T ≠ 0, then r is qNaN or sNaN and v is NaN regardless of S and then d₁ shall exclusively distinguish between qNaN and sNaN (see 6.2.1).

That is, the first bit of the significand must always determine whether it is an sNaN or a qNaN, but the standard doesn't mandate that a set bit means it is a qNaN.

[CAD97]

It's not possible to guarantee that a specific qNaN is produced everywhere

miscompilation causes the following to behave differently between debug and release

So is nondeterminism okay or not?

By the standard¹, qNaN * 1.0 may nondeterministically return any qNaN. Restating, bits(qNaN * 1.0) == bits(qNaN * 1.0) may be nondeterministically true or false. The same goes for any NaN-producing operation. The standard treats the sign and payload bits of NaN identically: no restrictions.

I'm not talking about whether results are portable between different targets; I'm talking about whether results are deterministic within a single execution.

And this is in fact a very important condition to discuss. NaN selection being either "deterministic across all targets" or "nondeterministic on all targets" are fine for optimizations, because all targets have the same behavior, and thus our target independent IR has a single behavior we can optimize over. If NaN selection must be deterministic within a single target but changes between targets, this changes the set of optimizations we're able to do.

Consider again the example of z * 1.0. We don't want to optimize this to just z, because this is incorrect when z is sNaN. But we can optimize it to quiet(z), where quiet is "if NaN, set $d_1$ to quiet"... but not if we have to respect the target's unknown NaN selection behavior. The target could only ever produce a single qNaN, or it could follow the NaN propagation rules suggested by the standard; we have no way to optimize floating-point operations that could produce NaN in a target independent manner. A transformation from z * 1.0 * 1.0 to z * 1.0 produces the same abstract value according to the standard, but also could change the concrete representation produced according to the standard.

This is why I say that the strongest semantic I think Rust will get is that every time a NaN is produced by a computational operation, that NaN shall be nondeterministically² selected from the set of all qNaN³. This gives us a single semantic to optimize over which is validly refined by every correct implementation of the IEEE standard.

supported by the targets

If you consider Rust as targeting LLVM, no, not really. LLVM makes no effort to preserve the signaling quality of NaN. Whether the disclaimer just covers fp exceptions or also covers the signaling bit is debatable, but in practice this miscompilation is exactly due to LLVM pretending sNaN doesn't exist.

The strictfp operations are still experimental, and llvm.canonicalize still causes an instselect error on most codegen backends. For most intents and purposes, LLVM doesn't support sNaN.

That is, the first bit of the significand must always determine whether it is an sNaN or a qNaN

I'm not saying that this reading of the standard is good. But so long as no bit other than $d_1$ influences whether $r$ is sNaN or qNaN, it's possible to argue that this clause is satisfied. If that's the case, I can say that both set and unset mean qNaN and be technically conforming by extremely questionable rules lawyering.

Footnotes

1. And very explicitly on wasm. ↩

2. This nondeterminism is probably demonic, meaning that if a selection that causes UB exists, that is the selection made. As a result, your program must be defined for all possible selections in order to be defined. ↩

3. Defining what the set of qNaN is is the other question. I maintain that all NaN is a possible choice, despite likely being in violation of the IEEE standard. It's essentially a given that Rust requires tonearest rounding mode and non-stop exception handling, and makes no guarantees about fp exception status flags. Under this environment, the only way to observe sNaN vs qNaN is by observing the signaling bit directly. ↩