What about: volatile accesses and memory-mapped IO

[cramertj]

Folks who want to write drivers and embedded code using Rust need to have a way to guarantee exactly-once access to certain memory locations. Today, the embedded wg makes extensive use of @japaric's VolatileCell crate, along with RegisterBlock structures containing VolatileCell wrappers around each field of the register block, and a function to provide a single access to the register block at a fixed address. The API exposed in the the stdm32f103xx crate and similar only expose *const RegisterBlock values (example) from the overall Peripherals object. This then requires unsafe code to access and mutate any particular field.

Asks:

• Is this pattern sufficient to guarantee that the number of writes to IO-mapped memory will exactly match the number of calls to unsafe { (*x.volatile_cell_field).set(...) }, and that the number of reads will exactly match the number of calls to unsafe { (*x.volatile_cell_field).get(...) }? it seems like it should be.
• Is it possible to provide the same guarantee while exposing the register block via a safe reference type such as &? It would be possible to provide a custom RegisterRef<'a, T> that consisted of a raw pointer internally as well as a custom derive for projecting this to fields of the register block, but this seems unfortunately complicated and unergonomic.

Complicating factors:

• LLVM's precise definition of "volatile" is a bit shakey. It says that optimizers must not change the number of volatile operations or change their order of execution relative to other volatile operations. However, it doesn't seem to specify that non-volatile operations can't be inserted-- this is something we need to prevent, but which LLVM might insert in an attempt to pre-load a value (as allowed by the "dereferencable" attribute that we apply to references). Can we make sure that LLVM doesn't do such a thing? If we fail in that, could we potentially make the compiler understand that VolatileCell is special, similar to UnsafeCell, and cannot have "dereferenceable" applied to references to it (and objects that contain it), in order to prevent this misoptimization? This seems potentially more complicated and intrusive, but IMO still worth considering.

cc @RalfJung @kulakowski @teisenbe @rkruppe

[strega-nil]

If an lvalue is accessed through only volatile loads, LLVM will not add accesses that aren't volatile.

[cramertj]

@ubsan Awesome! Is that documented somewhere?

[hanna-kruppe]

I've not seen this guarantee stated anywhere in LLVM docs. It seems reasonable on its own but the dereferencable attribute may very well throw a wrench in it. Without that attribute, LLVM may not assume it can insert loads from a pointer, so it won't insert any unless it sees a pre-existing load, which can be extended (with a bit of care) to take the volatile-ness of the existing loads into account and not take pre-existing volatile loads as permission to insert new loads. On the other hand, by definition dereferencable means it's OK to insert any loads from the address anywhere.

While one may be tempted to say "OK but don't do that if there are volatile accesses", that's not really possible. The transformation that wants to insert a load may not be able to see the volatile accesses (e.g., because they are in another function), or there may not even be any accesses at all to the location (e.g., if the source program creates a &VolatileCell but doesn't use it).

This is not an issue for Clang compiling C code handling pointers-to-volatile because C pointers are not annotated dereferencable. When compiling C++ code that handles references-to-volatile, Clang does insert dereferencable as it does for all other references (https://godbolt.org/z/z6X8Y6) but dereferencable-on-references is also unsound in other ways (see e.g. https://lists.llvm.org/pipermail/llvm-dev/2018-July/124555.html) so that doesn't count for much.

[strega-nil]

@cramertj shoot! I may absolutely be wrong. I am pretty sure I saw that before, but I may be imagining it; or I may have seen it in a talk. I'll see if I can find it -.-

[cramertj]

Asked on llvm-dev

[briansmith]

For reference, this is similar to the old, long, thread https://internals.rust-lang.org/t/volatile-and-sensitive-memory/3188/46.

My conclusion from the previous thread is that Rust should add a new type modifier (maybe two, one for reads and one for writes) similar to volatile that guarantees the exactly-once behavior that isn't conflated with unsafe. That is, the language should ensure one shouldn't need to use unsafe at all, even in the implementation of VolatileCell, to get the exactly-once semantics.

[cramertj]

@briansmith Thanks for the link! Yeah, that is similar. The bit I'm especially interested in is @nikomatsakis's conclusion here:

The key point is that the compiler will not randomly introduce reads of an &T – it will only introduce reads that it can prove may happen at some point in the future. This is (I believe) true even with the derefenceable attribute.

i've had several experts tell me this isn't true, and that speculative reads may be introduced even when no normal reads could possibly occur. In fact, there are several passes that almost definitely do today -- ASAN and structure splitting, for example. However, this would mean that the guarantee that you and I are asking for doesn't exist even in C/C++ today, and that all non-ASM MMIO code is busted in this way. If that's true, it might be necessary to introduce something new into LLVM to mark pointers with an "exactly once" property.

[briansmith]

However, this would mean that the guarantee that you and I are asking for doesn't exist even in C/C++ today

C11 defines "access" as "execution-time action〉 to read or modify the value of an object" and says:

An object that has volatile-qualified type may be modified in ways unknown to the
implementation or have other unknown side effects. Therefore any expression referring
to such an object shall be evaluated strictly according to the rules of the abstract machine,
as described in 5.1.2.3. Furthermore, at every sequence point the value last stored in the
object shall agree with that prescribed by the abstract machine, except as modified by the
unknown factors mentioned previously. What constitutes an access to an object that
has volatile-qualified type is implementation-defined.

A volatile declaration may be used to describe an object corresponding to a memory-mapped
input/output port or an object accessed by an asynchronously interrupting function. Actions on
objects so declared shall not be ‘‘optimized out’’ by an implementation or reordered except as
permitted by the rules for evaluating expressions.

So, I think that C does provide quite a strong guarantee as long as you only refer to this memory using volatile-qualified types.

My understanding is that LLVM has a pass that erases the volitile modifier from objects and attempts to replace all accesses to them with "volatile accesses". After that pass LLVM can optimize all non-volatile accesses as it pleases but it must never introduce any new non-volatile accesses, in order for this implementation strategy for C volatile semantics to be correct. In other words, in later passes of the compiler, unless/until an object has been accessed using a non-volatile access, the pass must assume the object is volatile. Obviously, this restriction only applies to passes that are invoked by clang/clang++ when compiling C/C++ code though; passes that aren't used by the C/C++ compilers may not preserve those semantics.

As far as Rust is concerned, there's no volatile qualifier so currently the only workable semantics is to assume that every object is volatile and every reference/pointer is volatile as defined by C11 unless/until a non-volatile access/assignment is made; i.e. act like those latter passes of LLVM are supposed to act. (Whether they do or do not behave that way, I don't know.) This seems way too pessimistic to me; it seems workable now because the only Rust compiler publicly available is based on LLVM that already does that, but I'd hate to restrict future Rust compilers to LLVM's implementation strategy. Instead, I'd prefer to introduce volatile or similar ASAP into Rust's type system.

[gnzlbg]

This is not an issue for Clang compiling C code handling pointers-to-volatile because C pointers are not annotated dereferencable

Are Rust pointers annotated dereferencable? I don't think that one can use Rust references to interact with MMIO [0], but using pointers with volatile read / writes should be ok if these Rust pointers are not marked dereferencable.

[0] MMIO is always mutably aliased, so if you create a & or a &mut to it, you are essentially violating the invariants of Rust references.

[cramertj]

@gnzlbg

[0] MMIO is always mutably aliased, so if you create a & or a &mut to it, you are essentially violating the invariants of Rust references.

The VolatileCell wrapper uses UnsafeCell. Rust shouldn't ever assume that mutation isn't occurring behind a reference to an UnsafeCell. Rust references are annotated dereferenceable, but pointers are not.

[cramertj]

@briansmith

A volatile declaration may be used to describe an object corresponding to a memory-mapped
input/output port or an object accessed by an asynchronously interrupting function. Actions on
objects so declared shall not be ‘‘optimized out’’ by an implementation or reordered except as
permitted by the rules for evaluating expressions.

I don't see where in here it makes any guarantees about not inserting extra reads or writes. It says they won't be optimized out, but not that new accesses won't be inserted.

[briansmith]

I don't see where in here it makes any guarantees about not inserting extra reads or writes. It says they won't be optimized out, but not that new accesses won't be inserted.

I think it's implied by "Therefore any expression referring to such an object shall be evaluated strictly according to the rules of the abstract machine, as described in 5.1.2.3." In particular, IIRC (I'm not going to re-read it now), section 5.1.2.3 at least implies that each expression is evaluated exactly once.

[cramertj]

@briansmith Perhaps? That wasn't my reading of it, but if you can find more support for that, I'd be glad to hear that was the case. That doesn't appear to be the behavior implemented by LLVM-- @hfinkel on the llvm-dev list said that "I don't believe that the LangRef provides that guarantee" and pointed out that llvm::FindAvailablePtrLoadStore and llvm::isSafeToLoadUnconditionally don't appear to check for volatility.

[hanna-kruppe]

They also said in the same email that LLVM probably should provide such a guarantee (citing that volatile accesses usually lower to the same instruction as normal ones, so adding non-volatile accesses is in many ways like adding another volatile access). FWIW, I agree. Let's try to get LLVM to guarantee that.

Dereferenceable is still incompatible with MMIO, though, for reasons I outlined earlier and as also confirmed on llvm-dev. That indicates we either need to find alternatives to &VolatileCell<T> (e.g. VolatilePtr<T> wrapping a raw pointer) and try to mitigate the ergonomics problems that has, or special case VolatileCell or a similar lower level primitive (which would be to VolatileCell as UnsafeCell is to Cell).