Warn and eventually forbid transmute::<T, U> for T or U with unspecified (Rust) layout

[nagisa]

It is a fairly commonplace mistake to do a transmute::<T, U> for T and U which are not necessarily compatible, but happen to work at that some particular point in time. These transmutes either change in behaviour when the compiler is updated or stop compiling altogether (because the size of T and size of U are not the same anymore (see e.g. #50830)).

The same way we error for mismatching sizes, we can also raise such an error (possibly disable-able by a #[feature(very_fast_such_dangerous)]) for types for which sizes may change over time with future compiler releases. Namely, this would prevent using transmute on stable for anything that

1. Does not have one of the whitelisted #[repr()] attribute on top of the type declaration;
2. Is not a FFI-safe type in the first place.

If we did that already, the issue linked before would’ve failed, because none of the enums have a #[repr] attribute on top of them, making their layout unspecified and therefore not transmutable.

[nagisa]

cc @nikomatsakis

---

I’ll be picking at the implementation of this during the impl days.

[the8472]

Couldn't code assert the layout compatibility of all fields with offset_from to make such transmutes safe and have a fallback codepath in case future changes alter the layout?

[est31]

There is prior art for this lint in the improper_ctypes lint which only fires for ffi interfaces though. Can the two be unified or share infrastructure?

Also, what about pointer casts?

Last question, as monomorphization time errors are ugly: We could add a Cast<U> builtin trait that is implemented for T iff it is castable to U. I guess the ship has sailed on adding that bound to transmute?

[hanna-kruppe]

There is prior art for this lint in the improper_ctypes lint which only fires for ffi interfaces though. Can the two be unified or share infrastructure?

I don't think they have a lot of common ground, neither in concept (non-C types can still have defined layout) nor in implementation.

Also, what about pointer casts?

What about it? Do you want to lint for *const T as *const U for T, U with unspecified layout?

Last question, as monomorphization time errors are ugly: We could add a Cast builtin trait that is implemented for T iff it is castable to U. I guess the ship has sailed on adding that bound to transmute?

Precisely to avoid monomorphization time errors, transmute already doesn't work on type parameters (or more precisely, on types whose size depends on a type parameter).

[est31]

Thanks for the answers this clarified some stuff.

Do you want to lint for *const T as *const U for T, U with unspecified layout?

How is *const T as *const U different from a transmute::<T,U> invocation?

[hanna-kruppe]

How is *const T as *const U different from a transmute::<T,U> invocation?

For starters, the pointer cast itself doesn't do anything of note and is always well-defined (assuming T and U are Sized, at least). It can be very useful even without ever dereferencing the pointer (e.g., smuggling a Rust type through a C API that deals in void *).

It's true that you can build your own transmute-like horror out of pointer casts and other primitives, but pointer casts alone are much more tame and general and more frequently necessary -- in fact many uses of transmute are best replaced with simpler pointer casts!

[est31]

@rkruppe makes sense, you convinced me.

[nagisa]

Experiments have been done in #51294 and conclusions drawn.

---

Sorted by impact, these are the "unspecified" layout types that are transmuted the most from or to in the crates ecosystem:

• &[T], &str, &variant_type::VariantTy (from glib, a repr(Rust) newtype over str);
• Box<T> (for both T: Sized and T: ?Sized; is it specified to be same as a pointer to T?);
• *mut dyn T, *const dyn T, &mut dyn T, &dyn T;
• std::sync::Arc<T> (mostly by futures?);
• std::net::SocketAddrV4, std::net::SocketAddrV6 (crate socket2);
• Result;
• utf8_char::Utf8Char (crate encode_unicode);
• *mut Self (crate unsafe_any);
• repr(Rust) structs, tuples;
• Identity transmutes that only transmute the lifetime;
• Transmutation to change the mutability of reference from immutable to mutable (even if it only occurs in one crate… what?);

---

In the compiler/libstd the following transmutes are common:

• Identity transmutes that only "transmute" the lifetime;
• Transmutes to/from raw::TraitObject;
• Transmutes from Box<Trait + Trait2> to Box<AnotherSetOfTraits>;

---

Many of these are genuine mistakes users make when transmuting, some others are "technically" safe from the standpoint of the layout (e.g. identity transmutes or lifetime transmutes) but questionable in other ways.

Note, that implementation of such check broke some 7k of the crates transitively. If it was a proper lint, the impact would be lesser, but still very widespread.

[gnzlbg]

Note, that implementation of such check broke some 7k of the crates transitively. If it was a proper lint, the impact would be lesser, but still very widespread.

Is there a single dependency breaking all those crates, or are those 7k crates using transmute incorrectly ?

Also, what's the plan here? I'd rather have this as a warning that can be turned off, than have nothing. Ideally, maybe one day some of those crates will be fixed, and we can enable this as a hard error.