Deterministic (but undefined) layout

[nikomatsakis]

From #31: Can we say that layout is some deterministic function of a certain, fixed set of inputs? This would allow you to be sure that if you do not alter those inputs, your struct layout would not change, even if it meant that you can't predict precisely what it will be. For example, we might say that struct layout is a function of the struct's generic types and its substitutions, full stop -- this would imply that any two structs with the same definition are laid out the same. This might interfere with our ability to do profile-guided layout or to analyze how a struct is used and optimize based on that. (Some would call that a feature.)

Also, this presumably applies to enums as well as other types.

[nikomatsakis]

Key comments from #11:

@asajeffrey writes:

Can we say something like we guarantee that the layout for a struct is only dependent on certain properties of its fields? Off the top of my head those are size, alignment and non-zero-ness, but there may be others. This would avoid having to propose an RFC with a language extension.

@RalfJung writes:

So, after all this talk about layout guarantees for structs... could we do a little practical exercise and see if rust-lang/rust#54922 if within bounds of the guarantees, or exceeding them? This does some manual layout computation that is supposed to recompute the layout of a repr(Rust) struct with an unsized tail.

@gankro writes:

Quickly chiming in with a +1 for restricting repr(rust) to only reordering and still mandating natural/greedy c-style padding. This would make (align=1,size=0) types not affect layout, newtypes not affect layout, alignment predictable, and the size of heterogeneous types predictable. At the same time it would still allow us to do the only really relevant optimization of ordering the fields optimally. The fact that a theoretical field fuzzer wouldn't be allowed to be as powerful as it could be is not especially concerning to me.

@gnzlbg writes (in response to @asajeffrey):

Can we say something like we guarantee that the layout for a struct is only dependent on certain properties of its fields? Off the top of my head those are size, alignment and non-zero-ness,

I don't know. For example, consider the repr(Rust) struct F { x: f32, y: f32, z: f32, w: f32 }, under the assumption that f32 has an alignment of 4 in the platform being targeted - the maximum alignment of a field of F is then 4. Would it be ok for the compiler to increase the alignment of F to say, 16, to facilitate SIMD operations?

We have been talking about guaranteeing the layout of homogenous tuples / structs, but I also imagine that this could happen somewhere else, e.g., struct G { a: u64, x: f32, y: f32, z: f32, w: f32, b: u64 } where {xyzw} get re-ordered at the front and the alignment increased to 16 to facilitate SIMD operations.

[nikomatsakis]

Sorry if I missed any.

[the8472]

• deterministic layout would disallow intentional randomization for fuzzing or defense purposes
• as written it would have to remain stable between compiler versions, e.g. to allow serialization or punning between libs compiled with different versions, which would prevent future changes. In essence the current implementation would become the implicit but ostensibly undefined standard
• this could have spooky action at a distance where upgrading a dependency which is a member of a struct suddenly causes the struct itself to change layout

So I would like to pose the counter-question: Why should this be guaranteed?

[ChrisJefferson]

I am +1 with @the8472 . My experience of C is that these kinds of limitations get annoying as time goes by, because newer ideas and compilers are limited by the old restrictions.

I'd actually be in favour of the compiler gaining randomisation, because (in my experience) even if something is technically allowed to vary, if it doesn't tend to change then people start to expect it being fixed.

Also randomisation seems (to me) a possibly cheap and effective anti-hacking technique -- if I can compile a program with a randomised order for every struct, it will be much harder for someone to make a repeatable hack, even if they find a suitable bug, as they won't know the layout of anything. Of course it isn't perfect by any means, but it's another layer of defense and I would want to see a good reason to throw it away.

If someone wants a well-defined fixed ABI, they can always use repr(C).

[rodrimati1992]

Deterministic layout seems appealing,especially if it is determined by the fields,not the generic parameters,

Even if layout is not deterministic,there should be ways to recover back some determinism through attributes.

One example attribute would be '#[phantomtype(C)]' which would guarantee that the generic parameters it lists cannot affect the layout of the type,only allowing PhantomData/types with the same attribute to mention those generic parameters.

#[phantomtype(C)]
struct WithMetadata<T,C>{
    value0:T,
    value1:Vec<u8>,
    _metadata:PhantomWrapper<C>,
}

#[phantomtype(C)]
struct PhantomWrapper<C>(PhantomData<fn()->C>)

The phantomtype attribute would allow these transmutes to be safe (always keeping the same T) :

• WithMetadata<u32,()>> to WithMetadata<u32,Initialized>
• WithMetadata<Rectangle<u64>,Uninitialized>> to WithMetadata<Rectangle<u64>,Initialized>
• Arc<WithMetadata<String,NoneMetadata>> to Arc<WithMetadata<String,SomeMetadata>>
• Box<WithMetadata<Arc<LargeStruct>,U10>> to Box<WithMetadata<Arc<LargeStruct>,U20>>

The phantomtype attribute would not guarantee these having the same memory layout:

• WithMetadata<u32,()>> compared to WithMetadata<i32,()
• WithMetadata<(),()>> compared to WithMetadata<PhantomData<fn()->Vec<u32>>,()>
• WithMetadata<Vec<usize>,()>> compared to WithMetadata<Vec<isize>,()>

[the8472]

That's quite a special case that does not generalize well to the myriad of heterogeneous structs with multiple non-ZST fields. It would fall under #34 already.

[rodrimati1992]

One example where it doesn't apply?
This is the very specific problem of wanting to guarantee stable layout for types with phantom generic parameters(when the phantom type parameter changes).

[the8472]

Well, it would still be determined by some of its generic parameters, T in that case, obviously each concrete T can result in different layouts even though it's always the same generic fields being present.

[Gankra]

I stand by my exact request; I'm fine with field re-ordering being the exact thing allowed.

[the8472]

@gankro couldn't that prevent some optimizations, e.g. reducing alignment (and thus size) when there are no references are taken on a private member?