Skip to content

(0-1) as *const T triggers horribly stupid behavior #43291

Closed
@retep998

Description

@retep998

During this issue all examples will be done using x86_64.

When you do 0 as *const T the type of the integer literal is inferred to be usize. This is fine.
When you do !0 as *const T you can observe that the resulting address is 0xffffffffffffffff which is fine.

When doing FFI sometimes a handle type will be represented as a raw pointer type and sometimes there will be constants of that handle type initialized to negative values, which is entirely normal.
The user's first choice is to try -1 as *const T which will of course fail because the type of the integer literal is inferred to be usize which cannot be negated because Rust hates me.
Ideally Rust would be able to infer the type as isize instead of usize in this case, but I know better than to expect Rust to be actually helpful.

The issue is if the user tries to be clever and do (0-1) as *const T which somehow seems to work but in reality they are triggering a combination of two horrible design decisions (or maybe bugs, it can be hard to tell sometimes) resulting in a giant footgun.

  1. The type of those integer literals is being inferred as i32 instead of usize. Rust earlier refused to infer the type as isize instead of usize, so why is it now inferring the type as i32?
  2. The conversion from i32 to *const T does zero extension instead of sign extension which is different than what C++ does. Add lint for u8 as *mut cast #42915

As a result, if you check the resulting address of (0-1) as *const T you'll see that it gives 0xffffffff which is wrong.
The correct answer is achieved by doing -1isize as *const T which results in the correct address of 0xffffffffffffffff.

Ideally both of those two points above would be fixed, but at the very least there should be a very loud warning that you are doing something very horribly wrong.

And yes, I found examples of this in winapi so some care will be needed when fixing this in order to not break the ecosystem.

Possible solutions

  1. Add a lint to forbid casts between pointer types and integer types other than usize/isize.
  2. Change casts from signed integer types to pointer types to perform signed conversion.
  3. Fix the inference issue causing the literals to be inferred as i32 instead of usize.

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-bugCategory: This is a bug.T-langRelevant to the language team

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions