3.12.0b4 Backwards incompatible change with reassignment of `cls.__new__` and `super()`

[cdce8p]

Bug report

Noticed this while testing 3.12.0b4. While in my particular case I can work around it, it never the less is a change in behavior to 3.11.

class FixedIntType(int):
    _signed: bool

    def __new__(cls, *args, **kwargs):
        print(f"{cls}, {args=}")
        return super().__new__(cls, *args, **kwargs)

    def __init_subclass__(cls, signed: int | None = None) -> None:
        super().__init_subclass__()
        if signed is not None:
            cls._signed = signed

        if "__new__" not in cls.__dict__:
            cls.__new__ = cls.__new__  # <-- This line triggers the error

class uint_t(FixedIntType, signed=False):
    pass

class CustomInt(uint_t):
    def __new__(cls, value: int = 0):
        return super().__new__(cls, value)

    def with_id(self, value: int):
        return type(self)(value)

CustomInt().with_id(1024)

<class '__main__.CustomInt'>, args=(0,)
<class '__main__.CustomInt'>, args=(<class '__main__.CustomInt'>, 1024)
Traceback (most recent call last):
  File "/.../cpython/test.py", line 26, in <module>
    CustomInt().with_id(1024)
  File "/.../cpython/test.py", line 24, in with_id
    return type(self)(value)
           ^^^^^^^^^^^^^^^^^
  File "/.../cpython/test.py", line 21, in __new__
    return super().__new__(cls, value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.../cpython/test.py", line 6, in __new__
    return super().__new__(cls, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: int() base must be >= 2 and <= 36, or 0

Without cls.__new__ = cls.__new__

<class '__main__.CustomInt'>, args=(0,)
<class '__main__.CustomInt'>, args=(1024,)

I bisected the issue to #103497.
/CC: @carljm

Your environment

• CPython versions tested on: 3.120b4
• Operating system and architecture: macOS ARM64

Linked PRs

• gh-106917: fix super classmethod calls to non-classmethods #106977
• [3.12] gh-106917: fix super classmethod calls to non-classmethods (GH-106977). #107204

[carljm]

Thanks for this report!

__new__ is automagically treated as a staticmethod (but only when it's part of the type at construction, not if it's assigned afterward.) So the repro case ultimately simplifies and de-sugars to this:

class A:
    @staticmethod
    def some(cls, *args, **kwargs):
        print(f"{cls}, {args=}")
        assert not args
        assert not kwargs

class B(A):
    def some(cls, *args, **kwargs):
        return super().some(cls, *args, **kwargs)

class C(B):
    @staticmethod
    def some(cls):
        return super().some(cls)

C.some(C)
C.some(C)

Note that B.some is not a @staticmethod, while A.some and C.some both are. This is equivalent to what happens in the OP. The assignment cls.__new__ = cls.__new__ looks up the staticmethod in the right hand side (from the base class via MRO lookup, since it doesn't exist yet on cls) and "unwraps" the staticmethod in descriptor resolution, returning a plain function object, and then assigns that plain function object (not a staticmethod) as the __new__ method on the new subclass. (And dynamic assignment of __new__ does not magically wrap it in staticmethod.) So the middle class in the hierarchy (uint_t in the OP) ends up with a non-staticmethod __new__, while the top and bottom classes have a normal staticmethod __new__.

This simplified and de-sugared repro works in 3.11, but fails in a similar way to the OP in main and 3.12:

➜ ../dbg/python superbug3.py
<class '__main__.C'>, args=()
<class '__main__.C'>, args=(<class '__main__.C'>,)
Traceback (most recent call last):
  File "/home/carljm/cpython-builds/repros/superbug3.py", line 18, in <module>
    C.some(C)
  File "/home/carljm/cpython-builds/repros/superbug3.py", line 15, in some
    return super().some(cls)
           ^^^^^^^^^^^^^^^^^
  File "/home/carljm/cpython-builds/repros/superbug3.py", line 10, in some
    return super().some(cls, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/carljm/cpython-builds/repros/superbug3.py", line 5, in some
    assert not args
AssertionError

The double call to C.some(C) in the repro is necessary so that we specialize LOAD_SUPER_ATTR to LOAD_SUPER_ATTR_METHOD, triggering the bug. The base LOAD_SUPER_ATTR goes through the same code path as an unoptimized LOAD_ATTR; CALL.

What we have in these examples is a "class-mode" super() call (e.g. super(MyClass, MyClass).foo(...), where the second super arg is also a class, not an instance). (Zero-arg super() implicitly uses the first argument of the method as the second arg to super(), and in this case that first arg of the method is a class, not an instance.)

In a class-mode super() call (just like for a normal lookup directly on a class), we do not pass an instance to the descriptor-getter, only a type (e.g. func_descr_get(descr, NULL, type) instead of func_descr_get(descr, obj, type). In the case of a regular function object, this causes the function descriptor to just return the function itself (no bound method), which is the same behavior the staticmethod descriptor always has! So as long as we are only doing class lookups, staticmethod has no effect, and therefore we can get away with this mixed staticmethod/non-staticmethod hierarchy just fine.

The bug here is that the LOAD_SUPER_ATTR_METHOD specialization tries to use the LOAD_METHOD optimization (avoid actually creating a bound method object via descriptor, and instead just detect that case and ensure the extra argument is provided), but fails to account for the fact that a class lookup of a regular function object (not a classmethod) is just a function call, not a method call.

Fix coming shortly.