[C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type

[vstinner]

BPO	39573
Nosy	@vstinner
PRs	bpo-39573: Use Py_REFCNT() macro #18388 bpo-39573: Add Py_SET_REFCNT() function #18389 bpo-39573: Use Py_TYPE() in abstract.c #18390 bpo-39573: Use Py_TYPE() macro in Python and Include directories #18391 bpo-39573: Use Py_TYPE() macro in Objects directory #18392 bpo-39573: Use Py_TYPE() macro in Modules directory #18393 bpo-39573: Add Py_SET_TYPE() function #18394 bpo-39573: Use Py_TYPE() macro in object.c #18398 bpo-39573: Add Py_SET_SIZE() function #18400 bpo-39573: Use Py_SET_SIZE() function #18402 bpo-39573: Use Py_TYPE() macro in ctypes.h #18411 bpo-39573: Use Py_SIZE() in s_set() of cfield.c #18419 bpo-39573: Add Py_IS_TYPE macro #18488 bpo-39573: Fix bad copy-paste in Py_SET_SIZE. #18496 bpo-39573: Update clinic to use Py_IS_TYPE macro #18507 bpo-39573: Update Include/* to use Py_IS_TYPE #18508 bpo-39573: Clean up modules and headers to use Py_IS_TYPE #18521 bpo-39573: Finish converting to new Py_IS_TYPE macro. #18601 bpo-39573: Use Py_IS_TYPE for type checking #18789 bpo-39573: Use Py_IS_TYPE for negative type checks #18798 bpo-39573: Make Py_IS_TYPE take const args. Add _PyObject_CAST_CONST. #18799 bpo-39873: PyObject_Init() uses PyObject_INIT() #18804 bpo-39573: Use Py_IS_TYPE to check for types #18809 bpo-39573: Use Py_IS_TYPE to check for types #19882 bpo-30459: Cast the result of PyList_SET_ITEM() to void #19975 bpo-39573: Convert Py_TYPE() to a static inline function #20290 bpo-39573: Fix buildbot failure for tupleobject.c #20391 bpo-39573: Convert Py_REFCNT and Py_SIZE to functions #20429 bpo-39573: Porting to Python 3.10: Py_SET_SIZE() macro #20610 [WIP, DO NOT MERGE] bpo-41188: Prepare CPython for opague PyObject structure. #21262 bpo-39573: Use the Py_TYPE() macro #21433 bpo-39573: Convert Py_TYPE() and Py_SIZE() back to macros #23366 bpo-39573: Remove What's new entry for Py_SIZE() #23375 bpo-39573: Py_TYPE becomes a static inline function #26493 bpo-11105: use a lower recursion limit for infinite recursion tests #26550 bpo-44348: Revert "bpo-39573: Py_TYPE becomes a static inline function (GH-26493)" #26596 bpo-39573: Py_TYPE becomes a static inline function #28128

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-09-08.16:32:15.871>
created_at = <Date 2020-02-06.23:07:12.131>
labels = ['expert-C-API', '3.11']
title = '[C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type'
updated_at = <Date 2022-01-20.00:24:00.180>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2022-01-20.00:24:00.180>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-09-08.16:32:15.871>
closer = 'vstinner'
components = ['C API']
creation = <Date 2020-02-06.23:07:12.131>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 39573
keywords = ['patch']
message_count = 96.0
messages = ['361513', '361514', '361515', '361516', '361517', '361518', '361519', '361522', '361523', '361526', '361527', '361529', '361531', '361540', '361549', '361555', '361557', '361590', '361593', '361607', '361611', '361626', '361631', '361639', '361904', '361960', '361961', '361963', '361964', '361965', '361971', '361977', '361987', '361988', '362033', '362034', '362133', '362134', '362166', '362212', '362216', '362445', '363345', '363494', '363564', '365690', '366473', '366493', '368047', '369896', '369898', '370074', '370303', '370638', '370663', '370665', '370666', '370671', '370729', '370902', '370932', '372308', '373460', '379675', '379679', '379680', '379757', '379759', '381337', '381345', '381365', '381374', '381403', '381404', '382260', '382534', '382539', '382780', '382781', '382783', '394954', '394971', '395018', '395205', '395206', '395287', '395323', '395536', '401365', '401370', '401378', '401395', '401396', '401399', '403252', '410995']
nosy_count = 1.0
nosy_names = ['vstinner']
pr_nums = ['18388', '18389', '18390', '18391', '18392', '18393', '18394', '18398', '18400', '18402', '18411', '18419', '18488', '18496', '18507', '18508', '18521', '18601', '18789', '18798', '18799', '18804', '18809', '19882', '19975', '20290', '20391', '20429', '20610', '21262', '21433', '23366', '23375', '26493', '26550', '26596', '28128']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue39573'
versions = ['Python 3.11']

Linked PRs

• gh-83754: Use the Py_TYPE() macro #120599

[vstinner]

Today, CPython is leaking too many implementation through its public C API. We cannot easily change the "default" C API, but we can enhance the "limited" C API (when Py_LIMITED_API macro is defined). Example of leaking implementation details: memory allocator, garbage collector, structure layouts, etc.

Making PyObject an opaque structure would allow in the long term of modify structures to implement more efficient types (ex: list specialized for small integers), and it can prepare CPython to experiment tagged pointers.

Longer rationale:

• https://pythoncapi.readthedocs.io/
• https://pythoncapi.readthedocs.io/bad_api.html
• https://pythoncapi.readthedocs.io/optimization_ideas.html

I propose to incremental evolve the existing limited C API towards opaque PyObject, by trying to reduce the risk of breakage.

We may test changes on PyQt which uses the limited C API.

Another idea would be to convert some C extensions of the standard library to the limited C API. It would ensure that the limited C API contains enough functions to be useful, but would also notify us directly if the API is broken.

[vstinner]

Another idea would be to convert some C extensions of the standard library to the limited C API. It would ensure that the limited C API contains enough functions to be useful, but would also notify us directly if the API is broken.

First issues that I met when I tried that:

• C code generated by Argument Clinic is incompatible the limited C API: METH_FASTCALL, _PyArg_CheckPositional(), static _PyArg_Parser, etc. are excluded from the limited C API.
• PyTypeObject is opaque and so it's not possible to implement a deallocator function (tp_dealloc) which calls tp_free like:
  Py_TYPE(self)->tp_free((PyObject*)self);
• _Py_IDENTIFIER() is not part of the limited C API

[vstinner]

New changeset a93c51e by Victor Stinner in branch 'master':
bpo-39573: Use Py_REFCNT() macro (GH-18388)
a93c51e

[vstinner]

it can prepare CPython to experiment tagged pointers

In September 2018, Neil Schemenauer did an experiment:

• https://mail.python.org/archives/list/capi-sig@python.org/thread/EGAY55ZWMF2WSEMP7VAZSFZCZ4VARU7L/
• https://github.com/nascheme/cpython/commits/tagged_int

More recent discussion on the capi-sig list:

https://mail.python.org/archives/list/capi-sig@python.org/thread/JPUNPN3AILGXOA3C2TTSLMOFNSWJE3QX/

See also my notes:
https://pythoncapi.readthedocs.io/optimization_ideas.html#tagged-pointers-doable

Wikipedia article: https://en.wikipedia.org/wiki/Tagged_pointer

[vstinner]

In the limited C API, Py_REFCNT() should be converted to:

static inline Py_ssize_t _Py_REFCNT(const PyObject *ob)
{ return ob->ob_refcnt; }
#define Py_REFCNT(ob) _Py_REFCNT(_PyObject_CAST(ob))

It would enforce the usage of newly added Py_SET_REFCNT() (PR 18389) and advertise that the object is not modified (const).

That would only be the first step towards a really opaque Py_REFCNT() function.

[vstinner]

TODO: Add Py_IS_TYPE() macro:

#define Py_IS_TYPE(ob, tp) (Py_TYPE(ob) == (tp)) 

For example, replace:

    #define PyBool_Check(x) (Py_TYPE(x) == &PyBool_Type) 

with:

    #define PyBool_Check(x) Py_IS_TYPE(x, &PyBool_Type)

IMHO it makes the code more readable.

nascheme@c156300

[vstinner]

New changeset c86a112 by Victor Stinner in branch 'master':
bpo-39573: Add Py_SET_REFCNT() function (GH-18389)
c86a112

[vstinner]

New changeset 0d76d2b by Victor Stinner in branch 'master':
bpo-39573: Use Py_TYPE() in abstract.c (GH-18390)
0d76d2b

[vstinner]

Py_TYPE() is commonly used to render the type name in an error message. Example:

PyErr_Format(PyExc_TypeError,
             "cannot convert '%.200s' object to bytearray",
             Py_TYPE(arg)->tp_name);

This code has multiple issues:

• It truncates type name to 200 characters: there is no Python exception, not even a marker to indicate that the string has been truncated
• It's only the short name: the qualified name (tp_qualname) would be more helpful. The best would be to generate the fully qualified name: module + qualname.
• Py_TYPE() returns a borrowed reference which is causing multiple issues: https://pythoncapi.readthedocs.io/bad_api.html#borrowed-references

In September 2018, I created bpo-34595: "PyUnicode_FromFormat(): add %T format for an object type name". But there was disagreement, so I rejected my change.

I started "bpo-34595: How to format a type name?" thread on python-dev:

• https://mail.python.org/archives/list/python-dev@python.org/thread/HKYUMTVHNBVB5LJNRMZ7TPUQKGKAERCJ/#3UAMHYG6UF4MPLXBZORHO4JVKUBRUZ53

I didn't continue this work (until now), since it wasn't my priority.

[vstinner]

New changeset a102ed7 by Victor Stinner in branch 'master':
bpo-39573: Use Py_TYPE() macro in Python and Include directories (GH-18391)
a102ed7

[vstinner]

Make PyObject an opaque structure is also a first step towards the more ambitious project "HPy" project which is fully opaque:
https://github.com/pyhandle/hpy

This API is written from scratch and currently implemented on top on the existing C API.

The following article is a nice introduction to the overall idea:
https://morepypy.blogspot.com/2019/12/hpy-kick-off-sprint-report.html

From my point of view, the long term goal would be to get better performance on PyPy and having a single API for C extension which would be efficient on all Python implementations (not only CPython).

Currently, the C API is not only a performance issue to run C extensions on PyPy. It's also an issue in CPython. Because the C API leaks too many implementation details, we cannot experiment optimizations.

See also: https://pythoncapi.readthedocs.io/rationale.html

[vstinner]

New changeset 58ac700 by Victor Stinner in branch 'master':
bpo-39573: Use Py_TYPE() macro in Objects directory (GH-18392)
58ac700

[vstinner]

New changeset daa9756 by Victor Stinner in branch 'master':
bpo-39573: Use Py_TYPE() macro in Modules directory (GH-18393)
daa9756