Adding an unstable C API for unique references

[ZeroIntensity]

Feature or enhancement

Proposal:

In #132070, there was a race brought up in free-threading through use of Py_REFCNT(op) == 1. The fix is to use _PyObject_IsUniquelyReferenced.

There were a couple additional comments about how we should handle this for the public API (for example, @godlygeek suggested implementing Py_REFCNT as _PyObject_IsUniquelyReferenced(op) ? 1 : INT_MAX). I think the best approach is to just expose an unstable API for this and point to it in Py_REFCNT's documentation.

I'm not imagining anything complex:

int
PyUnstable_Object_IsUniquelyReferenced(PyObject *op)
{
    return _PyObject_IsUniquelyReferenced(op);
}

@encukou, do you mind if this skips the C API WG? I'm not sure there's much to discuss about the API, other than some light bikeshedding.

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

Linked PRs

• gh-133140: Add PyUnstable_Object_IsUniquelyReferenced for free-threading #133144

[ngoldbaum]

I'm not sure if it's tracked elsewhere, but I'd also like to see an API function so NumPy can replace this code:

https://github.com/numpy/numpy/blob/d692fbccd98cb880812b32936e5f94fcfe55053f/numpy/_core/src/multiarray/temp_elide.c#L119-L152

We had to add it to fix NumPy on 3.14. See numpy/numpy#28681.

[Fidget-Spinner]

@ngoldbaum can you please open an issue for that? I'll mark it as a deferred blocker so it lands before 3.14.

Sam told me it's temporary code so it's fine for now. Just pointing out that the _PyStackRef API is really not something that can be relied on. It will very likely break often till we settle on something that is nice internally for us.

[colesbury]

Regarding the NumPy code, it can't use _PyObject_IsUniquelyReferenced (that won't fix the underlying problem). I agree with @Fidget-Spinner, we should open a new issue for it.

[ngoldbaum]

see #133164

[ZeroIntensity]

Actually, is there any use case for exposing _PyObject_IsUniquelyReferenced on its own if we're going to expose PyUnstable_Object_IsUniqueTemporary anyway? I'm not totally sure we need this one.

[colesbury]

Yeah, cases like enum_next_long where you want to re-use (or modify) an object that you have a unique reference to. PyUnstable_Object_IsUniqueTemporary isn't useful here because the object isn't a temporary on the interpreter stack. _PyObject_IsUniquelyReferenced is appropriate because we already know that en->en_result holds a strong (heap) reference to the relevant object:

cpython/Objects/enumobject.c

Lines 197 to 236 in c46635a

	static PyObject *
	enum_next_long(enumobject *en, PyObject* next_item)
	{
	PyObject *result = en->en_result;
	PyObject *next_index;
	PyObject *old_index;
	PyObject *old_item;
	Py_BEGIN_CRITICAL_SECTION(en);
	next_index = increment_longindex_lock_held(en);
	Py_END_CRITICAL_SECTION();
	if (next_index == NULL) {
	Py_DECREF(next_item);
	return NULL;
	}
	if (_PyObject_IsUniquelyReferenced(result)) {
	Py_INCREF(result);
	old_index = PyTuple_GET_ITEM(result, 0);
	old_item = PyTuple_GET_ITEM(result, 1);
	PyTuple_SET_ITEM(result, 0, next_index);
	PyTuple_SET_ITEM(result, 1, next_item);
	Py_DECREF(old_index);
	Py_DECREF(old_item);
	// bpo-42536: The GC may have untracked this result tuple. Since we're
	// recycling it, make sure it's tracked again:
	_PyTuple_Recycle(result);
	return result;
	}
	result = PyTuple_New(2);
	if (result == NULL) {
	Py_DECREF(next_index);
	Py_DECREF(next_item);
	return NULL;
	}
	PyTuple_SET_ITEM(result, 0, next_index);
	PyTuple_SET_ITEM(result, 1, next_item);
	return result;
	}

[encukou]

Does this mean that the Py_REFCNT docs are again wrong in a backwards-incompatible way?

do not rely on the returned value to be accurate, other than a value of 0 or 1.

---

@ZeroIntensity

do you mind if this skips the C API WG?

IMO, that's generally OK for unstable API. Don't forget docs & tests though.

But if this was sent to the WG I'd ask to avoid documenting it in terms of reference counting, which is an implementation detail you shouldn't rely on (as recent developments like this and immortality show).
Would it be correct to say that if this returns true, “immutable” objects can be safely mutated -- that is, to "other" code it will appear as if the object was destroyed and a new one with the same id was recreated? (And if so, what's "other" code and how long does this apply?)

[ZeroIntensity]

Does this mean that the Py_REFCNT docs are again wrong in a backwards-incompatible way?

Yes, but unless we want to revert PEP 703, I don't think there's any good way around that.

Would it be correct to say that if this returns true, “immutable” objects can be safely mutated -- that is, to "other" code it will appear as if the object was destroyed and a new one with the same id was recreated?

I think in-place updates would generally be checked by PyUnstable_Object_IsUniqueTemporary (gh-133170), not PyUnstable_Object_IsUniquelyReferenced. If I understand it correctly, the cases are:

• If you keep the strong reference in C code (e.g., owning some sub-data structure in your object, such as a dict to dict keys), then you want to check IsUniquelyReferenced. You just want to make sure there's no other threads messing with your object at the time. (Interpreter borrows don't matter here, because if the interpreter has a reference, then the refcnt will above > 1 anyway.)
• If you give the strong reference away to the interpreter (typically by returning it), then you need IsUniqueTemporary to check if the only reference is on the interpreter's stack in addition to being the only reference.

(@colesbury Fact check, please!)

But anyways, documenting this one in terms of reference counting would just make the API unclear. It acts as solely a replacement for Py_REFCNT(op) == 1, and since Py_REFCNT is already about reference counting, I think it's fine to document this one as such. The more important thing to me seems like documenting when to use both APIs.

[colesbury]

If you keep the strong reference in C code ... then you want to check IsUniquelyReferenced

Yeah, that's right. I think a enum_next_long above is a good example. The iterator holds a strong reference via en->en_result, so it's sufficient to call PyUnstable_Object_IsUniquelyReferenced. From context, you know that nobody else can be holding a borrowed reference to that object. In psuedo code:

static PyObject *
iter_next(PyObject *self)
{
   MyIterObject *iter = (MyIterObject *)self;
   if (PyUnstable_Object_IsUniquelyReferenced(iter->it_result) {
     // optimization: reuse iter->it_result
     // update iter->it_result
     ...
     return Py_NewRef(iter->it_result);
   }
   else {
     // create a new result value
     return ...
   }
}

You need PyUnstable_Object_IsUniqueReferencedTemporary (recently renamed from PyUnstable_Object_IsUniqueTemporary) if you don't hold the strong reference. This is a stricter check than PyUnstable_Object_IsUniquelyReferenced. The typical use cases involve arguments to C functions, because the callee doesn't own the arguments. In pseudo-code:

static PyObject *
my_nb_add(PyObject *a, PyObject *b)
{
   if (PyUnstable_Object_IsUniqueReferencedTemporary(a)) {
      // optimization: modify a in-place and return it
   }
   else {
     // add `a` and `b` and return the result as a new object
   }
}

PyUnstable_Object_IsUniqueReferencedTemporary(op) implies PyUnstable_Object_IsUniquelyReferenced(op), but the reverse is not true.