PEP 646: Decide on substitution behavior

[JelleZijlstra]

BPO	47006
Nosy	@gvanrossum, @serhiy-storchaka, @JelleZijlstra, @Fidget-Spinner, @mrahtz, @mrahtz, @AlexWaygood
PRs	gh-87390: Add tests demonstrating current type variable substitution behaviour #32341

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/JelleZijlstra'
closed_at = None
created_at = <Date 2022-03-13.20:46:16.314>
labels = ['type-bug', 'release-blocker', '3.11']
title = 'PEP 646: Decide on substitution behavior'
updated_at = <Date 2022-04-07.01:50:13.871>
user = 'https://github.com/JelleZijlstra'

bugs.python.org fields:

activity = <Date 2022-04-07.01:50:13.871>
actor = 'gvanrossum'
assignee = 'JelleZijlstra'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2022-03-13.20:46:16.314>
creator = 'JelleZijlstra'
dependencies = []
files = []
hgrepos = []
issue_num = 47006
keywords = ['patch']
message_count = 17.0
messages = ['415100', '415107', '415108', '415109', '415557', '415623', '415637', '415694', '415710', '415712', '415734', '415752', '415753', '416707', '416795', '416813', '416913']
nosy_count = 7.0
nosy_names = ['gvanrossum', 'serhiy.storchaka', 'JelleZijlstra', 'kj', 'matthew.rahtz', 'mrahtz', 'AlexWaygood']
pr_nums = ['32341']
priority = 'release blocker'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue47006'
versions = ['Python 3.11']

[JelleZijlstra]

We've had some disagreement about the behavior of TypeVarTuple substitution related to PEP-646, and the discussion has now spilled around multiple PRs. I'd like to use this issue to come to an agreement so we don't have to chase through so many different places.

Links:

• Clarify int division programming faq #75204 (merged) implements PEP-646 in typing.py. Matthew initially implemented full TypeVar substitution support, but took it out after I suggested an alternative solution in bpo-43224: Implement PEP 646 changes to typing.py #31021 (review).
• datetime.strptime: Support for parsing offsets with a colon #75981 (merged without review) implements TypeVarTuple substitution in Python.
• Support Py_tss_NEEDS_INIT outside of static initialisation #76009 implements TypeVarTuple substitution in C.
• HTMLParser: undocumented not implemented method #76025 and Error in 3.6.3 epub docs #76027 add additional test cases.
• multiprocessing calls flush on sys.stdout at exit even if it is None (pythonw) #75985 (closed) implements my proposed solution.

I'd like to ask that until we come to an agreement we hold off on making any more changes, so we don't have to go back and forth and we ensure that the eventual solution covers all edge cases.

The disagreement is about what to do with TypeVarTuple substitution: the behavior when a generic type is subscripted, like tuple[*Ts][int, str].

There are two possible extreme approaches:

• Implement full substitution support, just as we have it for existing TypeVars. This is complicated because TypeVarTuple makes it much harder to match up the types correctly. However, it is consistent with the behavior for other TypeVar-like objects. My example would turn into GenericAlias(tuple, (int, str)).
• Give up on substitution and just return a new GenericAlias object: GenericAlias(GenericAlias(tuple, Unpack[Ts]), (int, str). This avoids implementing any complex runtime behavior, but it inconsistent with existing behavior and less pretty when you print out the type. I prefer this approach because there's less risk that future enhancements to typing will break it. I also want to explore extending this approach to ParamSpec substitution.

[mrahtz]

Thanks for starting this, Jelle - I was a bit unsure about how to proceed here.

Given that #31800 is already merged, I'd also propose something halfway between the two extremes: return a sensible substitution when the logic to compute that isn't too onerous, and a new GenericAlias object when it is. The upsides are that we'd probably be able to return reasonable substitutions for the vast majority of cases, and that we wouldn't have to remove what's already been merged. The downsides would be lack of consistency, and the potential for changing rules about what does and doesn't return a full substitution as time goes on and new features are added.

[mrahtz]

(Having said that, to be clear: my preferred solution currently would still be the solution where we just return a new GenericAlias for anything involving a TypeVarTuple. The crux is what Serhiy is happy with.)

[JelleZijlstra]

Thanks Matthew! Merged PRs can still be reverted, and we have some time before the feature freeze. I'd like to hear what Guido and Ken think too.

If we go with the GenericAlias substitution, we need to make sure that such aliases still work as base class. That would need some C work to make types.GenericAlias.__mro_entries__ recurse if the alias's origin is itself a GenericAlias. There's a few other subtleties to think about; I can work on that but don't have a ton of time today.

[serhiy-storchaka]

I am for consistent behavior. If return GenericAlias(GenericAlias(tuple, Unpack[Ts]), (int, str)) for tuple[*Ts][int, str], we should also return GenericAlias(GenericAlias(list, T), int) for list[T][int], etc. And it will cause multiple problems:

• A repr can be less readable.
• It will break equality comparison and hashing. Good bye caching.
• What about __origin__, __parameters__, __args__? How will they be calculated?
• It can break code which uses annotations for something. For example it can break dataclasses.

It may be that will need to use it as a fallback for cases like tuple[T, *Ts][*Ts2] (currently it is error). But I am not sure that such cases should be supported.

[gvanrossum]

I think I'm with Serhiy, I don't understand the hesitance to transform tuple[*Ts][int, str] into tuple[int, str].

What would be an example of a substitution that's too complex to do?

[JelleZijlstra]

It's simple if you only look at simple examples.

Here are some examples current main (with Serhiy's patch for the Python version of typing) gets wrong:

>>> from typing import *
>>> Ts = TypeVarTuple("Ts")
>>> T1 = TypeVar("T1")
>>> T2 = TypeVar("T2")
>>> Tuple[T1, Unpack[Ts], T2][int, Unpack[tuple[int]]]  # expect error
typing.Tuple[int, *tuple[int]]
>>> Tuple[T1, Unpack[Ts], str, T2][int, Unpack[Ts]]  # expect error (T2 missing)
typing.Tuple[int, str, *Ts]  # it put *Ts in the wrong place
>>> Tuple[T1, Unpack[Ts], str, T2][int, Unpack[Ts], Unpack[Ts]]  # expect error (*Ts can't substitute T2)
typing.Tuple[int, *Ts, str, *Ts]
>>> class G(Generic[T1, Unpack[Ts], T2]): pass
... 
>>> G[int]  # expect error
__main__.G[int]

We can probably fix that, but I'm not looking forward to implementing the fixed logic in both Python and C. Also, I'm worried that it won't work with future extensions to the type system (e.g., the rumored Map operator) that may go into 3.12 or later versions.

[serhiy-storchaka]

The first case will be practically fixed by GH 32030 after chenging the grammar to allow unpacking in index tuple: A[*B].

Two other cases will be fixed by GH 32031. It does not require any C code.

In the last case no error is raised because some error checks are skipped if any of Generic arguments is a TypeVarTuple. We just need to add such checks. This is Python-only code too.

Note that the alternative proposition is even more lenient to errors.

[mrahtz]

[Guido]

What would be an example of a substitution that's too complex to do?

We also need to remember the dreaded arbitrary-length tuple. For example, I think it should be the case that:

T = TypeVar('T')
Ts = TypeVarTuple('Ts')
class C(Generic[*Ts]): pass
Alias = C[T, *Ts]
Alias2 = Alias[*tuple[int, ...]]
# Alias2 should be C[int, *tuple[int, ...]]

Ok, this is a bit of a silly example, but if we're committing to evaluating substitutions correctly, we should probably make even this kind of example behave correctly so that users who accidentally do something silly can debug what's gone wrong.

[Serhiy]

A repr can be less readable.

Definitely true.

It will break equality comparison and hashing. Good bye caching.

Huh, I didn't know about this one. Fair enough, this is totally a downside.

What about __origin__, __parameters__, __args__? How will they be calculated?

This could admittedly be thorny. We'd have to think it through carefully. Admittedly also a downside.

It can break code which uses annotations for something. For example it can break dataclasses.

Oh, also interesting - I didn't know about this one either. Could you give an example?

The first case will be practically fixed by GH 32030 after chenging the grammar to allow unpacking in index tuple: A[*B].

We actually deliberately chose not to unpack concrete tuple types - see the description of #30398, under the heading 'Starred tuple types'. (If you see another way around it, though, let me know.)

Two other cases will be fixed by GH 32031. It does not require any C code.

I'm also not sure about this one; disallowing unpacked TypeVarTuples in argument lists to generic aliases completely (if I've understood right?) seems like too restrictive a solution. I can imagine there might be completely legitimate cases where the ability to do this would be important. For example:

DType = TypeVar('DType')
Shape = TypeVarTuple('Shape')
class Tensor(Generic[DType, *Shape]): ...
Uint8Tensor = Tensor[uint8, *Shape]
Unit8BatchTensor = Uint8Tensor[Batch, *Shape]

Note that the alternative proposition is even more lenient to errors.

True, but at least it's predictably lenient to errors - I think the repr makes it very clear that "Woah, you're doing something advanced here. You're on your own!" I think it better fits the principle of least astonishment to have something that consistently lets through all errors of a certain class than something that sometimes catches errors and sometimes doesn't.

[mrahtz]

P.s. To be clear, (I think?) these are all substitutions that are computable. We *could* implement the logic to make all these evaluate correctly if we wanted to. It's just a matter of how much complexity we want to allow in typing.py (or in the runtime in general, if we say farmed some of this logic out to a separate module).

[gvanrossum]

I'd like to look at this as a case of simplifying something to its simplest canonical form, but no simpler. This is what the existing fixed-typevar expansion does: e.g. tuple[str, T, T][int] becomes tuple[str, int, int].

I propose that we try to agree on a set of rules for what can be simplified further and what cannot, when we have B = C[...]; A = B[...], (IOW A = C[...][...]), for various shapes of the subscripts to C and B. Note that what's relevant for the second subscript is C[...].__parameters__, so I'll call that "left" below.

1. Some edge case seems to be that if *tuple[...] is involved on either side we will never simplify. Or perhaps a better rule is that *tuple[...] is never simplified away (but fixed items before and after it may be).

2. Another edge case is that if neither side has any starred items we will always simplify (since this is the existing behavior in 3.10). This may raise an error if the number of subscripts on the right does not match the number of parameters on the left.

3. If there's a single *Ts on the left but not on the right, we should be able to simplify, which again may raise an error if there are not enough values on the right, but if there are more than enough, the excess will be consumed by *Ts (in fact that's the only way *Ts is fed).

4. If there's a *Ts on the right but not on the left, we should _not_ simplify, since whatever we have on the left serves as a constraint for *Ts. (E.g. tuple[int, int][*Ts] constrains *Ts to being (int, int).)

5. If there's exactly one *Ts on the left and one on the right, we _might__ be able to simplify if the prefix and suffix of the __parameters__ match the prefix and suffix of the subscript on the right. E.g. C[int, T, *Ts, float][str, *Ts] can be simplified to C[int, str, *Ts, float]. OTOH C[int, T, *Ts, float][*Ts] cannot be simplified -- but we cannot flag it as an error either. Note that __parameters__ in this example is (T, Ts); we have to assume that typevartuples in __parameters__ are always used as *Ts (since the PEP recognizes no valid unstarred uses of Ts).

TBH case 5 is the most complex and I may have overlooked something. I'm more sure of cases 1-4.