typing_extensions

[jbrockmendel]

ATM we vendor typing_extensions but mypy doesnt recognize it, so we can't actually use it. We should either remove it or make it a dependency.

[TomAugspurger]

So if I do mypy pandas on a file using something from our vendored typing extensions, mypy will error? Or ignore the types?

[jorisvandenbossche]

Can you also explain what it means to "make it a dependency"? Is that only a developer dependency for when you want to do type checking? Or would it actually be a runtime dependency?

[jbrockmendel]

So if I do mypy pandas on a file using something from our vendored typing extensions, mypy will error? Or ignore the types?

If I import from pandas._vendored.typing_extensions import final and then incorrectly decorate Index.get_loc with @final, mypy will fail to alert me that this is incorrect.

[jbrockmendel]

Can you also explain what it means to "make it a dependency"? Is that only a developer dependency for when you want to do type checking? Or would it actually be a runtime dependency?

I'm not aware of any way to make it a dev-only dependency. AFAIK it would have to be runtime-importable.

[jorisvandenbossche]

And the import can't be hidden behind a if TYPE_CHECKING clause as we do for other cases, because it are actual functions used as decorators and such, I suppose?
We could have a no-op version of the items we want to use, which we import at runtime, and override them with the actual working versions from typing_extensions behind if TYPE_CHECKING ?

[jbrockmendel]

And the import can't be hidden behind a if TYPE_CHECKING

That's a neat idea, but doesn't do it.

[jorisvandenbossche]

And can you explain why it doesn't do it? (see also my second sentence in the comment above)

(I don't know much about typing, just asking questions)

[simonjayhawkins]

And can you explain why it doesn't do it? (see also my second sentence in the comment above)

i did this in https://github.com/pandas-dev/pandas/pull/27646/files/9aa69fe5491e0b161e679c8f54cbb8f13a48b85a

didn't get any support at the time follow conversation from #27646 (comment) onwards

[simonjayhawkins]

see also #34869 (comment) for more discussion and a reference to similar workaround used by pytest for Protocol.

[jorisvandenbossche]

So that seems certainly doable then to have mypy_extensions as a dev dependency, without having it as a runtime dependency

[simonjayhawkins]

and we no longer need to add it explicitly to environment.yaml see #27646 (comment)

[jbrockmendel]

And can you explain why it doesn't do it?

I don't know, can only confirm that I tried it and it didn't work.

see also #34869 (comment) for more discussion and a reference to similar workaround used by pytest for Protocol.

@simonjayhawkins this looks really similar to Joris's suggestion which I was not able to implement succesfully. My prior has nonzero weight on "me messing up", can you confirm whether or not this is viable for us?

[simonjayhawkins]

I'm not sure if we could do everything that we could do if having typing_extensions as a runtime dependency would allow. but if having a runtime dependency is a definite no-no, then some more investigation of possible workarounds is worthwhile.

as a start, have opened #37137 as a POC for Literal in function signatures.