Remove numpy references from pandas._typing

[WillAyd]

Because we don't have type information from numpy AND because we specifically ignore_missing_imports for that in our setup, any numpy types currently resolve to Any. This may cause surprising behavior if not scrutinized...

To illustrate, we currently define DType in pandas._typing as Dtype = Union[str, np.dtype, "ExtensionDtype"]. The numpy item here basically makes this a Union[..., Any] so it will almost never fail static analysis. If you remove the np reference therein you will get errors like the following:

pandas/tests/arrays/sparse/test_dtype.py:74: error: Argument 1 to "SparseDtype" has incompatible type "Type[int]"; expected "Union[str, ExtensionDtype]"

In this particular case we probably want to replace np.dtype with Type[Union[float, bool, int, str, object]] to account for legitimate dtype=object calls (@angelaambroz this was mentioned in #29046).

The appearances in TypeVar items may not be as big of a deal, so open to thoughts on whomever wants to review this

@simonjayhawkins

[jbrockmendel]

xref https://github.com/numpy/numpy-stubs

[simonjayhawkins]

xref #27589

#28831 (comment) suggests that we currently could have around 62 errors to resolve if numpy stubs are added.

[WillAyd]

There is an open issue for numpy stubs already #27589 if we want to discuss adding that there, but I still think for now easy to remove these from pandas._typing and wait until they are actually useful to put in

[simonjayhawkins]

personally, I just avoid adding types that I know will resolve to Any, so tend not to use the types in _typing, but see no reason to remove them.

o illustrate, we currently define DType in pandas._typing as Dtype = Union[str, np.dtype, "ExtensionDtype"]. The numpy item here basically makes this a Union[..., Any] so it will almost never fail static analysis.

I don't think this is entirely true. mypy can still check against the known types. so in this case only string methods and methods in ExtensionDtype can be called, otherwise mypy will report an error.

from typing import Any, Union


def foo(a: Union[str, Any]):
    a.bar

$ mypy test.py
test.py:5: error: Item "str" of "Union[str, Any]" has no attribute "bar"
Found 1 error in 1 file (checked 1 source file)

[WillAyd]

Hmm that's fair. In any case can still lead to very surprising behavior though like this:

https://github.com/pandas-dev/pandas/pull/29046/files#r335777289

which is what brought this to mind

[simonjayhawkins]

@WillAyd With numpy types imminent, I think we can now close this issue. feel free to reopen if disagree.

[WillAyd]

Makes sense - good things coming

…

Sent from my iPhone

On Jan 12, 2021, at 9:53 AM, Simon Hawkins ***@***.***> wrote:  Closed #29064. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.