Missing Imports

[simonjayhawkins]

in setup.cfg we currently have

[mypy]
ignore_missing_imports=True

mypy states "We recommend using this approach only as a last resort: it’s equivalent to adding a # type: ignore to all unresolved imports in your codebase." https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports

the number of instances of missing imports resolving to Any is already significant..

$ mypy pandas --disallow-any-unimported
pandas\util\_test_decorators.py:106: error: Return type becomes "Any" due to an unfollowed import
pandas\util\_test_decorators.py:120: error: Return type becomes "Any" due to an unfollowed import
pandas\_typing.py:19: error: Constraint 5 becomes "Any" due to an unfollowed import
pandas\_typing.py:22: error: Constraint 2 becomes "Any" due to an unfollowed import
pandas\_typing.py:23: error: Constraint 1 becomes "Any" due to an unfollowed import
pandas\_typing.py:23: error: Constraint 2 becomes "Any" due to an unfollowed import
pandas\_typing.py:23: error: Constraint 3 becomes "Any" due to an unfollowed import
pandas\io\formats\printing.py:252: error: Base type BaseFormatter becomes "Any" due to an unfollowed import
pandas\core\dtypes\dtypes.py:653: error: Type of variable becomes "Type[Any]" due to an unfollowed import
pandas\core\dtypes\dtypes.py:812: error: Type of variable becomes "Type[Any]" due to an unfollowed import
pandas\core\dtypes\common.py:144: error: Return type becomes "Any" due to an unfollowed import
pandas\core\dtypes\common.py:144: error: Argument 1 to "ensure_int_or_float" becomes "Any" due to an unfollowed import
pandas\core\dtypes\common.py:181: error: Argument 1 to "ensure_python_int" becomes "Union[int, Any]" due to an unfollowed import
pandas\core\indexers.py:39: error: Argument 2 to "is_empty_indexer" becomes "Any" due to an unfollowed import
pandas\core\indexers.py:115: error: Argument 1 to "validate_indices" becomes "Any" due to an unfollowed import
pandas\core\dtypes\cast.py:765: error: Argument 1 to "maybe_convert_objects" becomes "Any" due to an unfollowed import
pandas\core\dtypes\cast.py:814: error: Argument 1 to "soft_convert_objects" becomes "Any" due to an unfollowed import
pandas\core\nanops.py:194: error: Return type becomes "Optional[Any]" due to an unfollowed import
pandas\core\nanops.py:194: error: Argument 1 to "_maybe_get_mask" becomes "Any" due to an unfollowed import
pandas\core\nanops.py:194: error: Argument 3 to "_maybe_get_mask" becomes "Optional[Any]" due to an unfollowed import
pandas\core\nanops.py:237: error: Return type becomes "Tuple[Any, Optional[Any], Any, Any, Any]" due to an unfollowed import
pandas\core\nanops.py:237: error: Argument 1 to "_get_values" becomes "Any" due to an unfollowed import
pandas\core\nanops.py:237: error: Argument 5 to "_get_values" becomes "Optional[Any]" due to an unfollowed import
pandas\core\nanops.py:634: error: Return type becomes "Tuple[Union[int, Any], Union[int, Any]]" due to an unfollowed import
pandas\core\nanops.py:634: error: Argument 2 to "_get_counts_nanvar" becomes "Optional[Any]" due to an unfollowed import
pandas\core\nanops.py:672: error: Type of variable becomes "Any" due to an unfollowed import
pandas\core\nanops.py:1117: error: Return type becomes "Union[Any, int]" due to an unfollowed import
pandas\core\nanops.py:1117: error: Argument 1 to "_maybe_arg_null_out" becomes "Any" due to an unfollowed import
pandas\core\nanops.py:1117: error: Argument 3 to "_maybe_arg_null_out" becomes "Optional[Any]" due to an unfollowed import
pandas\core\nanops.py:1141: error: Return type becomes "Union[int, Any]" due to an unfollowed import
pandas\core\nanops.py:1141: error: Argument 2 to "_get_counts" becomes "Optional[Any]" due to an unfollowed import
pandas\core\nanops.py:1185: error: Return type becomes "Any" due to an unfollowed import
pandas\core\nanops.py:1185: error: Argument 1 to "_maybe_null_out" becomes "Any" due to an unfollowed import
pandas\core\nanops.py:1185: error: Argument 3 to "_maybe_null_out" becomes "Optional[Any]" due to an unfollowed import
pandas\core\construction.py:51: error: Return type becomes "Any" due to an unfollowed import
pandas\core\construction.py:51: error: Argument 2 to "array" becomes "Union[str, Any, ExtensionDtype, None]" due to an unfollowed import
pandas\core\tools\datetimes.py:71: error: Argument 1 to "should_cache" becomes "Union[List[Any], Tuple[Any, ...], Any, Any]" due to an unfollowed import
pandas\core\tools\datetimes.py:161: error: Return type becomes "Any" due to an unfollowed import
pandas\core\tools\datetimes.py:161: error: Argument 1 to "_box_as_indexlike" becomes "Any" due to an unfollowed import
pandas\core\tools\datetimes.py:191: error: Return type becomes "Union[Any, Any]" due to an unfollowed import
pandas\core\tools\datetimes.py:191: error: Argument 1 to "_convert_and_box_cache" becomes "Union[Any, List[Any], Tuple[Any, ...], Any, Any]" due to an unfollowed import
pandas\core\tools\datetimes.py:191: error: Argument 2 to "_convert_and_box_cache" becomes "Any" due to an unfollowed import
pandas\tseries\offsets.py:157: error: Base type BaseOffset becomes "Any" due to an unfollowed import
pandas\tseries\offsets.py:2513: error: Base type _Tick becomes "Any" due to an unfollowed import
pandas\core\arrays\_ranges.py:15: error: Return type becomes "Tuple[Any, str]" due to an unfollowed import
pandas\core\arrays\_ranges.py:15: error: Argument 1 to "generate_regular_range" becomes "Any" due to an unfollowed import
pandas\core\arrays\_ranges.py:15: error: Argument 2 to "generate_regular_range" becomes "Any" due to an unfollowed import
pandas\core\ops\__init__.py:398: error: Argument 1 to "should_extension_dispatch" becomes "Any" due to an unfollowed import
pandas\core\ops\__init__.py:1152: error: Argument 1 to "maybe_dispatch_ufunc_to_dunder_op" becomes "Any" due to an unfollowed import
pandas\core\ops\__init__.py:1152: error: Argument 4 to "maybe_dispatch_ufunc_to_dunder_op" becomes "Any" due to an unfollowed import
pandas\core\arrays\period.py:230: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\period.py:230: error: Argument 2 to "_from_sequence" becomes "Sequence[Optional[Any]]" due to an unfollowed import
pandas\core\arrays\period.py:301: error: Argument 2 to "_unbox_scalar" becomes "Union[Any, Any]" due to an unfollowed import
pandas\core\arrays\period.py:313: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\period.py:655: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\period.py:655: error: Argument 2 to "_addsub_int_array" becomes "Union[Any, Any, Any, Any]" due to an unfollowed import
pandas\core\arrays\period.py:833: error: Argument 1 to "period_array" becomes "Sequence[Optional[Any]]" due to an unfollowed import
pandas\core\arrays\datetimes.py:330: error: Type of variable becomes "Union[Any, DatetimeTZDtype]" due to an unfollowed import
pandas\core\arrays\datetimes.py:604: error: Return type becomes "Union[Any, DatetimeTZDtype]" due to an unfollowed import
pandas\core\arrays\base.py:267: error: Argument 2 to "__setitem__" becomes "Union[int, Any]" due to an unfollowed import
pandas\core\arrays\base.py:392: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\base.py:414: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\base.py:525: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\base.py:637: error: Return type becomes "Tuple[Any, Any]" due to an unfollowed import
pandas\core\arrays\base.py:661: error: Return type becomes "Tuple[Any, Any]" due to an unfollowed import
pandas\core\arrays\base.py:764: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\base.py:855: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\base.py:865: error: Return type becomes "Union[Any, Any]" due to an unfollowed import
pandas\core\arrays\base.py:936: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\base.py:956: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\base.py:956: error: Argument 2 to "_concat_same_type" becomes "Sequence[Any]" due to an unfollowed import
pandas\core\arrays\base.py:980: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\numpy_.py:89: error: Base type NDArrayOperatorsMixin becomes "Any" due to an unfollowed import
pandas\core\arrays\datetimelike.py:58: error: Type of variable becomes "Any" due to an unfollowed import
pandas\core\arrays\datetimelike.py:65: error: Return type becomes "Type[Any]" due to an unfollowed import
pandas\core\arrays\datetimelike.py:74: error: Return type becomes "Union[Any, Any, Any, Any]" due to an unfollowed import
pandas\core\arrays\datetimelike.py:96: error: Argument 2 to "_unbox_scalar" becomes "Union[Any, Any, Any, Any]" due to an unfollowed import
pandas\core\arrays\datetimelike.py:115: error: Argument 2 to "_check_compatible_with" becomes "Union[Any, Any, Any, Any]" due to an unfollowed import
pandas\core\arrays\datetimelike.py:348: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\datetimelike.py:453: error: Argument 3 to "__setitem__" becomes "Union[Any, Any, Sequence[Any]]" due to an unfollowed import
pandas\core\base.py:990: error: Return type becomes "Any" due to an unfollowed import
pandas\core\indexes\frozen.py:123: error: Base type ndarray becomes "Any" due to an unfollowed import
pandas\core\indexes\base.py:3871: error: Return type becomes "Union[ExtensionArray, Any, Any]" due to an unfollowed import
pandas\core\arrays\sparse.py:109: error: Argument 2 to "__init__" becomes "Union[str, Any, ExtensionDtype]" due to an unfollowed import
pandas\core\arrays\sparse.py:393: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\sparse.py:393: error: Argument 1 to "_get_fill" becomes "Any" due to an unfollowed import
pandas\core\arrays\sparse.py:417: error: Argument 1 to "_sparse_array_op" becomes "Any" due to an unfollowed import
pandas\core\arrays\sparse.py:417: error: Argument 2 to "_sparse_array_op" becomes "Any" due to an unfollowed import
pandas\core\arrays\sparse.py:706: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\sparse.py:706: error: Argument 2 to "_simple_new" becomes "Any" due to an unfollowed import
pandas\core\arrays\sparse.py:706: error: Argument 3 to "_simple_new" becomes "Any" due to an unfollowed import
pandas\core\arrays\interval.py:143: error: Base type IntervalMixin becomes "Any" due to an unfollowed import
pandas\core\arrays\integer.py:519: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\integer.py:576: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\categorical.py:479: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\categorical.py:502: error: Argument 2 to "astype" becomes "Union[str, Any, ExtensionDtype]" due to an unfollowed import
pandas\core\arrays\categorical.py:502: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\categorical.py:2652: error: Argument 1 to "_recode_for_categories" becomes "Any" due to an unfollowed import
pandas\core\internals\blocks.py:1858: error: Argument 3 to "shift" becomes "Any" due to an unfollowed import
pandas\core\indexes\multi.py:64: error: Base type BaseMultiIndexCodesEngine becomes "Any" due to an unfollowed import
pandas\core\indexes\multi.py:64: error: Base type UInt64Engine becomes "Any" due to an unfollowed import
pandas\core\indexes\multi.py:103: error: Base type BaseMultiIndexCodesEngine becomes "Any" due to an unfollowed import
pandas\core\indexes\multi.py:103: error: Base type ObjectEngine becomes "Any" due to an unfollowed import
pandas\core\indexes\category.py:525: error: Argument 2 to "get_value" becomes "Any" due to an unfollowed import
pandas\core\indexes\range.py:79: error: Type of variable becomes "Any" due to an unfollowed import
pandas\core\indexes\range.py:363: error: Argument 2 to "__contains__" becomes "Union[int, Any]" due to an unfollowed import
pandas\core\indexes\interval.py:201: error: Base type IntervalMixin becomes "Any" due to an unfollowed import
pandas\core\indexes\interval.py:629: error: Argument 2 to "_can_reindex" becomes "Any" due to an unfollowed import
pandas\core\indexes\interval.py:790: error: Return type becomes "Union[int, slice, Any]" due to an unfollowed import
pandas\core\indexes\interval.py:861: error: Return type becomes "Any" due to an unfollowed import
pandas\core\indexes\interval.py:861: error: Argument 2 to "get_indexer" becomes "Any" due to an unfollowed import
pandas\core\indexes\interval.py:931: error: Return type becomes "Tuple[Any, Any]" due to an unfollowed import
pandas\core\indexes\interval.py:931: error: Argument 2 to "get_indexer_non_unique" becomes "Any" due to an unfollowed import
pandas\core\indexes\interval.py:972: error: Return type becomes "Any" due to an unfollowed import
pandas\core\indexes\interval.py:972: error: Argument 2 to "get_indexer_for" becomes "Any" due to an unfollowed import
pandas\core\indexes\interval.py:989: error: Argument 2 to "get_value" becomes "Any" due to an unfollowed import
pandas\plotting\_matplotlib\converter.py:123: error: Base type ConversionInterface becomes "Any" due to an unfollowed import
pandas\plotting\_matplotlib\converter.py:150: error: Base type Formatter becomes "Any" due to an unfollowed import
pandas\plotting\_matplotlib\converter.py:193: error: Base type DateConverter becomes "Any" due to an unfollowed import
pandas\plotting\_matplotlib\converter.py:252: error: Base type DateConverter becomes "Any" due to an unfollowed import
pandas\plotting\_matplotlib\converter.py:325: error: Base type AutoDateFormatter becomes "Any" due to an unfollowed import
pandas\plotting\_matplotlib\converter.py:330: error: Base type AutoDateLocator becomes "Any" due to an unfollowed import
pandas\plotting\_matplotlib\converter.py:355: error: Base type DateLocator becomes "Any" due to an unfollowed import
pandas\plotting\_matplotlib\converter.py:936: error: Base type Locator becomes "Any" due to an unfollowed import
pandas\plotting\_matplotlib\converter.py:1030: error: Base type Formatter becomes "Any" due to an unfollowed import
pandas\plotting\_matplotlib\converter.py:1098: error: Base type Formatter becomes "Any" due to an unfollowed import
pandas\io\formats\format.py:1294: error: Return type becomes "Any" due to an unfollowed import
pandas\io\formats\format.py:1400: error: Argument 2 to "__init__" becomes "Union[Any, Series, DatetimeIndex, DatetimeArray]" due to an unfollowed import
pandas\io\formats\format.py:1457: error: Argument 1 to "format_percentiles" becomes "Union[Any, List[Union[int, float]], List[float], List[Union[str, float]]]" due to an unfollowed import
pandas\io\formats\format.py:1527: error: Argument 1 to "_is_dates_only" becomes "Union[Any, DatetimeArray, Index, DatetimeIndex]" due to an unfollowed import
pandas\io\formats\format.py:1548: error: Argument 1 to "_format_datetime64" becomes "Union[Any, Any]" due to an unfollowed import
pandas\io\formats\format.py:1548: error: Argument 2 to "_format_datetime64" becomes "Union[Any, tzutc, None]" due to an unfollowed import
pandas\io\formats\format.py:1565: error: Argument 1 to "_format_datetime64_dateonly" becomes "Union[Any, Any]" due to an unfollowed import
pandas\io\formats\format.py:1592: error: Argument 1 to "_get_format_datetime64_from_values" becomes "Union[Any, DatetimeArray, DatetimeIndex]" due to an unfollowed import
pandas\io\formats\format.py:1623: error: Argument 2 to "__init__" becomes "Union[Any, TimedeltaIndex]" due to an unfollowed import
pandas\io\formats\format.py:1641: error: Argument 1 to "_get_format_timedelta64" becomes "Union[Any, TimedeltaIndex, TimedeltaArray]" due to an unfollowed import
pandas\io\formats\format.py:1719: error: Argument 1 to "_trim_zeros_complex" becomes "Any" due to an unfollowed import
pandas\io\formats\format.py:1730: error: Argument 1 to "_trim_zeros_float" becomes "Union[Any, List[str]]" due to an unfollowed import
pandas\io\formats\format.py:1872: error: Argument 1 to "_binify" becomes "List[Any]" due to an unfollowed import
pandas\io\formats\format.py:1872: error: Argument 2 to "_binify" becomes "Union[Any, int]" due to an unfollowed import
pandas\core\indexing.py:102: error: Base type _NDFrameIndexerBase becomes "Any" due to an unfollowed import
pandas\core\indexing.py:650: error: Argument 3 to "_align_series" becomes "Any" due to an unfollowed import
pandas\core\indexing.py:742: error: Argument 3 to "_align_frame" becomes "Any" due to an unfollowed import
pandas\core\indexing.py:2162: error: Base type _NDFrameIndexerBase becomes "Any" due to an unfollowed import
pandas\core\indexing.py:2371: error: Return type becomes "Any" due to an unfollowed import
pandas\core\internals\managers.py:1874: error: Return type becomes "Union[Any, ExtensionDtype, None]" due to an unfollowed import
pandas\core\generic.py:164: error: Argument 5 to "__init__" becomes "Union[str, Any, ExtensionDtype, None]" due to an unfollowed import
pandas\core\frame.py:6535: error: Return type becomes "Union[Series, Any]" due to an unfollowed import
pandas\core\frame.py:6535: error: Argument 4 to "_gotitem" becomes "Union[Series, Any, None]" due to an unfollowed import
pandas\core\groupby\groupby.py:1087: error: Return type becomes "Tuple[Any, Type[Any]]" due to an unfollowed import
pandas\core\groupby\groupby.py:1087: error: Argument 1 to "objs_to_bool" becomes "Any" due to an unfollowed import
pandas\core\groupby\groupby.py:1095: error: Return type becomes "Any" due to an unfollowed import
pandas\core\groupby\groupby.py:1095: error: Argument 1 to "result_to_bool" becomes "Any" due to an unfollowed import
pandas\core\groupby\groupby.py:1879: error: Return type becomes "Tuple[Any, Optional[Type[Any]]]" due to an unfollowed import
pandas\core\groupby\groupby.py:1879: error: Argument 1 to "pre_processor" becomes "Any" due to an unfollowed import
pandas\core\groupby\groupby.py:1894: error: Return type becomes "Any" due to an unfollowed import
pandas\core\groupby\groupby.py:1894: error: Argument 1 to "post_processor" becomes "Any" due to an unfollowed import
pandas\io\packers.py:800: error: Base type _Packer becomes "Any" due to an unfollowed import
pandas\io\packers.py:820: error: Base type _Unpacker becomes "Any" due to an unfollowed import
pandas\core\window\rolling.py:209: error: Return type becomes "Optional[Any]" due to an unfollowed import
pandas\core\window\rolling.py:222: error: Return type becomes "Any" due to an unfollowed import
pandas\core\window\rolling.py:222: error: Argument 2 to "_prep_values" becomes "Optional[Any]" due to an unfollowed import
pandas\core\window\rolling.py:335: error: Return type becomes "Any" due to an unfollowed import
pandas\core\window\rolling.py:352: error: Argument 4 to "_get_roll_func" becomes "Any" due to an unfollowed import
pandas\core\window\rolling.py:788: error: Return type becomes "Any" due to an unfollowed import
pandas\core\window\rolling.py:844: error: Argument 4 to "_get_roll_func" becomes "Any" due to an unfollowed import

Ideas for keeping this manageable?

cc @WillAyd

[WillAyd]

Yea I started along this path in #26645 . The hang up on that was some of the internal imports which (maybe?) require stubs. Never went far beyond that so if you want to dive deeper I think still worth exploring

[simonjayhawkins]

I think we should create stubs for pyx files.

It would be nice if this could be done automatically (@mrocklin wrote http://matthewrocklin.com/blog/work/2017/11/05/cython-everywhere which mentioned use of cython in regular .py files instead of creating .pyx files)

if we create the stubs manually, i suspect there would not be an issue keeping them in sync, since changes to the code for .pyx file changes would likely cause mypy to report an error if the stub wasn't updated as well.

[WillAyd]

Cool yea I agree with everything you've mentioned above. Article is certainly interesting as well - not sure what limitations there are to that approach but worth exploring if you've got time and interest I think

[rhshadrach]

I believe we will never be able to set ignore_missing_imports=False unless all dependencies (optional or not) publish stubs or add a py.typed marker. If this happens - great, but otherwise, I think there is nothing we can do here. Marking as a closing candidate.

[simonjayhawkins]

In order to close this issue appropriately, I propose that we move forward by setting ignore_missing_imports=False in our configuration. This change means that for every import from a library that lacks published stubs or a py.typed marker, we’ll need to explicitly add a # type: ignore comment. Although this might increase the number of ignore comments in our codebase, it aligns with our long-term goal of comprehensive type safety.

When this issue was first raised, the choice of ignore_missing_imports=True was driven by the concern that enforcing explicit annotations from untyped libraries would force us to sprinkle many # type: ignore statements—a practice that was regarded as undesirable by some maintainers at the time. However, as our ecosystem evolves and more dependencies adopt proper type support, targeted ignore comments can serve as a pragmatic, transitional measure. Any untyped imports become clear candidates for refinement once the libraries catch up with type annotations.

In summary, while a fully strict type checking environment (i.e., ignore_missing_imports=False without any ignores) would require every dependency to ship with stubs or a py.typed marker, the practical interim solution is to adopt ignore_missing_imports=False and manage exceptions on a case-by-case basis.

This approach makes our intention explicit and establishes a clear path forward while still accounting for external constraints. This issue was to discuss and manage that transition and probably should not yet be closed.