ENH: Add set of tests to type check the public APIs

[Dr-Irv]

From discussion on pandas typing meeting on January 7, 2022 (notes at https://docs.google.com/document/d/1tGbTiYORHiSPgVMXawiweGJlBw5dOkVJLY-licoBmBU/)

Follow up to discussion here: #28142 (comment)

Goal is to have a set of tests that make sure that users of the pandas API writing correct code work with whatever type stubs we publish.

Example of an approach that is suggested was developed within the pandas-stubs project based on various code snippets which are then tested with mypy : https://github.com/VirtusLab/pandas-stubs/tree/master/tests/snippets
@zkrolikowski-vl and @joannasendorek from that project have offered to help (nudge, nudge)

Ideally we would add this testing framework to the CI before starting to bring over the stubs.

[simonjayhawkins]

Example of an approach that is suggested was developed within the pandas-stubs project based on various code snippets which are then tested with mypy : https://github.com/VirtusLab/pandas-stubs/tree/master/tests/snippets

see #40202 for a less verbose but more primitive approach.

Using pytest is perhaps more robust. How easy is is to parametrize testing of types? I assume that parametrization cannot be used for static checks.

The method in #40202 has several issues. If anyone knows of other approaches to testing types, would much appreciate posting suggestions here.

[simonjayhawkins]

from #40202 (comment)

we must also remember that there is also the "do nothing" option and not explicitly test the types. IIRC this was suggested in the January dev meeting (Mostly relying on user feedback to report issues.)

repeating here so that this option is not overlooked if the solution gets too complex.

[twoertwein]

One potentially easier to implement option is to have something like mypy_primer but for pandas. Typeshed uses mypy_primer to automatically comment on each PR whether it would change type annotations as inferred by mypy.

"Pandas_primer" would help to 1) prevent involuntary changes to public type annotations and to 2) verify that intended changes actually have an effect.

[zkrolikowski-vl]

In pandas-stubs, we've decided to go with the simplest method, which doesn't require any extra setup or testing constructs. Here are some of the other tools we've looked at (not including dynamic type-checkers):

• pytest-mypy-plugins - I wasn't able to make it work with stub files - it seemed to ignore them.
• mypy.test.testcheck - built with testing mypy itself specifically. For a while I've tried to make it accept a different project. Looked promising (see example here ) but no luck.
• pytest-mypy-testing Worked somewhat like intended but it reported errors from inside the testing library itself. Very similar to TST/TYP: start testing public api types #40202 by @simonjayhawkins.

Coming back to our solution, there's a way to test negative scenarios in our approach but it's far from elegant:

    rdf: pd.Series = pd.concat({1: df, 2: df2})     # type: ignore

This will cause mypy not to return errors when the line in-fact is incorrect,

whereas using it incorrectly

    rdf: pd.DataFrame = pd.concat({1: df, 2: df2})     # type: ignore

will result in an error

tests/snippets/test_pandas.py:55: error: Unused "type: ignore" comment

The issue that remains is that our whole approach is static and I don't see it being easily integrated with parametric list of use-cases.

mypy_primer seems like an another interesting approach. Potentially difficult to integrated into pandas CI but might be worth the extra effort.

Overall it seems like either pytest-mypy-testing/ @simonjayhawkins 's (with some work) or our (pandas-stubs) solution might be a semi-immediate fit for pandas use-case.

@simonjayhawkins if I may ask: Why was your work discontinued? Was it just inconvenient at the time?

[Dr-Irv]

Here is why I like the approach taken by the pandas-stubs project: https://github.com/VirtusLab/pandas-stubs/tree/master/tests/snippets

• We need a way to test the typing stubs we publish.
• When they are published, it is like Goldilocks and the Three Bears. A particular stub could be too narrow, too wide, or just right.
• The community will discover that certain stubs are too narrow or too wide.
• As a stub is fixed, you can add tests for ones that were too narrow by adding test code that should pass a mypy test
• As a stub is fixed, you can add tests for ones that were too wide by adding test code with #type: ignore comments and use the mypy warn_unused_ignores option to make sure that the ignore line is correct (and tested with future changes to the stubs)

From my experience in submitting a bunch of PR's to the Microsoft python-type-stubs project, which has no testing for the pandas stubs, I can tell you that I am always concerned that a fix I submit for the stubs will break something else. If they had included something like what the pandas-stubs project did, it would make life a lot easier to do more test-driven development of the stubs. This is especially true when you make a change in pandas/_typing.py that changes one of the top-level types. It's hard to know if you have then made some downstream stub too narrow or too wide.

[joannasendorek]

The approach we took in pandas-stubs was the most pragmatical for our usage and capacity we have to maintain the project. We can assume that any stubs will not be perfect and when people will first start using them, they will get some errors (I applied our stubs for one of the thousand lines project and made 10 fixes to pandas-stubs based on that only). So whenever this happens, we would first re-create the false negative case provided, modify the stubs and make sure the exact usage is corrected. From this moment, this particular usage is guarded so people using the project shouldn't notice regression and we can add only small improvements at once.

[simonjayhawkins]

@simonjayhawkins if I may ask: Why was your work discontinued? Was it just inconvenient at the time?

abandoned due to #40202 (comment). In pandas we tend not merge stuff if there are any objections.

I assumed from that position and the notes from the meeting...

Pandas-stubs can reorganize their test files to correspond better to current pandas testing

that the pandas-stubs would be integrated into out pytest tests, but I guess I got that wrong from not attending the meeting.

@Dr-Irv I assume from #45252 (comment) that your objections to previous attempt no longer apply?

[Dr-Irv]

that the pandas-stubs would be integrated into out pytest tests, but I guess I got that wrong from not attending the meeting.

What we decided is that the testing approach from pandas-stubs would be integrated, but we would use the Microsoft stubs as a starting point, because they are more complete than the pandas-stubs .

@Dr-Irv I assume from #45252 (comment) that your objections to previous attempt no longer apply?

I'm not sure which objections you are referring to....

What I said there is that if we could annotate all of our existing tests (which means adding #type: ignore statements in lots of places), that would be a way to test the stubs. But I think that would take a lot of effort.

Since then, I've seen the pandas-stubs approach and I think it is easier to start from there, and expand the methodology in there to cover many use cases of any stubs we decide to publish.

[simonjayhawkins]

I'm not sure which objections you are referring to....

specifically from #40202 (comment)

I think it will be easier for the community to fix our source code that has typed methods for the public API (and add an appropriate test in pandas/tests) as opposed to using the reveal_type method proposed in this PR, which would require us to enumerate all the possibilities in two places - once in our regular testing code and once in the "testing types" code.

Since then, I've seen the pandas-stubs approach and I think it is easier to start from there, and expand the methodology in there to cover many use cases of any stubs we decide to publish.

IIUC the panda-stubs approach, if not using pytest, is not so different from #40202. i.e. we need to enumerate all the possibilities and have a separate set of tests.

also from #40202 (comment)

In addition, it's not clear to me how the current proposal helps us test the variety of arguments that can be passed to methods. We know that our existing testing code already tries different argument types, so if we typed our testing code (and ran mypy on it), then we're getting the benefit of one set of tests that tests functionality as well as typing signatures.

The pandas-stubs testing approach, if not integrated into the current testing, also does not satisfy this.