Unexpected, Misleading mypy Failures After pandas-stubs Update: DataFrame.from_dict() Behavior

[pmaier-bhs]

Describe the bug
The DataFrame.from_dict() method allows parsing lists of dictionaries, where each dictionary is interpreted as a single row. However, this behavior is not reflected in the typed method signatures coded in pandas-stubs. We use that behavior and add # type: ignore comments to suppress mypy errors.

This worked without issue up until pandas-stubs version 2.2.1.240316, but with the update to version 2.2.2.240514, it leads to unexpected mypy failures, see below. The specific change responsible might be this commit.

To Reproduce

1. Provide a minimal runnable pandas example that is not properly checked by the stubs.

import pandas as pd

# %% mypy error Returning Any from function declared to return "int"  [no-any-return]


def f() -> int:
    b = [
        {"key1": "value1", "key2": 42},
        {"key1": "value2", "key2": 123},
    ]
    df = pd.DataFrame.from_dict(b)  # type: ignore
    return df.shape[0]

3. Indicate which type checker you are using (mypy or pyright).

mypy

5. Show the error message received from that type checker while checking your example.

✕ mypy failed.
17:19:33.84 [ERROR] Completed: Typecheck using MyPy - mypy - mypy failed (exit code 1).
src/python/mdl-data-insertion/mdl_data_insertion/util/typing_problem.py:12: error: Returning Any from function declared to return "int"  [no-any-return]
Found 1 error in 1 file (checked 1 source file)

Please complete the following information:

• OS: MacOS
• OS Version 14.5 (23F79)
• python version 3.10.13
• version of type checker: 1.9.0
• version of installed pandas-stubs: 2.2.2.240514

[pmaier-bhs]

Btw. the following example works, i.e. shows no issue:

b = [
    {"key1": "value1", "key2": 42},
    {"key1": "value2", "key2": 123},
]
df = pd.DataFrame.from_dict(b)  # type: ignore
i: int = df.shape[0]

[Dr-Irv]

You wrote:

The DataFrame.from_dict() method allows parsing lists of dictionaries

If I look at the docs https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.from_dict.html , it doesn't say that a list of dict is allowed. So while that may work, that may be a pandas bug or a documentation bug. Can you create an issue in pandas about the docs of to_dict() and see what the pandas core team has to say?

[Dr-Irv]

Related to #929 .

[pmaier-bhs]

You wrote:

The DataFrame.from_dict() method allows parsing lists of dictionaries

If I look at the docs https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.from_dict.html , it doesn't say that a list of dict is allowed. So while that may work, that may be a pandas bug or a documentation bug. Can you create an issue in pandas about the docs of to_dict() and see what the pandas core team has to say?

Sure!

[pmaier-bhs]

Created a pandas issue, see pandas-dev/pandas#58862.

[loicdiridollou]

I think we should be able to close that issue since there will be a check not to allow the input of list.
Did I miss anything on this one @Dr-Irv ?

[Dr-Irv]

Since pandas will eventually deprecate allowing a list of dictionaries to be passed to from_dict() (see pandas-dev/pandas#58862 (comment)) we will not support this in the stubs.