BUG: pandas 2.1.2 changes how `copy` works

[flying-sheep]

Pandas version checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of pandas.
• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

class Sub(pd.DataFrame): pass

assert not isinstance(Sub().copy(), Sub)

Issue Description

Since very long ago, calling .copy() on a DataFrame subclass returned a DataFrame object. This should be changed in a major release, not a feature release, and definitely not a patch release.

Expected Behavior

The above assert statement to succeed

See also

• Probably related: BUG: Infinite recursion on inheritance and overloading __getattr__ in v2.1.0 #55120
• Downstream bug report: DataFrameView in pandas 2.1.2 changed behavior, breaking AnnData in several ways scverse/anndata#1210

Installed Versions

pandas 2.1.2 or 2.2.0.dev0+447

[flying-sheep]

We’re pinning Pandas to !=2.1.2.

Can we rely on Pandas 2.2.0 to fix this?

[ivirshup]

I believe this was specifically caused by the changes to: _constructor_from_mgr in #54922. Previously (2.1.1), this looked like:

    def _constructor_from_mgr(self, mgr, axes):
        df = self._from_mgr(mgr, axes=axes)

        if type(self) is DataFrame:
            # fastpath avoiding constructor call
            return df
        else:
            assert axes is mgr.axes
            return self._constructor(df, copy=False)

But now it looks like:

    def _constructor_from_mgr(self, mgr, axes):
        if self._constructor is DataFrame:
            # we are pandas.DataFrame (or a subclass that doesn't override _constructor)
            return self._from_mgr(mgr, axes=axes)
        else:
            assert axes is mgr.axes
            return self._constructor(mgr)

Behaviour changed for other methods, like .groupby, as well.

I believe we were relying on the constructor being called for our subclass, since we want to coerce to apd.DataFrame here. However, it's unclear if there is any mechanism for doing that because we would want ._constructor to be pd.DataFrame, which is specifically branching here.

I've opened a draft PR which may fix this:

• REGR: fix return class in _constructor_from_mgr for simple subclasses #55764

This PR reverts the branching logic to check the class being passed in instead of the class of _constructor. It seems to work locally, but I'm not sure I totally understand why the bug inducing change was made.

Would appreciate a look from @jorisvandenbossche.

[jorisvandenbossche]

Thanks for the detailed report and analysis, @flying-sheep and @ivirshup !
It's unfortunate that the fix for a recursion error in one subclass caused a recursion error in a different subclass ..

But yes, this is something we definitely should fix. I didn't consider the case of subclasses that don't get preserved but always return a plain pandas object for each operation.