DEPR: __finalize__ and _metadata

[jbrockmendel]

We have some support for propagating metadata, but it is incomplete (38 xfails in tests/generic/test_finalize.py) for plain methods and has no real prospect for getting robust support for concat/merge etc.

At the same time, there is a real performance cost to calling __finalize__ everywhere, in an extreme case like #46505 these calls take up a quarter of the runtime.

AFAICT this is useful for a fraction of users and perf-hurting for everyone else. Do we have any idea how common it is to rely on these?

cc @MarcoGorelli

[MarcoGorelli]

attrs already says

attrs is experimental and may change without warning.

TBH I'd just remove it as a breaking change

EDIT: as clarified below, although __finalize__ does set attrs, this issue is specifically about the former. See #52166 for discussion around attrs

[MarcoGorelli]

> 38 xfails in tests/generic/test_finalize.py

And that's without counting those which are failing, but which the test isn't catching because it's written incorrectly: #49916

> Do we have any idea how common it is to rely on these?

From my scrape of the 100 most upvoted Python notebooks on Kaggle, it appears exacty 0 times

If anyone's using it, then their code's already broken as attrs never worked to begin with

EDIT: as clarified below, although __finalize__ does set attrs, this issue is specifically about the former. See #52166 for discussion around attrs

[mroeschke]

We do use this in the interchange protocol:

pandas/pandas/core/interchange/from_dataframe.py

Line 135 in 180d81f

pandas_df.attrs["_INTERCHANGE_PROTOCOL_BUFFERS"] = buffers

[jorisvandenbossche]

I think this is mixing different things: __finalize__ and _metadata are long-standing documented features available for developers subclassing DataFrame/Series (https://pandas.pydata.org/docs/dev/development/extending.html#subclassing-pandas-data-structures, __finalize__ actually isn't mentioned, but _metadata is and works through __finalize__ propagating it), while attrs is a newer (officially still experimental?), end-user facing attempt to provide some form of metadata.

Do we have any idea how common it is to rely on these?

From my scrape of the 100 most upvoted Python notebooks on Kaggle, it appears exacty 0 times

For the __finalize__/_metadata, you don't have to look for public notebooks, but for packages depending on pandas that implement subclasses.
And to give one example: GeoPandas makes use of both __finalize__ and _metadata.

[MarcoGorelli]

Thanks @jorisvandenbossche !

A question is have is - is GeoPandas affected by the many bugs in __finalize__?

EDIT: I'll take a look at how geopandas uses them, let's not rush anything

[jbrockmendel]

Looks like geopandas uses __finalize__ mostly for merge/concat, which is not a place with an easy "ask them to do X instead". The _metadata they define is _geometry_column_name which looks like it has a reasonable default value that users can override. IIUC this is used to check that when you concat/merge you don't end up with duplicates of that particular label.

Also our own flags is propagated via __finalize__. The only flag is allows_duplicate_labels. I don't see a clear way to deprecate __finalize__ without also deprecating flags.

I'd be OK with deprecating flags and having allow_duplicate_labels checking move to third-party validation libraries.

[jorisvandenbossche]

Looks like geopandas uses __finalize__ mostly for merge/concat,

GeoPandas customizes __finalize__ for merge/concat, but we still make use of the default __finalize__ (propagating _metadata) for many other operations where pandas ensures __finalize__ is called.
So we basically use __finalize__ for any method on a GeoDataFrame that we inherit from pandas.DataFrame and that returns a new GeoDataFrame (to ensure that the geometry name information is preserved).

A question is have is - is GeoPandas affected by the many bugs in __finalize__?

Occasionally we run into an issue where the metadata is not properly propagated from eg a missing __finalize__ call, but then we can fix this, and go on. But generally it's not a big problem at all. GeoPandas is a happy user of this feature.
(and it's not because a feature has bugs that that in itself is a reason to throw it away, otherwise we can just stop with developing the full of pandas ;))

Performance is certainly an important aspect that I don't want to just ignore. But I think we should then do some more analysis of the extent of the problem in actual use cases, to know if this is actually a problem (or how big it is, and to see if there are ways to address it without throwing away the feature in its entirety).
#46505 is one such example, but I want to note that I would consider that as quite a corner case that also slowed down for other reasons than only __finalize__ (the reproducer there uses 500_000 groups each consisting of just 1 row, so creating a lot of tiny DataFrame or Series objects, exaggerating the slow down you will typically see). And one that we can also already improve without removing __finalize__ (will comment on that issue about that).

[jbrockmendel]

Another option (i think this was @phofl's idea) would be to keep __finalize__ but make it a no-op, so libraries could override it if they need the metadata/attrs/flags bits.