DEPR: attrs

[jbrockmendel]

Discussion broken off from #51280
PR #52152

Propagation of attrs in __finalize__ is a small-but-everywhere performance hit that we should deprecate.

[jorisvandenbossche]

For context, this feature was added in pandas 1.0 (#29062, cc @TomAugspurger).

I personally have no idea how much attrs specifically is being used since it was introduced, but in general the ability to store metadata is a topic that has come up a lot. From a quick browsing of our issues, some related ones:

• Allow custom metadata to be attached to panel/df/series? #2485 (one of our most upvoted issues, which was closed pointing to attrs)
• DOC/API: document how to use metadata #8572
• ENH: pandas.DataFrame.attrs #44689
• QST: When will attrs be not experimental? #49085

(from the last two issues, it seems there is certainly some user interest in the specific attrs feature)

[TomAugspurger]

From Joris in #51280 (comment)

If performance is the main argument that we would want to deprecate the related features to metadata propagation (attrs/flags, #52165 and #52166, where it is currently the only argument), I think we need some more investigation / proof that this is actually a problem.

That's been bugging me too. I haven't looked at the performance, but copying the metadata should just be a dictionary merge / update.

At the end of the day we'll be making a value judgement: is the performance cost worth it. We'll need a clearer idea of performance cost.

[jbrockmendel]

where it is currently the only argument

The other argument is that attrs/_metadata is only half-implemented, with a bunch of the test_finalize tests xfailed and a bunch more just wrong. And there is no real prospect of getting these fully working.

If we do decide this is worth keeping, we should have Only One way to do it. _metadata and attrs do effectively the same thing in slightly different ways.

[jorisvandenbossche]

If we do decide this is worth keeping, we should have Only One way to do it. _metadata and attrs do effectively the same thing in slightly different ways.

That is not really true I think. attrs allows users to use this for standard DataFrames, _metadata allows subclasses to use custom metadata that is not directly exposed to users through attrs.

[MarcoGorelli]

And there is no real prospect of getting these fully working.

Personally, this is the argument I find most persuading

I encountered this in the USC contract too, they said they couldn't use attrs as they'd tried to and it was too unreliable (having tried fixing up the finalize tests, I'm not surprised)

One could make the argument that some feature not working completely isn't a reason to deprecate it, but I'm not sure that's valid if the feature isn't being worked on (by contrast, datetime parsing has bugs, but it's actively being worked on, so the prospect of fixing them is realistic).
Given that it's marked as "experimental" anyway, I'd suggest just deprecating/removing it, rather than leaving it hanging around unfinished.

As for users wanting to store metadata - does any other DataFrame library support this? If not, we shouldn't be saying "yes" to everything, especially given how limited maintenance resources are.

As for what users should do - I'd suggest they define their own dataclass where one field is metadata and another is the dataframe, and then take care of how to propagate it themselves

[jorisvandenbossche]

but I'm not sure that's valid if the feature isn't being worked on

To be clear I am not working on this myself, so I don't know the details. But I am not sure that this is true that it is not being worked on: judging by the the activity and linked PRs in #28283, there is some work going on to improve this? (it might have slowed down the last months, but for example generally speaking for the year 2022, quite some PRs have been merged related to this)

I think the bigger problem is that there is no longer an active champion following up on this within the core team

[TomAugspurger]

I can chip away at these as I have free time.

As for users wanting to store metadata - does any other DataFrame library support this? If not, we shouldn't be saying "yes" to everything, especially given how limited maintenance resources are.

xarray does, and I think is a good analog here.

[MarcoGorelli]

I can chip away at these as I have free time.

Awesome!

I think the bigger problem is that there is no longer an active champion following up on this within the core team

Yeah if someone's willing to step up and champion it (like it looks Tom might be doing?) then I have no objections to salvaging this, apologies for having made some too heavy-handed comments earlier on this