BUG: Subclassed DataFrame doesn't persist _metadata properties across binary operations

[clausmith]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

When subclassing a DataFrame, fields added to the _metadata property are only persisted across some operations (such as slicing) and not others (such as any arithmetic operation).

I would expect any properties defined on the subclass to persist whenever the result of an operation is an instance of the subclass.

The following is the example taken from the "Extending Pandas" docs: https://pandas.pydata.org/pandas-docs/stable/development/extending.html

import pandas as pd
class SubclassedDataFrame2(pd.DataFrame): 
 
    # temporary properties 
    _internal_names = pd.DataFrame._internal_names + ['internal_cache'] 
    _internal_names_set = set(_internal_names) 
 
    # normal properties 
    _metadata = ['added_property'] 
 
    @property 
    def _constructor(self): 
        return SubclassedDataFrame2 

df = SubclassedDataFrame2({'A': [1, 2, 3], 'B': [4, 5, 6], 'C': [7, 8, 9]})
df.internal_cache = "cached"
df.added_property = "property"

With the above setup, here's how to reproduce the problem:

>>> df.added_property
'property'

>>> df[["A", "B"]].added_property # this works as expected
'property'

>>> (df * 2).added_property # I would expect this to work
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/clausmith/Developer/pandas-bug/pandas/pandas/core/generic.py", line 5220, in __getattr__
    return object.__getattribute__(self, name)
AttributeError: 'SubclassedDataFrame2' object has no attribute 'added_property'

Problem description

The current behavior means that you can almost never rely on custom properties to persist on a subclassed DataFrame. This substantially reduces the utility of these custom properties.

Expected Output

I would expect the added_property property in the example above to persist after performing the arithmetic operation on the DataFrame. Especially because the result of (df * 2) is still an instance of SubclassedDataFrame2.

Output of pd.show_versions()

commit           : 507cb1548d36bbf48c3084a78d59af2fed78a9d1
python           : 3.7.3.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 18.7.0
Version          : Darwin Kernel Version 18.7.0: Mon Feb 10 21:08:45 PST 2020; root:xnu-4903.278.28~1/RELEASE_X86_64
machine          : x86_64
processor        : i386
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.1.0.dev0+1576.g507cb1548
numpy            : 1.18.4
pytz             : 2020.1
dateutil         : 2.8.1
pip              : 19.0.3
setuptools       : 40.8.0
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
pyxlsb           : None
s3fs             : None
scipy            : None
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
numba            : None

[TomAugspurger]

DataFrame binary ops (like mul) don't currently call NDFrame.__finalize__. We have an xfailing test for this at

pandas/pandas/tests/generic/test_finalize.py

Lines 127 to 135 in 085af07

	pytest.param(
	(
	pd.DataFrame,
	frame_data,
	operator.methodcaller("add", pd.DataFrame(*frame_data)),
	),
	marks=not_implemented_mark,
	),
	# TODO: div, mul, etc.

.

I don't think there would be any objection to calling finalize here. The primary API question is what to do with metadata / attrs when other is a Series or DataFrame. But for scalars there's no issue.

@clausmith are you interested in working on this?

[TomAugspurger]

xref #28283 for the general issue. This can be specific to binops.

[clausmith]

@TomAugspurger ah that's what I thought. I saw #28283 and figured this might be related. Unfortunately I'm a little out of my depth here (I'm a PM more so than an engineer 😬).

[jbesomi]

Hi @TomAugspurger, I'm working on a side project with Pandas and need this problem to be fixed. I would love to help you. Can you please explain me how I should proceed to first understand the issue and then fix it? Thank you.

[TomAugspurger]

Great, thanks. I would start with the `Series.op(Series)` case, and then work up to Series & Frame and Frame & Frame. Series._binop is probably the place this should go. We'll need a call to `NDFrame.__finalize__` somewhere in https://github.com/pandas-dev/pandas/blob/42a5c1c1aac401735a9e06e21fece93f58a4b4ec/pandas/core/series.py#L2593-L2627. Perhaps it could be done as part of `_construct_result`, not sure. As I mentioned earlier, it's not 100% clear how we'll propagate metadata / `.attrs` when the values differ. I think just ignore this for now. We can define and document the behavior we want once we figure that out.

…

On Fri, May 22, 2020 at 2:16 AM Jonathan Besomi ***@***.***> wrote: Hi @TomAugspurger <https://github.com/TomAugspurger>, I'm working on a side project with Pandas and need this problem to be fixed. I would love to help you. Can you please explain me how I should proceed to first understand the issue and then fix it? Thank you. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#34177 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKAOIUNP6AMSIE57VZGHP3RSYREFANCNFSM4NA5GKRA> .

[jbesomi]

Great, thank you. Will start from Series._binop then.

"As I mentioned earlier, it's not 100% clear how we'll propagate metadata / .attrs when the values differ": what do you mean with values differ?

A related yet-different problem is to assign to a new or existing column of a pandas Dataframe a pandas Series containing metadata. Also in this case the metadata information is lost. Do you think it will be possible to solve this problem too or there might be some other reasons why we should keep things as they are?