BUG: Series attrs disappear after modifcation to the DataFrame that contains them.

[lendres]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

# Example 1 - changing the data type.
df = pd.DataFrame(dict(column1=[1,2,3,4], column2=[1,2,3,4]))
df["column1"].attrs['name'] = 'hello'

print("\nExample 1 before:")
print(df)
print("Column1 attributes:", df["column1"].attrs)

# Changing the data type of the DataFrame drops the series attributes.
df1 = df.astype(float)
print("\nAfter type change:")
print(df1)
print("Column1 attributes:", df1["column1"].attrs)



# Example 2 - assignment to a DataFrame.
df = pd.DataFrame(dict(column1=[1,2,3,4], column2=[1,2,3,4]))
df["column1"].attrs['name'] = 'hello1'
df["column2"].attrs['name'] = 'hello2'

print("\n\n\nExample 2 before:")
print(df)
print("Column1 attributes:", df["column1"].attrs)
print("Column2 attributes:", df["column2"].attrs)

# Assigning a column to the DataFrame drops the attributes from the Series in DataFrame.
df["column1"] = df["column1"] + 5
print("\nAfter assignment to DataFrame:")
print(df)
print("Column1 attributes:", df["column1"].attrs)
print("Column2 attributes:", df["column2"].attrs)

Issue Description

Issue

The attrs dictionary of Series are not carried through when operations are performed on a DataFrame.

Example 1

# Example 1 - changing the data type.
df = pd.DataFrame(dict(column1=[1,2,3,4], column2=[1,2,3,4]))
df["column1"].attrs['name'] = 'hello'

print("\nExample 1 before:")
print(df)
print("Column1 attributes:", df["column1"].attrs)

"""
OUTUPUT:
Example 1 before:
   column1  column2
0        1        1
1        2        2
2        3        3
3        4        4
Column1 attributes: {'name': 'hello'}
"""

The first series has an attr stored. However, after changing the data types, the attribute is lost.

# Changing the data type of the DataFrame drops the series attributes.
df1 = df.astype(float)
print("\nAfter type change:")
print(df1)
print("Column1 attributes:", df1["column1"].attrs)

"""
OUTUPUT:
After type change:
   column1  column2
0      1.0      1.0
1      2.0      2.0
2      3.0      3.0
3      4.0      4.0
Column1 attributes: {}         <-- NO ATTRS
"""

Example 2

# Example 2 - assignment to a DataFrame.
df = pd.DataFrame(dict(column1=[1,2,3,4], column2=[1,2,3,4]))
df["column1"].attrs['name'] = 'hello1'
df["column2"].attrs['name'] = 'hello2'

print("\n\n\nExample 2 before:")
print(df)
print("Column1 attributes:", df["column1"].attrs)
print("Column2 attributes:", df["column2"].attrs)

"""
OUTPUT:
Example 2 before:
   column1  column2
0        1        1
1        2        2
2        3        3
3        4        4
Column1 attributes: {'name': 'hello1'}
Column2 attributes: {'name': 'hello2'}
"""

The first series has an attr stored. However, reassigning a column drops the attributes.

# Assigning a column to the DataFrame drops the attributes from the Series in DataFrame.
df["column1"] = df["column1"] + 5
print("\nAfter assignment to DataFrame:")
print(df)
print("Column1 attributes:", df["column1"].attrs)
print("Column2 attributes:", df["column2"].attrs)

"""
OUTPUT:
After assignment to DataFrame:
   column1  column2
0        6        1
1        7        2
2        8        3
3        9        4
Column1 attributes: {}        <-- NO ATTRS
Column2 attributes: {}        <-- NO ATTRS
"""

Expected Behavior

The attributes (attrs) should be maintained in the DataFrame when operations are performed on it.

Tested on 2.1.1 and 2.2.0

Installed Versions

INSTALLED VERSIONS ------------------ commit : f538741 python : 3.11.4.final.0 python-bits : 64 OS : Windows OS-release : 10 Version : 10.0.19044 machine : AMD64 processor : Intel64 Family 6 Model 165 Stepping 2, GenuineIntel byteorder : little LC_ALL : None LANG : en LOCALE : English_United States.1252

pandas : 2.2.0
numpy : 1.26.3
pytz : 2024.1
dateutil : 2.8.2
setuptools : 68.0.0
pip : 23.2.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.3
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.15.0
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.2
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.7.1
numba : None
numexpr : 2.8.4
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : None
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.11.1
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.4
qtpy : None
pyqt5 : None

[rhshadrach]

Thanks for the report; attr propagation is only partially implemented currently. PRs to fix are welcome!

[dontgoto]

take

[dontgoto]

On main this type of Series.attr propagation is basically nonexistent. Practically, only DataFrame.attrs are accessible and get attached to the Series when the Series representation of a column is retrieved.

I would try to implement a solution that introduces hidden column_attrs to the DataFrame that would support this type of Series.attrs propagation. Solutions that only make use of the DataFrame.attrs by design either lack propagation functionality or lead to confusing behaviour.