Description
Pandas version checks
-
I have checked that this issue has not already been reported.
-
I have confirmed this issue exists on the latest version of pandas.
-
I have confirmed this issue exists on the main branch of pandas.
Reproducible Example
Pandas Memory duplication
Hello!
First of all Thanks for this amazing work and library!
We spotted a major memory duplication when dropping a column from a Pandas dataframe.
(we tested and noticed the issue under pandas 1.4.3, 1.4.4, 1.5.0)
here is the code example to reproduce the problem
import numpy as np
import pandas as pd
from memory_profiler import profile
@profile
def pouet():
df = pd.DataFrame(np.random.randint(0,1000000,size=(1000000, 6)), columns=list('ABCDEF'))
df.drop(columns=['A'], inplace=True)
if __name__ == '__main__':
pouet()After running the script you should expect a similar output
Filename: /tmp/toto.py
Line # Mem usage Increment Occurrences Line Contents
=============================================================
6 54.1 MiB 54.1 MiB 1 @profile
7 def pouet():
8 100.0 MiB 46.0 MiB 1 df = pd.DataFrame(np.random.randint(0,1000000,size=(1000000, 6)), columns=list('ABCDEF'))
9 138.4 MiB 38.4 MiB 1 df.drop(columns=['A'], inplace=True)
As you can see, after the drop, the memory went from 148MB to 186MB, which corresponds to the new data frame size (here 38MB).
After some research in the core/internals sources, we noticed that the duplication occurs in reindex_indexer when calling the _slice_take_blocks_ax0 function on line 739 , which is in core/internals/managers.py.
Memory duplication also occurs when dropping a column that is not on the 0 axis.
Forcing the only_slice parameter of the _slice_take_blocks_ax0 function to True seems to solve the duplication problem when dropping a column on axis 0.
here is an example
if axis == 0:
new_blocks = self._slice_take_blocks_ax0(
indexer,
fill_value=fill_value,
only_slice=True, #<======= HERE
use_na_proxy=use_na_proxy,
)by running the same script again, you should expect similar output
Filename: /tmp/toto.py
Line # Mem usage Increment Occurrences Line Contents
=============================================================
6 51.4 MiB 51.4 MiB 1 @profile
7 def pouet():
8 97.4 MiB 46.0 MiB 1 df = pd.DataFrame(np.random.randint(0,1000000,size=(1000000, 6)), columns=list('ABCDEF'))
9 97.6 MiB 0.2 MiB 1 df.drop(columns=['A'], inplace=True)
The data framework has been updated and the memory usage is still stable and acceptable.
For small data frames this is not a big problem, but in our use cases we are dealing with "big" data frames (about 8gb to 10-20gb), which can lead to random OOM kills (out of memory) when dropping columns on these data frames.
Are you aware of this problem, and have you planned a solution? If not, we are willing to contribute and solve the problem if you can provide us with more context and use cases related to the Drop function, to make sure we don't break features using the same functions.
The doc should also inform the user about the behavior of the drop function, the inplace parameters could lead to confusion, indeed the df is updated but it is also duplicated and the operations are done on the duplicated one, and inform the user about this memory duplication problem.
Installed Versions
INSTALLED VERSIONS
commit : 87cfe4e
python : 3.10.6.final.0
python-bits : 64
OS : Darwin
OS-release : 21.6.0
Version : Darwin Kernel Version 21.6.0: Wed Aug 10 14:25:27 PDT 2022; root:xnu-8020.141.5~2/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.UTF-8
pandas : 1.5.0
numpy : 1.23.3
pytz : 2022.2.1
dateutil : 2.8.2
setuptools : 63.2.0
pip : 22.2.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None
tzdata : None
Prior Performance
No response