PERF: regression on mean(axis=1) compared to old pandas version

[Remy-ooo]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this issue exists on the latest version of pandas.

• I have confirmed this issue exists on the main branch of pandas.

Reproducible Example

While working on upgrading pandas version on an old project, I found these perf regressions. mean(axis=1) seems around 30% slower on a dataframe with nans, then if there's a copy(), it becomes almost twice slower. I have tried profiling with no luck.

import pandas as pd
import numpy as np

p = 0.1

df = pd.DataFrame(np.random.rand(2988, 1471))
df.columns = df.columns.astype(str)
df_nan = df.mask(np.random.choice([True, False], size=df.shape, p=[p, 1 - p]))
df_nan_copy = df_nan.copy()

pandas 2.2.2:

In [4]: %timeit df_nan.mean(axis=1)
9.71 ms ± 66.9 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [5]: %timeit df_nan_copy.mean(axis=1)
16 ms ± 174 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)

pandas 0.22.0 (python 3.6):

In [2]: %timeit df_nan.mean(axis=1)
6.68 ms ± 46 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [3]: %timeit df_nan_copy.mean(axis=1)
7.98 ms ± 94.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Installed Versions

INSTALLED VERSIONS ------------------ commit : d9cdd2e python : 3.12.4.final.0 python-bits : 64 OS : Darwin OS-release : 24.1.0 Version : Darwin Kernel Version 24.1.0: Thu Oct 10 21:03:11 PDT 2024; root:xnu-11215.41.3~2/RELEASE_ARM64_T6020 machine : arm64 processor : arm byteorder : little LC_ALL : None LANG : None LOCALE : None.UTF-8

pandas : 2.2.2
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.9.0.post0
setuptools : 69.5.1
pip : 24.3.1
Cython : None
pytest : 7.4.4
hypothesis : None
sphinx : 7.3.7
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 5.2.1
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.4
IPython : 8.25.0
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
bottleneck : 1.3.7
dataframe-api-compat : None
fastparquet : None
fsspec : 2024.3.1
gcsfs : None
matplotlib : 3.8.4
numba : 0.59.1
numexpr : 2.8.7
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : 14.0.2
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : 2024.3.1
scipy : 1.13.1
sqlalchemy : 2.0.30
tables : 3.9.2
tabulate : 0.9.0
xarray : 2023.6.0
xlrd : None
zstandard : 0.22.0
tzdata : 2023.3
qtpy : 2.4.1
pyqt5 : None

Prior Performance

pandas 0.22.0

[rhshadrach]

Thanks for the report! Trying to investigate a performance difference across 7 years of development does not seem likely to yield fruitful results to me. There have been many changes in those years to improve the behavior.

However the difference between the no copy / copy performance is certain worth further investigation.

[auderson]

@rhshadrach
I noticed this performance difference in .sum() as well recently:

import numpy as np
import pandas as pd

sr = pd.Series(np.arange(500000, dtype=float))

sr_nan = sr.copy()
sr_nan[-1] = np.nan

sr_nan_half = sr.copy()
sr_nan_half[: len(sr_nan_half) // 2] = np.nan

sr_nan_all = sr.copy()
sr_nan_all[:] = np.nan

note how np.nansum is getting slower as more nans are in the data:

After some search, I found this post in numpy:

coypto is more optimized for sparse masks where it can really fast scan the subspace and use memcpy on the them while putmask is better for dense masks

numpy/numpy#2494 (comment)

In pandas nanops.py, putmask is used:

pandas/pandas/core/nanops.py

Lines 314 to 321 in d067e08

	if fill_value is not None:
	if mask.any():
	if datetimelike or _na_ok_dtype(dtype):
	values = values.copy()
	np.putmask(values, mask, fill_value)
	else:
	# np.where will promote if needed
	values = np.where(~mask, values, fill_value)

Not sure if np.copyto should be preferred as nans are generally sparse in data.

[snitish]

This seems to have something to do with the row major order. Initially, df row-major but df_nan.copy() seems to be converting it to column-major. This seems to be leading to extra time inside values.copy() in the masking logic in _get_values.
To prevent the change of order, I wrapped the df_nan_copy with a pd.DataFrame() to force it back to row-major order, and the run times are much more comparable now.

import pandas as pd
import numpy as np

p = 0.1
df = pd.DataFrame(np.random.rand(2988, 1471))
df.columns = df.columns.astype(str)
df_nan = df.mask(np.random.choice([True, False], size=df.shape, p=[p, 1 - p]))
df_nan_copy = pd.DataFrame(df_nan.values.copy())

%timeit -n 100 df_nan.mean(axis=1)
# 19.9 ms ± 3.88 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)

%timeit -n 100 df_nan_copy.mean(axis=1)
# 16.5 ms ± 485 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Will investigate why df_nan.copy() is changing the ordering..

[snitish]

Here are the numpy array flags before and after the .copy() call:

> df_nan = df.mask(np.random.choice([True, False], size=df.shape, p=[p, 1 - p]))
> print(df_nan.values.flags)
  C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False

> df_nan_copy = df_nan.copy()
> print(df_nan_copy.values.flags)
  C_CONTIGUOUS : False
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False

[snitish]

Found the issue. Turns out ndarray.copy() which is used by DataFrame.copy() sets the order to column-major by default.
https://numpy.org/doc/2.1/reference/generated/numpy.ndarray.copy.html#numpy.ndarray.copy
One solution is to pass the current order to values.copy() inside Block.copy() so that ordering is maintained. Keen to hear your thoughts on this.

[Remy-ooo]

Just ran some more tests and found that the slowdown happens between 1.4.4 and 1.5.3 for the basic df_nan.mean(axis=1). 1.4.4 takes 9ms and 1.5.3 takes 37ms.
From prun output, I can see that 1.5.3 makes an additional ndarray.copy and the operation went from bottleneck nanmean to numpy reduce.

Finally I was able to get the initial perf with pandas 2.2.3 by doing :

import bottleneck as bn
pd.Series(bn.nanmean(df_nan, axis=1)

I hope this helps, and thank you all for looking into this 😀

[rhshadrach]

One solution is to pass the current order to values.copy() inside Block.copy() so that ordering is maintained. Keen to hear your thoughts on this.

I'd recommend doing a full ASV run with the proposed change. Note this takes some time, ~4 hours.

cc @mroeschke @jorisvandenbossche