Performance regression in 0.24+ on GroupBy.apply

[blu3r4y]

Code Sample

import time

import numpy as np
import pandas as pd

nrows, ncols = 1000, 100

# data frame with random values and a key to be grouped by
df = pd.DataFrame(np.random.rand(nrows, ncols))
df["key"] = range(nrows)

numeric_columns = list(range(ncols))

grouping = df.groupby(by="key")

# performance regression in apply()
start = time.time()
grouping[numeric_columns].apply(lambda x: x - x.mean())
end = time.time()

print("[pandas=={}] execution time: {:.4f} seconds".format(pd.__version__, end - start))

# [pandas==0.23.4] execution time: 0.8700 seconds
# [pandas==0.24.0] execution time: 24.3790 seconds
# [pandas==0.24.2] execution time: 23.9600 seconds

Problem description

The function GroupBy.apply is a lot slower (~ 25 times) with version 0.24.0, compared to the 0.23.4 release. The problem still persists in the latest 0.24.2 release.

The code sample above shows this performance regression.
The purpose of the sample is to subtract the group mean from all elements in this group.

The problem does only occur when the lambda for apply() returns a data frame.
There are no performance issues with scalar return values, e.g. lambda x: x.mean().

Output of pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.6.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 158 Stepping 9, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.24.0
pytest: None
pip: 10.0.1
setuptools: 39.2.0
Cython: None
numpy: 1.16.2
scipy: 1.2.1
pyarrow: None
xarray: None
IPython: 6.4.0
sphinx: None
patsy: 0.5.0
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 3.0.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: None
bs4: None
html5lib: 0.9999999
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

[WillAyd]

Can you isolate the frame operation from the groupby? Curious if the regression is noticeable in the former

[blu3r4y]

You mean like so? Well, there is a slight difference (~ 30 % slower) but I wouldn't trust my "benchmark" here too much. I ran the code a few times and the numbers varied between 0.5 and 1 second.

import time

import numpy as np
import pandas as pd

nrows, ncols = 1000, 1000

df = pd.DataFrame(np.random.rand(nrows, ncols))

start = time.time()
for _ in range(100):
    df.apply(lambda x: x - x.mean())
end = time.time()

print("[pandas=={}] execution time: {:.4f} seconds".format(pd.__version__, end - start))
# [pandas==0.23.4] execution time: 25.8880 seconds
# [pandas==0.24.0] execution time: 36.0216 seconds
# [pandas==0.24.2] execution time: 34.6180 seconds

Additionally, i tested the original code sample with having only one group. It's still about 2 times slower in 0.24+ compared to 0.23.4, but not as drastic as with multiple groups.

nrows, ncols = 10000, 10000

df["key"] = [1] * nrows

# [pandas==0.23.4] execution time: 5.5250 seconds
# [pandas==0.24.0] execution time: 12.1590 seconds
# [pandas==0.24.2] execution time: 12.1540 seconds

[WillAyd]

Right I'm just trying to isolate potential regressions in GroupBy versus Frame operations. Given you don't see the same regression with scalars I'm inclined to believe it's the latter that may be at fault here.

Can you try your last example on master? I think those results might be misleading as the apply operation would still get called twice even with only one group in 0.24.2 (see #24748 which just changed this behavior) so might not be a clean comparison to make

@TomAugspurger we were never able to get the ASV site back up running were we?

[blu3r4y]

I guess I got closer to the problem, it really seems to be related to data frame operations, i.e. subtracting a scalar from a data frame already seems to be the regression. And it's huge, on column-heavy shapes, this simple operation is 153 times slower ô.O

import timeit
import numpy as np  # 1.16.2
import pandas as pd

def benchmark():
    nrows, ncols = 100, 100
    df = pd.DataFrame(np.random.rand(nrows, ncols))
    _ = df - 1

time = timeit.timeit(benchmark, number=100)
print("# {:>8.4f} sec   pandas=={}".format(time, pd.__version__))

Here are my benchmarking results of the df - 1 operation with different shapes compared to master.

shape	0.23.4	3855a27	factor
100, 100	0.0343	2.3835	x69
10000, 100	1.5811	5.0893	x3
100, 10000	1.4902	229.1662	x153
1000, 1000	1.5120	26.9160	x18

---

Concerning GroupBy.apply, there seems to be just a slight performance regression of up to 30 % slower performance, as indicated in my earlier comment.

import timeit
import numpy as np  # 1.16.2
import pandas as pd

def benchmark():
    nrows, ncols = 1000, 100
    df = pd.DataFrame(np.random.rand(nrows, ncols))
    df["key"] = range(nrows)

    numeric_columns = list(range(ncols))
    
    grouping = df.groupby(by="key")
    grouping[numeric_columns].apply(lambda x: x.mean())

time = timeit.timeit(benchmark, number=10)
print("# {:>8.4f} sec   pandas=={}".format(time, pd.__version__))

shape	0.23.4	3855a27	factor
100, 100	0.2715	0.3211	x1.18
1000, 100	19.4875	26.0909	x1.34
100, 1000	0.6296	0.6103	x0.96
1000, 1000	5.0933	5.6909	x1.11

[chris-b1]

Note that if this is your actual function, you can/should instead do this, which has always been faster

df[numeric_columns] - grouping[numeric_columns].transform('mean')

[TomAugspurger]

Can you check for duplicates? We have another issue for ops that were previously blockwise, but are now columnwise.

[WillAyd]

Local ASV result comparing current head to the last commit on 0.23.4 confirms a regression for the frame ops:

       before           after         ratio
     [af7b0ba4]       [95c78d65]
                      <master>
+      3.30±0.2ms          249±3ms    75.37  binary_ops.Ops2.time_frame_float_div_by_zero
+      8.08±0.4ms         256±10ms    31.75  binary_ops.Ops2.time_frame_int_div_by_zero
+      12.6±0.2ms          261±7ms    20.78  binary_ops.Ops2.time_frame_float_floor_by_zero
+      30.2±0.4ms          106±1ms     3.52  binary_ops.Ops.time_frame_multi_and(False, 1)
+      29.9±0.7ms          102±4ms     3.43  binary_ops.Ops.time_frame_multi_and(False, 'default')
+      34.1±0.4ms          110±2ms     3.24  binary_ops.Ops.time_frame_multi_and(True, 1)
+      39.2±0.4ms        111±0.9ms     2.83  binary_ops.Ops.time_frame_multi_and(True, 'default')
+     3.85±0.06ms       5.18±0.1ms     1.35  binary_ops.Ops.time_frame_add(True, 1)
+      28.6±0.1μs       37.4±0.2μs     1.31  binary_ops.Ops2.time_series_dot
+     3.54±0.06ms       4.29±0.3ms     1.21  binary_ops.Ops.time_frame_add(False, 1)
+         512±1μs          600±3μs     1.17  binary_ops.Ops2.time_frame_series_dot
-         107±2ms         61.2±1ms     0.57  binary_ops.Ops.time_frame_comparison(False, 'default')
-         108±1ms       61.1±0.9ms     0.57  binary_ops.Ops.time_frame_comparison(False, 1)

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.

I've updated the title to reflect this as I think that is the larger issue. @blu3r4y if you can run ASVs for GroupBy apply to confirm regression there that could be helpful as another issue

[blu3r4y]

Can you check for duplicates? We have another issue for ops that were previously blockwise, but are now columnwise.

Yes, regression with DataFrame + Scalar Ops in 0.24+ have already been reported in #24990.

@WillAyd I will run ASV for GroupBy.apply soon, so that we keep this issue isolated on GroupBy.apply, right? Or should I make a new issue then?