BUG: agg on groups with different sizes fails with out of bounds IndexError

[valkum]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

See here as well: https://repl.it/@valkum/WrithingNotablePascal

import numpy as np
import pandas as pd

data = {
  'date': ['2000-01-01', '2000-01-02', '2000-01-01', '2000-01-02'],
  'team': ['client1', 'client1',  'client2', 'client2'],
  'temp': [0.780302, 0.035013, 0.355633, 0.243835],
}
df = pd.DataFrame( data )
df['date'] = pd.to_datetime(df['date'])

df = df.drop(df.index[1])
sampled=df.groupby('team').resample("1D", on='date')
#Returns IndexError
sampled.agg({'temp': np.mean})
#Returns IndexError as well
sampled['temp'].mean()

Problem description

agg fails with IndexError: index 3 is out of bounds for axis 0 with size 3

Note that this does work as expected when I do not drop a row after createing the DataFrame, so I assume it is caused by the index.

Expected Output

No fail.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.8.3.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.0-1009-gcp
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : en_US.UTF-8
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.0.5
numpy : 1.19.0
pytz : 2020.1
dateutil : 2.8.1
pip : 20.1.1
setuptools : 47.3.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : 3.2.2
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pytest : None
pyxlsb : None
s3fs : None
scipy : 1.5.0
sqlalchemy : 1.3.17
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
numba : None

[valkum]

It seems that sampled=df.reset_index().groupby('team').resample("1D", on='date') fixes the issue, but I am not sure if this would still be considered a bug.

[AlexKirko]

@valkum Thanks for the bug report!

This is likely related to #33548. I don't think it has anything to do with group sizes, as this code produces the same out of bounds error:

import numpy as np
import pandas as pd

data = {
  'date': ['2000-01-01','2000-01-01', '2000-01-02', '2000-01-01', '2000-01-02'],
  'team': ['client1', 'client1', 'client1',  'client2', 'client2'],
  'temp': [0.780302, 0.780302, 0.035013, 0.355633, 0.243835],
}
df = pd.DataFrame( data )
df['date'] = pd.to_datetime(df['date'])
df = df.drop(df.index[1])

sampled=df.groupby('team').resample("1D", on='date')

#Returns IndexError
sampled.agg({'temp': np.mean})
#Returns IndexError as well
sampled['temp'].mean()

Also sampled.mean() works, it's only sampled['temp'].mean() that breaks.

Seeing as reset_index fixes it, maybe the break in the index causes the bug.

[valkum]

Thanks for your reply.

sampled.agg(np.mean) works too, only when you try to a pass a dict (to only cover specific columns) it breaks.
Furthermore your example does work for me with out an out of bounds error, but creates different results nevertheless. See here

Its only when you drop a row after the DataFrame is created, and as you pointed out, the Index is not continous anymore.
So it is somehow a bug caused by non-continous indices combined with selecting an aggregation function on specific columns (either bei sampled['temp'].mean() or sampled.agg({'temp': np.mean}))

But I see that it might be related to #33548

[AlexKirko]

Interesting. For me my code breaks both on 1.0.5 and on the latest commit of master.

UPDATE: ah, forgot to drop the second row. @valkum , could you run the updated code to make sure that it breaks, and that we aren't dealing with something super-weird?

[AlexKirko]

Investigated this a bit. The object we end up with is of class pandas.core.resample.DatetimeIndexResamplerGroupby, which is a non-transparent descendant of GroupByMixin and DatetimeIndexResampler , and uncovering what exactly is causing bugs when using aggregate functions is non-trivial.

I'll try to track down this bug next week.

[AlexKirko]

take

[AlexKirko]

Interesting. The bug can be "fixed" by using a deep copy in _apply in _GroupByMixin. We must be forgetting something when creating a shallow copy, which causes _set_grouper to crash. Will keep investigating.

[AlexKirko]

Okay, so what happens is that df.index values get used deep down the call stack to draw dates from the DatetimeIndex that the grouping and resampling operations create. This is done through Index.take, and because the DatetimeIndex has only four elements in it, and we are trying to get the element with index 4, we get a KeyError. This is why resetting the index fixes this.

The whole process is necessary, because we apply aggregation functions by creating shallow copies of Series objects and applying the functions to them.

Here is a link to the relevant code.

As far as I can tell, we don't need to preserve the original row index before applying aggregation functions to a DatetimeIndexResamplerGroupby, so the obvious way would be to reset the index somewhere down the call stack to be safe. I'll see if I can find a good candidate spot.