BUG: Unexpected behavior of df.rolling(indexer).mean()

[philippesamuel]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
from pandas.api.indexers import BaseIndexer

df = pd.DataFrame({
        'A': [10.0, 12.0, 14.0],
        'B': [20.0, 22.0, 24.0],
        'C': [30.0, 32.0, 34.0],
        'D': [40.0, 42.0, 44.0]
    })

expected = pd.DataFrame({
        'A': [ 5.0, 11.0, 12.0],
        'B': [10.0, 21.0, 22.0],
        'C': [15.0, 31.0, 32.0],
        'D': [20.0, 41.0, 42.0]
    })


class VariableWindowSizeIndexer(BaseIndexer):

    def __init__(self, window_sizes: Sequence[int], **kwargs) -> None:
        super(VariableWindowSizeIndexer, self).__init__(**kwargs)
        self.window_sizes = np.array(window_sizes, dtype='int64')

    def get_window_bounds(
            self,
            num_values: int = 0,
            min_periods: int | None = 1,
            center: bool | None = None,
            closed: str | None = None,
            step: int | None = None,
    ) -> tuple[np.ndarray, np.ndarray]:
        end = np.arange(len(self.window_sizes)) + 1
        start = end - self.window_sizes
        return start.astype('int64'), end.astype('int64')


window_sizes = [2, 2, 3]
indexer = VariableWindowSizeIndexer(window_sizes)
result = df.rolling(indexer).mean()
pd.testing.assert_frame_equal(result, expected)

Issue Description

result
A B C D
0 5.0 17.0 27.0 37.0
1 11.0 21.0 31.0 41.0
2 12.0 22.0 32.0 42.0

Expected Behavior

expected = pd.DataFrame({
'A': [ 5.0, 11.0, 12.0],
'B': [10.0, 21.0, 22.0],
'C': [15.0, 31.0, 32.0],
'D': [20.0, 41.0, 42.0]
})

Installed Versions

INSTALLED VERSIONS

commit : 2e218d1
python : 3.11.2.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.22000
machine : AMD64
processor : Intel64 Family 6 Model 142 Stepping 12, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252
pandas : 1.5.3
numpy : 1.24.3
pytz : 2023.3
dateutil : 2.8.2
setuptools : 67.8.0
pip : 23.0.1
Cython : None
pytest : 7.3.1
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.10.1
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None
tzdata : None

[jbrockmendel]

pls give the issue a helpful title

[stevenschaerer]

@philippesamuel: I don't know if this is still useful to you, but your indexer returns start values of -1, which causes the rolling calculation to pick up values outside of the column that is being processed (e.g., in col B, 17 = mean(20, 14), where 14 is the last value of col A). You also seem to expect that, given a window size of 2 and only one data point, the mean is calculated between that data point and 0. That is not how the rolling computations work. They just consider the data points that are available. You could pad your input with an all 0 row for this behavior.

[philippesamuel]

Hi @stevenschaerer, thank you for your response.
I managed to solve the problem by making sure that my indexer explicitly handles -1 indices. Thank you for pointing out how the rolling operation treats this case. I hadn't noticed that the last column is used in this case.

I still think there are some inconsistencies with the behavior of the rolling operation here. See for instance the first element of column A. Here, since there's no previous column, the mean of (10, 0) was calculated. I'd expect that all columns are treated the same.

[stevenschaerer]

I wouldn't call it inconsistent but rather undefined behavior. I maybe gave the wrong impression, but the "fact" that accessing the -1 index of the second columns values gives you the last value of the first column is not something that can or should be relied on as it depends on how the values are stored in memory. Neither is the "fact" that 0 is returned for the -1 index in the case of the first column.

The underlying Cython function that does the calculation would typically not even be expected to behave this way, but for performance reasons negative indexing support is deactivated. Otherwise, you'd get 12, 22, 32 and 42 in the first results row, because the mean would be calculated between the first (index 0) and last (index -1) elements of each column. And that's actually also what df.rolling(indexer).mean(engine='numba') returns, which uses a different implementation that does support negative indexing.

@jbrockmendel: Would it be worth it to add a check in _check_window_bounds in rolling.py that min(start), min(end) >= 0 and max(start), max(end) < num_values? Or even that start[i] < end[i] for all i? This would slow things down for non-custom indexer as well, though.

[jbrockmendel]

Seems reasonable, but I'll defer to @mroeschke on this