BUG: Up-Resample data with PeriodIndex has unexpected behavior

[garylavayou]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

---

Code Sample, a copy-pastable example

import pandas as pd
# Example-1: 
s = pd.Series([1, 2], index=pd.period_range('2012-01-01',
                                            freq='A',
                                            periods=2))
print(s)         
# 2012    1
# 2013    2
# Freq: A-DEC, dtype: int64
                               
print(s.resample('Q', kind='period', convention='start').count())
# 2012Q1    1.0
# 2012Q2    NaN
# 2012Q3    NaN
# 2012Q4    NaN
# 2013Q1    2.0
# 2013Q2    NaN
# 2013Q3    NaN
# 2013Q4    NaN

# Example-2: the result is all NaN, but the data is actually grouped
idx = pd.period_range(start='1/10/2000', periods=2, freq='D')
series = pd.Series(range(2,4), index=idx)
print(series.resample('12H',  convention='end').count())
# 2000-01-10 12:00   NaN
# 2000-01-11 00:00   NaN
# 2000-01-11 12:00   NaN
print(series.resample('12H',  convention='start').count())
# 2000-01-10 00:00    2.0
# 2000-01-10 12:00    NaN
# 2000-01-11 00:00    3.0
# 2000-01-11 12:00    NaN
# Freq: 12H, dtype: float64

for idx, ss in rs: 
    if ss.empty == False: print(ss)
# 2000-01-10    2
# Freq: D, dtype: int64
# 2000-01-11    3
# Freq: D, dtype: int64

Problem description

Try to up-resample a time series with PeriodIndex as the index of the series, and then count the number of records in each time-group. However, the output is not the "count", but it seems to be the only record's value in the group.

In the second example with convention='end', the result is all NaN. If switches to 'convention='start'', then the result is similar to that of the Example-1. In addition, from the grouped details, we can see that the data has been correctly grouped, while the aggregate functions (e.g., count) behave unexpected.

Expected Output

The number of samples for each group (call the count() method).

As a comparison of Example-1, I make another example, which tries to up-sample a time series with Timestamp as index. It behaves as expected.

s = pd.Series([1, 2], index=pd.date_range('2012-01-01',
                                            freq='A',
                                            periods=2))
display(s)                                        
# 2012-12-31    1
# 2013-12-31    2
# Freq: A-DEC, dtype: int64

display(s.resample('Q', convention='start').count())
# 2012-12-31    1
# 2013-03-31    0
# 2013-06-30    0
# 2013-09-30    0
# 2013-12-31    1

Output of pd.show_versions()

INSTALLED VERSIONS
------------------
commit           : c7f7443c1bad8262358114d5e88cd9c8a308e8aa
python           : 3.7.10.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
Version          : 10.0.19041
machine          : AMD64
processor        : Intel64 Family 6 Model 85 Stepping 4, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : None.None

pandas           : 1.3.1
numpy            : 1.21.1
pytz             : 2021.1
dateutil         : 2.8.2
pip              : 21.2.1
setuptools       : 49.6.0.post20210108
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 3.0.1
IPython          : 7.25.0
pandas_datareader: None
bs4              : None
bottleneck       : 1.3.2
fsspec           : None
fastparquet      : None
gcsfs            : None
matplotlib       : 3.4.2
numexpr          : 2.7.3
odfpy            : None
openpyxl         : 3.0.7
pandas_gbq       : None
pyarrow          : 4.0.1
pyxlsb           : None
s3fs             : None
scipy            : 1.7.0
sqlalchemy       : 1.4.22
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
numba            : None

[dicristina]

Upsampling with PeriodIndex is treated as an asfreq for aggregating methods, except for ohlc. Maybe size and count should be handled as ohlc.

pandas/pandas/core/resample.py

Lines 1288 to 1301 in 860ff03

	if is_subperiod(ax.freq, self.freq):
	# Downsampling
	return self._groupby_and_aggregate(how, grouper=self.grouper, **kwargs)
	elif is_superperiod(ax.freq, self.freq):
	if how == "ohlc":
	# GH #13083
	# upsampling to subperiods is handled as an asfreq, which works
	# for pure aggregating/reducing methods
	# OHLC reduces along the time dimension, but creates multiple
	# values for each period -> handle by _groupby_and_aggregate()
	return self._groupby_and_aggregate(how, grouper=self.grouper)
	return self.asfreq()
	elif ax.freq == self.freq:
	return self.asfreq()

---

The all NaN case is due to a reindexing operation in which there are no values in common.

pandas/pandas/core/resample.py

Lines 1330 to 1336 in 860ff03

	new_index = self.binner
	# Start vs. end of period
	memb = ax.asfreq(self.freq, how=self.convention)
	# Get the fill indexer
	indexer = memb.get_indexer(new_index, method=method, limit=limit)

We can reproduce what happens in the previous code by doing the following:

import pandas as pd
# Like example 2 above
idx = pd.period_range(start='1/10/2000', periods=2, freq='D')
series = pd.Series(range(2,4), index=idx)

offset, conv = "12H", "end"
resampler = series.resample(offset, convention=conv)
assert resampler.ax is series.index
memb = resampler.ax.asfreq(offset, conv) 
memb.get_indexer(resampler.binner)

The last line produces array([-1, -1, -1]), which means that there were no values in common between the the original index (memb) and the new index (resampler.binner). This happens because resampler.ax.asfreq does not take into consideration the multiple of the offset, meaning that memb always is

PeriodIndex(['2000-01-10 23:00', '2000-01-11 23:00'], dtype='period[H]')

for any offset corresponding to Hour, i.e. "H", "2H", "12H", etc.

[garylavayou]

@dicristina
Why not adopt the same procedure as down-sampling for aggregation functions like 'max', 'min', 'count', 'mean', etc.

# Downsampling 
     return self._groupby_and_aggregate(how, grouper=self.grouper, **kwargs) 

So, the behavior is consistent for down/up-sampling.

Intuitively, the up-sampling operation results in some empty groups, and the aggregate functions applied to empty groups might return NaN or 0 (like count and size).

[dicristina]

That is the way to go but there are some tests that must be dealt with. As a workaround you can use Resampler.agg with a dict. For instance your first example could be:

s.resample('Q', kind='period', convention='start').agg({"count": "count"})["count"]

This gives the expected result because _groupby_and_aggregate gets called.