BUG: groupby.shift return keys filled with `fill_value` when `fill_value` is specified

[isVoid]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

In [2]: df = pd.DataFrame({'a': [2, 1, 2, 1], 'b': ['x', 'x', 'y', 'y']})

In [3]: df.groupby('a').shift(1)
Out[3]: 
     b
0  NaN
1  NaN
2    x
3    x

In [4]: df.groupby('a').shift(1, fill_value='fill')
Out[4]: 
      a     b
0  fill  fill
1  fill  fill
2     2     x
3     1     x

Problem description

When specifying fill_value in groupby.shift, the returned result includes the key column with keys filled with fill_value. When fill_value is unspecified (None), the key column is not included.

Expected Output

It seems pretty strange that keys are to be filled with fill_value. This makes more sense to me:

In [4]: df.groupby('a').shift(1, fill_value='fill')
Out[4]: 
      b
0  fill
1  fill
2    x
3    x

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 2cb9652
python : 3.7.10.final.0
python-bits : 64
OS : Linux
OS-release : 4.15.0-76-generic
Version : #86-Ubuntu SMP Fri Jan 17 17:24:28 UTC 2020
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : None
LOCALE : en_US.UTF-8

pandas : 1.2.4
numpy : 1.20.2
pytz : 2021.1
dateutil : 2.8.1
pip : 21.1.1
setuptools : 52.0.0.post20210125
Cython : 0.29.23
pytest : 6.2.4
hypothesis : 6.12.0
sphinx : 3.5.4
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.3
IPython : 7.23.1
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : 2021.04.0
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 1.0.1
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : 0.53.1

[mzeitlin11]

Thanks for the report @isVoid! Can confirm the issue persists on master, investigations to fix are welcome! At a brief look, looks to be because fill_value is not supported in the cython groupby shift algo, so a different path is taken.

[isVoid]

Another related corner case (slightly different behavior)

>>> df = pd.DataFrame({'a': [2, 1, 2, 1], 'b': ['x', 'x', 'y', 'y']})
>>> df.groupby('a').shift(0, fill_value=42)
     a  b
a        
1 1  1  x
  3  1  y
2 0  2  x
  2  2  y

When periods=0, the return index includes redundant information.

Expected behavior:

>>> df.groupby('a').shift(0, fill_value=42)
     b
0    x
1    x
2    y
3    y

[smithto1]

take