SeriesGroupBy.first / last loses categorical dtype

[dsaxton]

On 1.0.3 and master:

import pandas as pd

df = pd.DataFrame({"a": [1, 2, 3]})
df["b"] = df["a"].astype("category")

print(df.groupby("a")["b"].first())
print(df.groupby("a")["b"].last())

gives

a
1    1
2    2
3    3
Name: b, dtype: int64
a
1    1
2    2
3    3
Name: b, dtype: int64

but the dtype should still be categorical and not int64. This seemingly wrong output is explicitly tested for here: https://github.com/pandas-dev/pandas/blob/master/pandas/tests/groupby/aggregate/test_aggregate.py#L461

[simonjayhawkins]

Thanks @dsaxton for the report. for reductions that are elements of the original this makes sense to me too, but IIUC the change in behaviour is for reductions for which the return value may not be in the categories.

for me, this is a regression.

[simonjayhawkins]

for me, this is a regression.

in 0.25.3 this incorrectly returned pandas.core.arrays.categorical.Categorical

>>> import pandas as pd
>>>
>>> pd.__version__
'0.25.3'
>>>
>>> import pandas as pd
>>>
>>> df = pd.DataFrame({"a": [1, 2, 3]})
>>> df["b"] = df["a"].astype("category")
>>>
>>> print(df.groupby("a")["b"].first())
[1, 2, 3]
Categories (3, int64): [1, 2, 3]
>>> print(df.groupby("a")["b"].last())
[1, 2, 3]
Categories (3, int64): [1, 2, 3]
>>>
>>> type(df.groupby("a")["b"].last())
<class 'pandas.core.arrays.categorical.Categorical'>

6d35836 is the first bad commit
commit 6d35836
Author: jbrockmendel jbrockmendel@gmail.com
Date: Thu Oct 17 15:17:13 2019 -0700

Stop catching TypeError in groupby methods (#29060)

after this gave

0.26.0.dev0+595.g6d35836ec
Traceback (most recent call last):
  File "C:/Users/simon/test.py", line 8, in <module>
    print(df.groupby("a")["b"].first())
  File "c:\users\simon\pandas\pandas\core\groupby\groupby.py", line 1342, in f
    return self._cython_agg_general(alias, alt=npfunc, **kwargs)
  File "c:\users\simon\pandas\pandas\core\groupby\groupby.py", line 885, in _cython_agg_general
    result, names = self.grouper.aggregate(obj.values, how, min_count=min_count)
  File "c:\users\simon\pandas\pandas\core\groupby\ops.py", line 608, in aggregate
    "aggregate", values, how, axis, min_count=min_count
  File "c:\users\simon\pandas\pandas\core\groupby\ops.py", line 469, in _cython_operation
    raise NotImplementedError("{} dtype not supported".format(values.dtype))
NotImplementedError: category dtype not supported

[simonjayhawkins]

some overlap with #29746

[Puumanamana]

Not sure if I should create a new issue or not, but I also wanted to point out some inconsistencies regarding this bug. When using agg(), the output dtype changes depending on whether we use a dictionary notation or not (see below):

import pandas as pd # v1.0.4

# Same example data as above
df = pd.DataFrame({"a": [1, 2, 3]})
df["b"] = df["a"].astype("category")

print(df.groupby("a").agg("first").b) # Categorical dtype
print(df.groupby("a").agg({"b": "first"}).b) # int64 dtype

gives

a
1    1
2    2
3    3
Name: b, dtype: category
Categories (3, int64): [1, 2, 3]

a
1    1
2    2
3    3
Name: b, dtype: int64

[jbrockmendel]

Looks like this is fixed on master. could use bisection/test