BUG: DataFrame.groupby drops timedelta column in v1.3.0

[venaturum]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

import pandas as pd
df = pd.DataFrame(
    {
        "duration":[pd.Timedelta(i, unit="days") for i in range(1,6)],
        "value":[1,0,1,2,1]
    }
)

df.groupby("value").sum()

results in:

Empty DataFrame
Columns: []
Index: [0, 1, 2]

The same behaviour occurs whether it's sum, mean, median etc

Note the following gives close to the expected result:

df.groupby("value")["duration"].sum()

value
0   2 days
1   9 days
2   4 days
Name: duration, dtype: timedelta64[ns]

Also, this problem may be unique to Timedeltas as the following similar example, has no problem:

df = pd.DataFrame(
    {
        "duration":[i for i in range(1,6)],
        "value":[1,0,1,2,1]
    }
)

df.groupby("value").sum()

gives:

       duration
value
0             2
1             9
2             4

Problem description

In v1.3.0 (and master, commit dad3e7) when performing a groupby operation on a dataframe a timedelta column goes missing.
This behaviour does not occur in pandas 1.2.5

Expected Output

The expected output is given by pandas 1.2.5:

      duration
value
0       2 days
1       9 days
2       4 days

It is a dataframe, with one column "duration", indexed by the groupby keys

Output of pd.show_versions()

INSTALLED VERSIONS

commit : f00ed8f
python : 3.7.5.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19041
machine : AMD64
processor : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.None

pandas : 1.3.0
numpy : 1.21.0
pytz : 2021.1
dateutil : 2.8.1
pip : 19.2.3
setuptools : 41.2.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[venaturum]

note using

df.groupby("value").sum(numeric_only=False)

gives the desired result

[jmcomie]

take

[jmcomie]

From git bisect, the first commit since v1.2.5 exhibiting a repro is 6b94e24.

commit 6b94e2431536eac4d943e55064d26deaac29c92b
Author: jbrockmendel <jbrockmendel@gmail.com>
Date:   Wed Jun 2 08:14:08 2021 -0700

    BUG: DataFrameGroupBy with numeric_only and empty non-numeric data (#41706)

I'm reviewing that code change and calls below it. My early thought from looking at _reduce and elsewhere is that the numeric_only implementations are a touch more scattered/ad-hoc than they could be. Will keep exploring and update with progress.

[jmcomie]

I'm reviewing the cython -> python aggregation fallback behavior and specifically how 6b94e24 affects our test case. Once I understand the specific error sequence, and details like how numeric_only is shepherded, I should have a good idea of where to pursue a clean fix. If it's just incidental that the cython code can't process these timedelta aggregations, then that strikes me as the place to make a change, rather than restoring the catch-all fallback behavior.

[jmcomie]

Update: The behavior is consistent with the documentation if my understanding is correct, since the pandas.core.groupby.GroupBy.sum numeric_only kwarg defaults to True. https://pandas.pydata.org/docs/reference/api/pandas.core.groupby.GroupBy.sum.html#
The behavior prior to 6b94e24 was not consistent with the spec.

When sum is called with numeric_only False or None, our repro/test-case behaves now as it did before. Datetime and timedelta are expressly classified as non-numeric in the docs and multiple places in the code. See core/internals/blocks.py:class DatetimeLikeBlock and core/dtypes/common.py:def is_numeric_dtype for examples of this non-numeric determination.

Perhaps it's worth contemplating a longer-term change to the API to broaden the numeric_only classification to include datetime and timedelta since they can be operated on arithmetically, but that's beyond the scope of this issue imo.

[jmcomie]

@jreback I defer to you as to whether a code fix should still be pursued here.

[venaturum]

Good work @jmcomie, to add some things to the discussion:

1. df.sum() sums the timedelta column

2. df.groupby("value").apply(pd.DataFrame.sum) sums the timedelta column (but also the groupby column)

3. The results for

   • df.groupby("value").agg("sum")
   • df.groupby("value").sum()
   • df.groupby("value").agg(numpy.sum)
     are all consistent, however the documentation for https://pandas.pydata.org/docs/reference/api/pandas.core.groupby.DataFrameGroupBy.aggregate.html does not mention anything about not aggregating across non-numeric columns. I can pass numeric_only parameter when using agg("sum"), but not agg(numpy.sum), so perhaps documentation needs tweaking.

My 2 cents is that ideally any method X that works with DataFrame.X and df.groupby( ).apply(DataFrame.X) should be consistent with df.groupby( ).X