BUG: DataFrameGroupBy.value_counts fails with a TimeGrouper

[rhshadrach]

df = pd.DataFrame(
    {
        "Timestamp": [
            1565083561,
            1565083561 + 86400,
            1565083561 + 86500,
            1565083561 + 86400 * 2,
            1565083561 + 86400 * 3,
            1565083561 + 86500 * 3,
            1565083561 + 86400 * 4,
        ],
        "Food": ["apple", "apple", "banana", "banana", "orange", "orange", "pear"],
    }
).drop([3])

df["Datetime"] = to_datetime(df["Timestamp"].apply(lambda t: str(t)), unit="s")
gb = df.groupby(pd.Grouper(freq="1D", key="Datetime"))
print(gb.value_counts())

raises ValueError: operands could not be broadcast together with shapes (5,) (6,) (5,)

[AfolabiOlaoluwa]

I don't know what your environment looks like, but this is what I did to investigate the problem.

It appears you are passing in an object for value_counts(). You CAN'T supply in a Class Object to value_counts() function

df["Datetime"] = pd.to_datetime(df["Timestamp"].apply(lambda t: str(t)), unit="s")
# note that I had to pass in the pd constructor to "to_datetime"
# print(df["Datetime"]) # Note that printing this returned properly 

gb = df.groupby(pd.Grouper(freq="1D", key="Datetime"))
# Note that the response was <pandas.core.groupby.generic.DataFrameGroupBy object at 0x3e58080>
# print(df.groupby(pd.Grouper(freq="1D", key="Datetime")))

If you look at the code below, it returns a TimeGrouper Class

print(pd.Grouper(freq="1D", key="Datetime")) 
>>> TimeGrouper(key='Datetime', freq=<Day>, axis=0, sort=True, closed='left', label='left', how='mean', convention='e', origin='start_day')

If you use a dot notation on the response, you can group by any item from the response. e.g. pd.Grouper(freq="1D", key="Datetime").key, which picks the value for the "key" which is "Datetime".

When you call .value_counts() on that, you will have an output. See the code below and the output:

veee = df.groupby(pd.Grouper(freq="1D", key="Datetime").key)
print(veee) # This outputs an object class which you are passing to value_counts()
print(veee.value_counts())

Your output will be:

# print(veee)
<pandas.core.groupby.generic.DataFrameGroupBy object at 0x4308910>

# print(veee.value_counts())
Datetime             Timestamp   Food  
2019-08-06 09:26:01  1565083561  apple     1
2019-08-07 09:26:01  1565169961  apple     1
2019-08-07 09:27:41  1565170061  banana    1
2019-08-09 09:26:01  1565342761  orange    1
2019-08-09 09:31:01  1565343061  orange    1
2019-08-10 09:26:01  1565429161  pear      1
dtype: int64

I hope this leads you to fixing the problem.

You may look more in detail about .groupby function here.
Also you can look at the value_counts() function here

[AfolabiOlaoluwa]

Also, note that if you leave your code untouched, you can .count() and .size().

[rhshadrach]

Thanks @AfolabiOlaoluwa - however the code in the top comment (without any modification) should not raise an error.

[AfolabiOlaoluwa]

I agree it shouldn't fail!

It looks like TimeGrouper is only imported when freq is not None and then calls super() function on it to access its components.

If we look at "new" function under grouper.py, when freq is absent (None) in the Grouper class, it works fine, and your solutions work fine without freq, but as a Grouper class.

import pandas as pd
import numpy as np

df = pd.DataFrame(
    {
        "Timestamp": [
            1565083561,
            1565083561 + 86400,
            1565083561 + 86500,
            1565083561 + 86400 * 2,
            1565083561 + 86400 * 3,
            1565083561 + 86500 * 3,
            1565083561 + 86400 * 4,
        ],
        "Food": ["apple", "apple", "banana", "banana", "orange", "orange", "pear"],
    }
).drop([3])

df["Datetime"] = pd.to_datetime(df["Timestamp"].apply(lambda t: str(t)), unit="s")

gb = df.groupby(pd.Grouper(key="Datetime"))
print(gb.value_counts())

output:

Datetime             Timestamp   Food  
2019-08-06 09:26:01  1565083561  apple     1
2019-08-07 09:26:01  1565169961  apple     1
2019-08-07 09:27:41  1565170061  banana    1
2019-08-09 09:26:01  1565342761  orange    1
2019-08-09 09:31:01  1565343061  orange    1
2019-08-10 09:26:01  1565429161  pear      1
dtype: int64

Its output is not different from the solution with df.groupby(pd.Grouper(freq="1D", key="Datetime").key) but as a Grouper instead of TimeGrouper

So, I am worried about why it looks like freq is empty in the TimeGrouper as freq=<Day>.

TimeGrouper(key='Datetime', freq=<Day>, axis=0, sort=True, closed='left', label='left', how='mean', convention='e', origin='start_day')