API: inconsistent return format of groupby apply

[edfall]

related issues:

• groupby(..).apply() has inconsistent behavior when there is only one group #13121
• df.groupby().apply() with only one group returns wrong shape! #5839
• It's difficult to predict what DataFrame.groupby().apply() will return: #9867
• df.groupby(col).resample('x') returning both unstacked or multiindex result #13255
• Dataframe group apply can fail on trivial cases. #20066
• DataFrame.groupby.apply returns different results with copy lambda functions. #14927 (about detecting mutating functions)

Thank you very much for provide us pandas which is really a good packages I have used!

There were several times when I use pandas but get inconsistent return format which finally leads to a break!

• for example:

import numpy as np
import pandas as pd
df = pd.DataFrame(np.random.randn(10,1), columns = ['x'])
df['y'] = 1
df.groupby('y').apply(lambda x: x.x)

#x        0         1         2         3         4         5         6  \
#y                                                                        
#1 -1.12114  0.679616 -1.392863  0.032637 -0.051134  0.594201 -0.238833   

#x        7        8         9  
#y                              
#1  0.95173  1.07469 -0.062198  

df2 = df.copy()
df2['y'] = np.random.randint(1,3,10)
# do something similar like above
df2.groupby('y').apply(lambda x: x.x)
#y   
#1  3    0.032637
#   8    1.074690
#2  0   -1.121140
#   1    0.679616
#   2   -1.392863
#   4   -0.051134
#   5    0.594201
#   6   -0.238833
#   7    0.951730
#   9   -0.062198
#Name: x, dtype: float64

even though, I knew the way to avoid this issue, I still believe it is better to return the same format. It's really annoying when some inconsistent return format happened which force me to look backwards to use some awkward methods to avoid it.

So, if possible, I do suggest the above function will only return one kind of format.

[jreback]

You are trying to do too much inside an apply which is completely non-performant. .apply has to guess your return output. I suppose this is slightly confusing, in any event you should simply be doing this.

In [14]: df.groupby('y').x.apply(lambda x: x)
Out[14]: 
0    2.481148
1   -1.324170
2    0.783518
3    0.869827
4   -0.080157
5    0.071685
6    0.987246
7    0.099149
8   -0.159449
9   -0.383200
Name: x, dtype: float64

In [15]: df2.groupby('y').x.apply(lambda x: x)
Out[15]: 
0    2.481148
1   -1.324170
2    0.783518
3    0.869827
4   -0.080157
5    0.071685
6    0.987246
7    0.099149
8   -0.159449
9   -0.383200
Name: x, dtype: float64

I'll mark it as an issue if you'd like to investigate whether this could be made consistent w/o breaking anything else (might not be possible).

[jreback]

this was changed by this: #3239

sort of a special case was added I think.

[edfall]

yep, I think the way you(@jreback) use do solve problem of inconsistent return format, However, sometimes it's important to keep the groupby variables as a label, which your method failed to.

what's more, the function in .apply formula can be more complicated as the aim and data changes.

[jreback]

@edfall and so how would you reconcile this issue?

I don't think you can infer what format the user wants and meet all users expectations.

[edfall]

@jreback

of cause, it's really difficult to meet all users' satisfaction, but return format of any specific application should always keep the same. and I think the result should at least be something like:

 df.groupby('y').apply(lambda x: x.x)
#y
#1 0    2.481148
#  1   -1.324170
#  2    0.783518
#  3    0.869827
#  4   -0.080157
#  5    0.071685
#  6    0.987246
#  7    0.099149
#  8   -0.159449
#  9   -0.383200

Maybe I have use R package data.table, which can do similar things as '.groupby().apply' do, but all results of its' application is predictable.