No API reference for DataFrameGroupBy and "combining" step

[jankatins]

At least by searching google, I found no references for the DataFrameGroupBy. The docs at http://pandas.pydata.org/pandas-docs/dev/groupby.html speak about a "GroupBy object"...

What are all the functions of that object? Using IPython, I get inline help, but this informations seems to be missing in the online docs.

I'm also missing the "combine" step in the split-apply-combine doc: how can values be added to the original dataframe? I only found something about transform, which returns a complete dataframe. grouped.apply() returns a single Series. Did I miss something or is the combine step really just a "merge the new dataframe/series with the old one".

[mhlr]

+1. An undocumented GroupBy really limits the usefulness of Pandas.

[jreback]

well the groupby docs are quite complete, what exactly is the problem?

if you read thru them, you will see exactly what a GroupBy object is:

http://pandas.pydata.org/pandas-docs/dev/groupby.html#splitting-an-object-into-groups

And the fact that it applies the functions directly

http://pandas.pydata.org/pandas-docs/dev/groupby.html#aggregation

If you think their is something missing, the pls be specific about it

+1. An undocumented GroupBy really limits the usefulness of Pandas.

is a very odd statement

[cpcloud]

There are some API docs missing for GroupBy objects, I think that is what @JanSchulz is talking about

• filter: (even though that is documented elsewhere)
• first/last: (these are different from the DataFrame and Series methods and should be documented)
• name: not sure what the purpose of this is
• nth: very useful when you want to try out an operation on a more expensive groupby apply (e.g., many levels)

[jorisvandenbossche]

Yep, I also follow the initial poster. @jreback I think the groupby guide is already in very good shape, certainly. But I personally think it is important to see that the user guide and reference docs are complementary, and both important.
And there are indeed some things missing in the api docs (I was also working on a draft issue to list some of the missing docs, will try to finish that).

@mhlr It would indeed be a bit more constructive to say what you miss than just saying that it limits the usefulness. In fact, if you would want to do that, give some concrete things you miss in the docs about this, that would be very helpful for us! Really.
(a PR even more helpful .. :-) but already knowing what is missing also. If you are working a lot on this, it is sometimes difficult to see what information is missing for a new user).

@JanSchulz What do you exactly mean with the missing "combine" step? Because I think you misinterpret the combine step, as this is about combining the different groups together in one dataframe, not about combining the output of the groupby with the original array (for that you can just assign it to a column in the original array, or concat both, ..) That is not clear enough from the first paragraph in the docs?

[jorisvandenbossche]

And something else, in the original issue @JanSchulz also mentions the fact that the docs only speak about the GroupBy object, while in practice you always get a DataFrameGroupBy of SeriesGroupBy object. For us it is maybe obvious, but not for a lot of users I thinkt (but not much to do about that I suppose? apart from mentioning it in the docs).

[cpcloud]

In my experience with pandas i've almost never cared about the distinction between the types of groupby objects, whereas with series vs frame i have to be more cognizant of any api differences (but even then not so much). IMO distinguishing between the different kinds of groupbys in the main docs is too much detail and is best left for api docs.

[mhlr]

DataFrame.groupby is well documented. It returns a DataFrameGroupBy object which is not documented.
I see no document which gives a definitive list of DataFrameGroupBy methods with their signatures and assumtions/requirements/contracts the way that http://pandas.pydata.org/pandas-docs/version/0.13.1/generated/pandas.DataFrame.html does for DataFrame. There is no http://pandas.pydata.org/pandas-docs/version/0.13.1/generated/pandas.DataFrameGroupBy.html
There is a an introduction with examples at
http://pandas.pydata.org/pandas-docs/dev/groupby.html
but that does not constitute an API reference.

[jorisvandenbossche]

@cpcloud yes, I agree fully with that, but still the user sees DataFrameGroupBy, and maybe you know that you shouldn't care about, but some other users don't

[jreback]

I suppose easy enough to add the whitelisted methods in the groupby API section, but seems kind of repetive, maybe list them in the groupby docs (so we can auto-generate them)...that is the big issue here.

DataFrameGroupby/SeriesGroupby.sum() is de facto DataFrame.sum()

with the exception of certain methods which are slightly different (which is itself an API bug, they should be the same)

[cpcloud]

i don't think listing whitelisted methods is appropriate ... why not just list the methods that are specific to groupby-type objects?

[jorisvandenbossche]

@mhlr Thanks for the elaboration. The GroupBy object is indeed missing in the API docs (and it is not much work to add it actually, just one line in api.rst). As said in the previous comments, the DataFrameGroupby is actually the 'same' (for a user).

[jorisvandenbossche]

@cpcloud Why isn't it appropriate? Why otherwise whitelist them? They are whitelisted because they are usefull, so user should be able to know about that?

[jreback]

well, can we auto-generate them in the API section? (want to avoid manually doing this, so they they are only in 1 place, namely the whitelist)

[mhlr]

I think it would be useful to {DataFrame,Series}GroupBy API pages as well even if they just link or redirect to the GroupBy page just to avoid the initial confusion caused by not being able to find them. Definitely should have a GroupBy page.

[jreback]

@mhlr did you see this:

Well some are already documented: http://pandas.pydata.org/pandas-docs/stable/api.html#groupby