DOC: improve groupby reference docs

[jorisvandenbossche]

An overview of the reference doc on groupby is given here: http://pandas.pydata.org/pandas-docs/dev/api.html#groupby (apart from the extensive user guide: http://pandas.pydata.org/pandas-docs/dev/groupby.html)

There are some things that could use some improvement:

• add some missing functions to the overview in api.rst (Groupbydocs #8231)
  • GroupBy.filter
  • first/last/nth
  • count, cumcount, ..
  • name: not sure what the purpose of this is
• add the GroupBy object itself to the api docs (and so automatically all its methods) (DOC: SeriesGroupby/DataFrameGroupBy is missing class documentaion from doc index #19302)
• put all relevant docstrings in the GroupBy class, and not only in the subclasses DataFrameGroupBy, SeriesGroupBy (eg now the aggregate and transform docstrings of GroupBy are empty, but are more elaborate in the subclasses) (Groupbydocs #8231)
• general clean-up of all the docstrings
  • especially the apply docstring is not very clear to me
• expand DataFrame/Series.groupby() docstring:
  • clearly list all possibilities for the by arg (and provide some short examples in the 'Examples' section)
• document the whitelisted methods: (Groupbydocs #8231)
  • this could eg be done by injecting it in the docstring automatically based on _apply_whitelist
  • or alternatively by ensuring they appear in the methods list of the GroupBy class (which is not the case at the moment, only in instantiated objects) (see also discussion in No API reference for DataFrameGroupBy and "combining" step #2644)
• More clearly document the DataFrameGroupBy and SeriesGroupBy classes (Groupbydocs #8231)
  • at least mention them in the docs
  • one idea is to have a DataFrameGroupBy api pages that just redirects to the general GroupBy page
• Add docstrings to the wrapped whitelisted functions. Eg at present g = df.groupby(...); g.count? is returning <no docstring> (see Docs for lurking groupby methods #4500 (comment) for explanation how)
• Make a clear distinction, about what to expect for the return values of a grouped-apply, e.g. head/tail/nth are basically filter type of functions, fillna/shift are transformers, while almost everything else is a reducer (e.g. sum/mean/describe), while apply/agg can be any of the above. hmm. maybe needs a separate section for this. (and of course as_index just makes this crazy)

If someone wants to tackle this (or parts of this), go ahead!

[cpcloud]

👍 nice list

[mcjcode]

Question: Are all of the items on the SeriesGroupBy and DataFrameGroupBy whitelists meant to pass through to the underlying Series and DataFrame members? From what I can tell there are many items on the respective whitelists that are handled before they can ever be so dispatched. Here is a list:

first, last, min, max, sum - all defined as members in the GroupBy class definition (using _groupby_function). (GroupBy.prod is also defined in this way, but is not whitelisted.) And boxplot is also a defined member of DataFrameGroupBy (but not SeriesGroupBy).

count, cumcount, head, tail, mean, median - all defined, with def keyword, in the GroupBy class definition.

I ask because I'm proposing to define the whitelisted methods at class definition time, instead of relying on the GroupBy.__getattr__ to dynamically create them at the time they are invoked. And so sphinx will generate the documentation for all passed-through methods just like the ones that have first-class definitions in the Class.

But that often overrides the explicit method definitions listed above. And sometimes they don't even do the same thing. Which makes me suspect that these dozen or so whitelisted methods no longer really should be handled in this way (i.e. they already have explicit class definitions that take priority.)

So, can anyone think of a reason why we shouldn't just remove these names from the {Series,DataFrame}GroupBy whitelists? If not, I'll make a pull request addressing #6944 that uses this as part of the solution (with an appropriately updated test_groupby:test_groupby_whitelist.)

[jreback]

by definition the cythonized functions (and a few others which sometimes be cythonized) are defined inline on the groupby objects (eg min,mean,sum etc)
so these are not passed thru

you can define the whitelist methods in the classes (and just have the getattr raise an error I suppose)
(and this define the doc strings) - if a method exists in the class it is called before getattr is ever called so of course these override the whitelist

prod is prob an error that it is not defined
some methods only work in frames and not on series so that is done on purpose (boxplot)