DEPR: Change default to `observed=True` in `DataFrame.groupby`

[Seon82]

Is your feature request related to a problem?

The default behaviour of pandas.DataFrame.groupby is currently different depending on the type of the groupers (when one of the groupers is categorical, unobserved categories are added to the groupby by default. This behaviour can be overriden by setting the observed argument to False).

I feel like making the groupby API consistent by default and regardless of the underlying data type would provide a much better user experience.

Describe the solution you'd like

Default to observed=False in pandas.DataFrame.groupby.

API breaking implications

Would break backwards-compatibility.

Describe alternatives you've considered

So far the only option I can think of is to add observed=True to every groupby I write to make sure it will behave correctly no matter what kind of data gets passed to it.

[jreback]

pls search the tracker this is a duplicate request

[Seon82]

Sorry, I completely missed it! And I still seem unable to find it no matter what synonyms I try, would you mind sending a link if you have one handy?

[jreback]

see #35967 and linked issues

i guess we don't have an actual issue for this (or maybe one of the linked ones)

cc @jseabold made a really good effort here

[PMLP-novo]

An alternative suggestion could be to that the observed was determined at runtime by default. So if there will be created more groups than lets say 100,000,000 if groups are created in the Cartesian way, then we automatically change to observed = true.
I the code this should be having the default observed = None. This solution will be backwards compatible if users have set observed.