Zero counts in Series.value_counts for categoricals

[fkaufer]

Series.value_counts() also shows categories with count 0.

Thought this would be a bug but according to doc it is intentional.

This makes the output of value_counts inconsistent when switching between category and non-category dtype. Apart from that it blows up the value_counts output for series with many categories.

I would prefer to hide counts (i.e. zero) for non-occuring categories by default and rather consider a parameter dropzero=True similar to dropna (see also #5569).

[jreback]

can u show a specific example (of what u think it should do)
this with 0,15rc1?

[jreback]

cc @JanSchulz

[fkaufer]

s = pd.Series(['a','b','a','c','d','c'])
count_str = s[s.isin(['a','u'])].value_counts()
count_cat = s.astype('category')[s.isin(['a','u'])].value_counts()

count_str
a    2

count_cat
a    2
d    0
c    0
b    0

assert count_str==count_cat
...
ValueError: Series lengths must match to compare

[fkaufer]

.. and yes version is 0.15.0rc1-24-g56dbb8c

[jankatins]

I think the current behaviour is correct: A categorical is not a more memory efficient string dtype but a dtype with a fixed set of values. One of the main points for categoricals is that "unused" categories show up in all kind of operations, e.g. during groupby and during value counts. This will come in handy in ggplot, where plot axis should be the same for all facets and unused cats should show up with length zero bars.

If you want to have the same output, you need to do the "isin" with the results of value_count() (untested, I don't have a recent pandas env right now):

count_str = s[s.isin(['a','u'])].value_counts()
temp = s.astype('category').value_counts()
count_cat = temp[temp.index.isin(["a","u"])] # untested, I hope index has that method

[jankatins]

IMO, it's also consistent, as 'value_count' counts every value it knows about and in case of categoricals it knows that there are more than only the "used" categories.

[jreback]

what about adding the dropna=False arg?
since the default is different would this be confusing?

[jankatins]

dropzero=False would be ok, but on the other hand you can do that afterwards as well in a similar manner:

ret= ... 
if dropzero: 
    return ret[ret.count!=0] # probably needs a copy to get around the 'setting_with_copy`thingy...
else 
   return ret

[jankatins]

@fkaufer What is actually the use case here, i.e., why do you need a Categorical and zero-cats removed?

[fkaufer]

• Series example with isin was only a minimalistic example. Does not reflect my use cases.
• Typical application is complex filtering on many df columns, then using value_counts as convenient tool to find out for which categories this filter holds true. Typical use case: interactive data cleansing/exploration with negative filter supposed to return only those - few - categories for which idiosyncrasies (in the other columns) are found. For instance:
  • df_quotes[outlier_filter].symbol.value_counts()
  • df_shipment[outlier_filter].airport_dep.value_counts()
• This issue seems to be deeper and not restricted to value_counts, Series.unique() also returns all categories which I consider even more problematic. So df_quotes[outlier_filter].symbol.unique() is equivalent to df_quotes.symbol.cat.categories. Only df_quotes[outlier_filter].symbol.astype(str).unique() does what I'd expect but I hope I don't have to do that. Gotcha alert!
• Regarding the plotting argument: I guess there are situations where one or the other (keep zero-cats or not) comes in handy (similar to dropna). And speaking of facet plotting, I'm having trouble with seaborn.FacetGrid right now due to keep-all-categoricals behaviour (many empty facet subplots), so I have to convert back to string before using FacetGrid. FacetGrid relies on Series.unique, see https://github.com/mwaskom/seaborn/blob/master/seaborn/axisgrid.py#L205. That's exactly what I meant regarding the inconsistency between explicit categoricals and implicit string categoricals as used so far. So this is really a matter of being backward-compatible in a way.
• Conceptual argument: IMO categorical is a separation of internal and external representation. The external representation, the label, is only meta-data but not data and meta-data should not be present if the underlying data is - virtually - not existent. But admittedly there is no consensual view. R's table keeps zero frequencies for factor variables, Stata's tabulate doesn't for encoded variables.
• Re "A categorical is not a more memory efficient string dtype": I'd say "not only", but for me memory efficiency is a very important and for now the most important reason to use categoricals. Thanks to that (and thanks to you!) I'm currently working with a dataset on my laptop which before I could only handle on a server.

[jankatins]

Re plotting: having all values preserved in all facets (so not cat x cat facets but a cat variable as x axis in each facet -> zero values turn up as zero length bars) is actually the use case for ggplot where cats are wanted (in ggplot this relies on value_counts()). I'm not sure what to make of the 'cat as facet variable" case. R's ggplot2 removes empty categories in that case:

library(reshape2)
library(ggplot2)
levels(tips$sex) <- c("f", "m", "-")
sp <- ggplot(tips, aes(x=total_bill, y=tip/total_bill)) + geom_point(shape=1)
sp + facet_grid(sex ~ .)

Interestingly unique returns a factor (with all levels, but only the "used" levels as values) when the input is a factor:

> unique(tips$sex)
[1] f m
Levels: f m -
> unique(as.character(tips$sex))
[1] "f" "m"

This is IMO a argument to drop unused categories in unique().

As a workaround for your seaborn problem, you can use df.variable.cat.drop_unused_categories(inplace=True) before faceting

Re value_count: value_count() is equivalent to:

> table(tips$sex)
  f   m   - 
 87 157   0 

I think a dropzero=False default argument can be done for value_count() but that essential means that this is either ported to all value_count() methods (where it makes not sense) or you have to test for categorical series (in which case you could also simple remove all zero counted values of the returned dataframe).

Or you have a "remove unused categories" step in between...

What will happen in your app when you reorder the categories (e.g., "one" < "two" < "three")?

Re metadata: I see it that the levels are actually part of every item of the categorical data (which it is in R, but right now not in pandas: getting a single item will return a single item cat in R but a int/string/... in pandas):

> tips$sex[1]
[1] f
Levels: f m -

From your metadata comment and the last bullet I think what you want is a memory efficient string dtype. This could actually be done by subclassing categorical and "hiding" the categorical thingies and add categories during set automatically. Should actually be almost trivial... This was actually one argument to implement such a data type in numpy so that they have a proper variable length string dtype :-)

=> I see the problem with unique() but not with value_count() and seeing categories as "to be hidden" meta data.

[jankatins]

Oh my:

> library(reshape2)
> library(dplyr)
> levels(tips$sex) <- c("f", "m", "-")
> gb = group_by(tips, sex)
> summarise(gb,count = n())
Source: local data frame [2 x 2]

  sex count
1   f    87
2   m   157

-> dplyr omits unused levels in group_by

Following this would mean that pandas groupby should also not return empty (unused) categories...

@hadley is that intentional?