Raise ValueError When Attempting to Rank Object Dtypes

[WillAyd]

Referencing the comments in #19481, right now rank operations performed against objects have a few issues, namely that they:

• Are inherently ambiguous, relying on lexical encoding AND
• Are not consistent across Series, DataFrame and GroupBy objects with various arguments

To illustrate the latter:

In [1]: vals = ['apple', 'orange', 'banana']
In [2]  pd.Series(vals).rank()  # this will "work"
Out[8]: 
0    1.0
1    3.0
2    2.0
dtype: float64

In [3]: pd.Series(vals).rank(method='first')  # raises
ValueError: first not supported for non-numeric data

In [4]: pd.DataFrame({'key': ['foo'] * 3, 'vals': vals}).groupby('key').rank(method='first')  # should raise?
Out[4]: 
Empty DataFrame
Columns: []
Index: []

(see also #19482)

With this change I'd propose that we simply raise ValueError consistently for rank against object dtypes regardless of which type of object performs the transformation and regardless of arguments.

One known caveat is that Categorical types currently use the rank_object methods in algos. My assumption is that we would want to continue supporting ranking for ordered Categoricals but raise for unordered Categoricals.

[jreback]

this is manifesting in #11759 as well. just need a better error message.

[ishaan007]

@jreback I am looking to work on this, any pointers on which particular classes should I start inspecting ?

[WillAyd]

@ishaan007 you want to be looking at the various rank implementations in pandas.core.generic.NDFrame, pandas.core.algorithms and pandas.core.groupby.GroupBy

[mapehe]

In case @ishaan007 isn't still on this I could have a look since this is tagged "good first issue" and would seem like a good way to get to know the codebase a bit.

[WillAyd]

@mapehe sure give it a shot

[mapehe]

• Are not consistent across Series, DataFrame and GroupBy objects with various arguments

So I've had a look at the above point, please correct me if I've misunderstood something. Series and DataFrame both inherit rank() from pandas.core.generic.NDFrame which essentially executes rank() in pandas.core.algorithms. This eventually calls a cython implementation in algos_rank_helper.pxi. The function rank() in GroupBy goes through a completely distinct set of functions and eventually calls an implementation in groupby_helper.pxi.

Running pd.Series(vals).rank(method='first') would cause an error because of this check:

pandas/pandas/_libs/algos_rank_helper.pxi.in

Lines 166 to 167 in 2431641

	elif tiebreak == TIEBREAK_FIRST:
	raise ValueError('first not supported for non-numeric data')

On the other hand pd.Series(vals).rank() works because a similar error is not raised for method=average.

In the case of GroupBy, an implementation for object ranking doesn't seem to exist because of line 412?

pandas/pandas/_libs/groupby_helper.pxi.in

Lines 412 to 419 in 2431641

	{{if name != 'object'}}
	@cython.boundscheck(False)
	@cython.wraparound(False)
	def group_rank_{{name}}(ndarray[float64_t, ndim=2] out,
	ndarray[{{c_type}}, ndim=2] values,
	ndarray[int64_t] labels,
	bint is_datetimelike, object ties_method,
	bint ascending, bint pct, object na_option):

I don't know the codebase well enough to make a good decision about how these issues should be handled. If GroupBy.rank() is not supposed to support non-numeric data, maybe a type check would be appropriate around here?

pandas/pandas/core/groupby/groupby.py

Lines 2453 to 2461 in 2431641

	def _cython_operation(self, kind, values, how, axis, min_count=-1,
	**kwargs):
	assert kind in ['transform', 'aggregate']
	# can we do this operation with our cython functions
	# if not raise NotImplementedError
	# we raise NotImplemented if this is an invalid operation
	# entirely, e.g. adding datetimes

I guess the inconsistency between pd.Series(vals).rank() and pd.Series(vals).rank(method='first') could be solved by modifying algos_rank_helper.pxi.in, but I have no idea how many things it would break. :)

[WillAyd]

Nice research. Best approach would be to simply raise at the start of the method you found in algos for object dtypes, rather than only doing it for the TIEBREAK_FIRST argument

[mapehe]

Thanks for the reply @WillAyd. So you'd suggest replacing the currentrank_1d_object() with a function that just runs raise ValueError('first not supported for non-numeric data')?