Description
Pandas version checks
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd
import numpy as np
#case 1
cat1 = pd.Series([np.nan, 'a', 'b', 'c', 'd', 'd']).astype(
pd.CategoricalDtype(categories=['a', 'b', 'c', 'd'], ordered=True))
# next line works because if-block in quantile.py (line 193)
# contains cast from float to int (result = result.astype(values.dtype, copy=False))
cat1.quantile(0.5)
#case 2
cat2 = pd.Series(['a', 'b', 'c', 'd', 'd']).astype(
pd.CategoricalDtype(categories=['a', 'b', 'c', 'd'], ordered=True))
# next line raises AssertionError because else-block in quantile.py (line 215)
# does NOT contain cast from float to int (result = result.astype(values.dtype, copy=False))
cat2.quantile(0.5)Issue Description
The first case works as expected and outputs 'c'.
The second case doesn't work as expected.
I have traced this back to the lines of code 193-225 in quantile.py of Pandas 1.5.3.
When the Series contains NA-values, an if-block will be executed because mask.any() returns true (line 193-214) and this will result in an extra cast of a float value to int.: result = result.astype(values.dtype, copy=False)
In the other case, the else-block (lines 215-225) is executed, and this block doesn't contain the same cast from float to int.
In both cases, though, the underlying result of the quantile computation is correct (i.e.: 2.0 --> 'c').
Expected Behavior
The quantile function should function in all cases where the quantile value can be computed.
In the case where the quantile(0.5) function can't be computed, the same AssertionError is raised, but this not very informative.
A more informative error would be warranted.
#case with extra 'a' --> has no Q2 value.
s3 = pd.Series([np.nan, 'a', 'a', 'b', 'c', 'd', 'd']).astype(
pd.CategoricalDtype(categories=['a', 'b', 'c', 'd'], ordered=True))
#next line raises sames AssertionError as in s2, but this is not very informative.
cat3.quantile(0.5)
Tested solution
Change line 216 in quantile.py
return np.percentile(
values,
qs,
axis=1,
# error: No overload variant of "percentile" matches argument types
# "ndarray[Any, Any]", "ndarray[Any, dtype[floating[_64Bit]]]",
# "int", "Dict[str, str]" [call-overload]
**{np_percentile_argname: interpolation}, # type: ignore[call-overload]`with this:
result = np.percentile(
values,
qs,
axis=1,
# error: No overload variant of "percentile" matches argument types
# "ndarray[Any, Any]", "ndarray[Any, dtype[floating[_64Bit]]]",
# "int", "Dict[str, str]" [call-overload]
**{np_percentile_argname: interpolation}, # type: ignore[call-overload]
return result.astype(values.dtype, copy=False)Installed Versions
pyxlsb : None
s3fs : None
scipy : 1.10.1
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None
tzdata : None