BUG: When there are too many occurrences of the minimum value in a sequence, it can prevent the minimum value from being treated as a distinct and separate group

[wfwei]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

data = [0] * 3 + [1] * 3 + [ 2, 3, 4, 5]  # 10 sample 

pd.qcut(data, q=5, duplicates='drop').value_counts()

# (-0.001, 1.0]    6
# (1.0, 1.4]       0
# (1.4, 3.2]       2
# (3.2, 5.0]       2
# Name: count, dtype: int64

Issue Description

When there are too many occurrences of the minimum value in a sequence, it can prevent the minimum value from being treated as a distinct and separate group

Expected Behavior

data = [0] * 3 + [1] * 3 + [ 2, 3, 4, 5]  # 10 sample 

pd.qcut(data, q=5, duplicates='drop').value_counts()

# (-0.001, 0]    3
# (0, 1]       3
# (1, 3]       2
# (3 5]       2

Installed Versions

INSTALLED VERSIONS

commit : 37ea63d
python : 3.8.16.final.0
python-bits : 64
OS : Linux
OS-release : 4.19.91-011.ali4000.alios7.x86_64
Version : #1 SMP Fri Jun 4 21:54:18 CST 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : zh_CN.UTF-8
LANG : zh_CN.UTF-8
LOCALE : zh_CN.UTF-8

pandas : 2.0.1
numpy : 1.23.5
pytz : 2022.7
dateutil : 2.8.2
setuptools : 67.8.0
pip : 23.0.1
Cython : 3.0.0
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.3
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.12.0
pandas_datareader: None
bs4 : 4.12.2
bottleneck : None
brotli : None
fastparquet : None
fsspec : 2023.1.0
gcsfs : None
matplotlib : 3.7.1
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 12.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.9.1
snappy : None
sqlalchemy : None
tables : None
tabulate : 0.9.0
xarray : None
xlrd : None
zstandard : 0.19.0
tzdata : 2023.3
qtpy : None
pyqt5 : None

[wfwei]

a short-term solution :

def subtract_min_random(col, min_value=1e-10):
    min_value = col.min()
    random_index = np.random.choice(col[col == min_value].index)
    col[random_index] -= min_value
    return col
    
data = [0] * 3 + [1] * 3 + [ 2, 3, 4, 5]  # 10 sample 
data_m = subtract_min_random(pd.Series(data))
pd.qcut(data_m, q=5, duplicates='drop').value_counts()


# (-0.101, 0.0]    3
# (0.0, 1.0]       3
# (1.4, 3.2]       2
# (3.2, 5.0]       2
# (1.0, 1.4]       0
# Name: count, dtype: int64

[ChrisMountn]

take

[ChrisMountn]

I have added a fix here, guided by the examples and expected behavior. That being said, for cases like this, it might make more sense to leave duplicate intervals in instead of dropping them and re-cutting the data across fewer quantiles. Should I add a 'keep' option to the duplicates parameter?

For example, for the example above (0, 0, 0, 1, 1, 1, 2, 3, 4, 5), the actual quantiles with q = 5 are:
0%: 0
20%: 0
40%: 1
60%: 1
80%: 3
100%: 5

We can remove the duplicates as is the current behaviour of this function, but I think often users would want to keep the duplicates so as to get an actual distribution that aligns with q=5 rather than implicitly reducing it to q=4 (shown in the expected output in the example above). Let me know what you think and we can adjust the PR accordingly.

[rhshadrach]

That being said, for cases like this, it might make more sense to leave duplicate intervals in

The result of qcut itself should have duplicate intervals; this is just a guess, but did you by chance miss that the OP used .value_counts on the result of qcut?