BUG: unexpected cut() results

[oleksiyskononenko]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

import pandas as pd
nbins = 42
maxs = [96, 97, 98]

for max in maxs:
    data = [-max, 0, max]
    bins = pd.cut(data, nbins, labels=False)
    print("Data: ", data)
    print("Bins: ", bins, "\n")

Problem description

According to documentation, I would expect the 0 value to go to the bin 20. For max == 96 and max == 98 it does go to the bin 20, however, when max == 97 for some reason 0 goes to the bin 21:

Data:  [-96, 0, 96]
Bins:  [ 0 20 41]

Data:  [-97, 0, 97]
Bins:  [ 0 21 41]

Data:  [-98, 0, 98]
Bins:  [ 0 20 41]

Feels like a bug, or may be I'm missing something on the docs that explains such a cut() behavior. Would appreciate any help on this. Thanks.

Expected Output

Data:  [-96, 0, 96]
Bins:  [ 0 20 41]

Data:  [-97, 0, 97]
Bins:  [ 0 20 41]

Data:  [-98, 0, 98]
Bins:  [ 0 20 41]

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.6.8.final.0
python-bits : 64
OS : Darwin
OS-release : 19.2.0
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.0.5
numpy : 1.16.1
pytz : 2018.9
dateutil : 2.8.0
pip : 20.0.2
setuptools : 40.6.3
Cython : None
pytest : 4.1.1
hypothesis : None
sphinx : 2.4.4
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.10
IPython : 7.8.0
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pytest : 4.1.1
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : 1.2.0
xlwt : None
xlsxwriter : None
numba : None

[sdfordham]

The difference is from numpy.searchsorted here

pandas/pandas/core/reshape/tile.py

Line 388 in b687cd4

ids = ensure_int64(bins.searchsorted(x, side=side))

When in case 96.0 or 98.0 the value in the middle of the array is 0.0 but for 97.0 it is very small but not 0.0 so the side='left' option changes it. The following seems to sum up the situation

print(-99.0 + ((2 * 99.0)/42.0 * 21.0))
print(-98.0 + ((2 * 98.0)/42.0 * 21.0))
print(-97.0 + ((2 * 97.0)/42.0 * 21.0))
print(-96.0 + ((2 * 96.0)/42.0 * 21.0))
print(-95.0 + ((2 * 95.0)/42.0 * 21.0))

Output

0.0
0.0
-1.4210854715202004e-14
0.0
0.0

[oleksiyskononenko]

Thanks for the info. Yeah, when I looked at the binning intervals for max == 97, I did notice this small number instead of zero.

[oleksiyskononenko]

This also looks wrong (15 doesn't fall into the (15.0, 35.568] bin)

>>> pandas.cut([-890, 15, 920], 88)
[(-891.81, -869.432], (15.0, 35.568], (899.432, 920.0]]
Categories (88, interval[float64]): [(-891.81, -869.432] < (-869.432, -848.864] < (-848.864, -828.295] <
                                     (-828.295, -807.727] ... (837.727, 858.295] < (858.295, 878.864] <
                                     (878.864, 899.432] < (899.432, 920.0]]

[TomAugspurger]

Any recommendations on what should be done here? I'm inclined to just follow NumPy's behavior.

[sdfordham]

This is some kind of floating point arithmetic issue in cpython. I assumed it was due to representing the prime 97 as a float, but now actually I see you dont get the same behaviour with e.g. 89.0 but you do with e.g. 87.0, 92.0, 102.0.

You would need to include some kind of very small number check but I'm not sure where/when you need to put it.

[jaepil-choi]

Any recommendations on what should be done here? I'm inclined to just follow NumPy's behavior.

I think it is a very counterintuitive behavior. Any Python user would expect the range to be [a, b) instead of (a, b] just like np.digitize(np_arr, bins=[0, 5, 8]) or range(1, 10).

For example, say I want to categorize age.

dd = pd.DataFrame(data=[-1, 0, 1, 29, 30, 31, 59, 60, 61, 149, 150, 151], columns=['age'])

To categorize age

• <30: young
• 30<= 30 < 60: middle
• 60<: old

I have to do

pd.cut(dd.age, bins=[0, 29, 59, 150], labels=['young', 'middle', 'old'])

instead of

pd.cut(dd.age, bins=[0, 30, 60, 150], labels=['young', 'middle', 'old'])

which is in my opinion not quite intuitive.

[sdfordham]

@jaepil-choi Use the right argument

import pandas as pd

dd = pd.DataFrame(data=[-1, 0, 1, 29, 30, 31, 59, 60, 61, 149, 150, 151], columns=['age'])
print(pd.cut(dd.age, bins=[0, 30, 60, 150], labels=['young', 'middle', 'old'], right=False))

In future please open a new issue, this is not strictly relevant to the original issue opened

[jaepil-choi]

@sdfordham Oh I thought it was related. Thanks, I'll use right= then.