BUG: Cut fails when number of bins is more than 102 on a 32-bit machine.

[QubinX]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import numpy as np
import pandas as pd

number = 102
example = pd.cut(x=pd.Series(range(100)), bins=pd.Series(range(number))) 
print(example)

Issue Description

The above code works as expected when number is <102, but fails when number >=102. This only occurs on a 32 bit machine. When this is run on a 64 bit machine, it does not fail.

The traceback when this fails is:
Traceback (most recent call last):

File "C:\Users\careyc\OneDrive - Harman Technology Ltd\Python\densitometer\cutbugtest.py", line 10, in
example = pd.cut(x=pd.Series(range(100)), bins=pd.Series(range(102)))

File "C:\Users\careyc\Anaconda3\lib\site-packages\pandas\core\reshape\tile.py", line 287, in cut
fac, bins = _bins_to_cuts(

File "C:\Users\careyc\Anaconda3\lib\site-packages\pandas\core\reshape\tile.py", line 450, in _bins_to_cuts
labels = Categorical(

File "C:\Users\careyc\Anaconda3\lib\site-packages\pandas\core\arrays\categorical.py", line 456, in init
codes = _get_codes_for_values(values, dtype.categories)

File "C:\Users\careyc\Anaconda3\lib\site-packages\pandas\core\arrays\categorical.py", line 2666, in _get_codes_for_values
return coerce_indexer_dtype(categories.get_indexer_for(values), categories)

File "C:\Users\careyc\Anaconda3\lib\site-packages\pandas\core\indexes\base.py", line 5274, in get_indexer_for
if self._index_as_unique:

File "pandas_libs\properties.pyx", line 37, in pandas._libs.properties.CachedProperty.get

File "C:\Users\careyc\Anaconda3\lib\site-packages\pandas\core\indexes\interval.py", line 727, in _index_as_unique
return not self.is_overlapping and self._engine._na_count < 2

File "C:\Users\careyc\Anaconda3\lib\site-packages\pandas\core\indexes\interval.py", line 454, in is_overlapping
return self._engine.is_overlapping

File "pandas_libs\properties.pyx", line 37, in pandas._libs.properties.CachedProperty.get

File "C:\Users\careyc\Anaconda3\lib\site-packages\pandas\core\indexes\interval.py", line 325, in _engine
return IntervalTree(left, right, closed=self.closed)

File "pandas_libs\intervaltree.pxi", line 77, in pandas._libs.interval.IntervalTree.init

File "pandas_libs\intervaltree.pxi", line 1025, in pandas._libs.interval.Int64ClosedRightIntervalNode.init

File "pandas_libs\intervaltree.pxi", line 1069, in pandas._libs.interval.Int64ClosedRightIntervalNode.new_child_node

File "pandas_libs\intervaltree.pxi", line 196, in pandas._libs.interval.take

TypeError: Cannot cast array data from dtype('int64') to dtype('int32') according to the rule 'safe'

Expected Behavior

Would expect the output to be:
0 NaN
1 (0.0, 1.0]
2 (1.0, 2.0]
3 (2.0, 3.0]
4 (3.0, 4.0]
.....
95 (94.0, 95.0]
96 (95.0, 96.0]
97 (96.0, 97.0]
98 (97.0, 98.0]
99 (98.0, 99.0]
100 (99.0, 100.0]
Length: 101, dtype: category
Categories (101, interval[int64, right]): [(0, 1] < (1, 2] < (2, 3] < (3, 4] ... (96, 97] <
(97, 98] < (98, 99] < (99, 100] < (100, 101]]

Installed Versions

INSTALLED VERSIONS

commit : 66e3805
python : 3.8.12.final.0
python-bits : 32
OS : Windows
OS-release : 10
Version : 10.0.19042
machine : x86
processor : x86 Family 6 Model 23 Stepping 10, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en
LOCALE : English_United Kingdom.1252

pandas : 1.3.5
numpy : 1.21.2
pytz : 2021.3
dateutil : 2.8.2
pip : 21.2.4
setuptools : 58.0.4
Cython : 0.29.25
pytest : 6.2.4
hypothesis : None
sphinx : 3.2.1
blosc : None
feather : None
xlsxwriter : 3.0.2
lxml.etree : 4.7.1
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.3
IPython : 7.29.0
pandas_datareader: None
bs4 : 4.10.0
bottleneck : 1.3.2
fsspec : 2022.01.0
fastparquet : None
gcsfs : None
matplotlib : 3.5.0
numexpr : 2.8.1
odfpy : None
openpyxl : 3.0.9
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.6.2
sqlalchemy : 1.4.27
tables : 3.6.1
tabulate : None
xarray : None
xlrd : 2.0.1
xlwt : 1.3.0
numba : 0.51.2

[JBEKaduceo]

Hello, I found a similar problem.

I'm using Ubuntu 20.04 with Python 3.8.2 and pandas 1.3.4.

There is no error thrown, but output are wrong when involving NaN values. Here is a small code :

import pandas as pd

series = pd.Series([1, 5, 10, None])

# Block A (output is correct)
i = [j for j in range(0, 101)]
display(pd.cut(x=series, bins=i))
display(pd.cut(x=series, bins=pd.IntervalIndex.from_breaks(i)))

print()

# Block B (output is incorrect)
i = [j for j in range(0, 102)]
display(pd.cut(x=series, bins=i))
display(pd.cut(x=series, bins=pd.IntervalIndex.from_breaks(i)))

Block A output is correct, either using bins as list or IntervalIndex, and is :

0     (0.0, 1.0]
1     (4.0, 5.0]
2    (9.0, 10.0]
3            NaN
dtype: category
Categories (100, interval[int64, right]): [(0, 1] < (1, 2] < (2, 3] < (3, 4] ... (96, 97] < (97, 98] < (98, 99] < (99, 100]]

As for Block B, the first output using bins as list is correct and similar to block A's output. But using bins as IntervalIndex produces a interval for the NaN (None) value, as following :

0      (0, 1]
1      (4, 5]
2     (9, 10]
3    (49, 50]
dtype: category
Categories (101, interval[int64, right]): [(0, 1] < (1, 2] < (2, 3] < (3, 4] ... (97, 98] < (98, 99] < (99, 100] < (100, 101]]

There is no problem below 102.

Moreover, there seems to be a pattern for that incorrect value, which is probably always equal to the "halfway class". For example if I set 300, I get the (148, 149] intervall, using following code :

i = [j for j in range(0, 300)]
display(pd.cut(x=series, bins=i))
display(pd.cut(x=series, bins=pd.IntervalIndex.from_breaks(i)))

that gives the following output for the IntervalIndex :

0        (0, 1]
1        (4, 5]
2       (9, 10]
3    (148, 149]
dtype: category
Categories (299, interval[int64, right]): [(0, 1] < (1, 2] < (2, 3] < (3, 4] ... (295, 296] < (296, 297] < (297, 298] < (298, 299]]

Output for the bins as list is correct.

Is that a pandas bug?

Thanks for your help!