BUG: is_unique fails when using float128 indexes, since pandas 2.0

[morotti]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

Hello,

Noticed a regression when using float128. Code to reproduce the bug below.

• It works fine on pandas 1.0.5 + 1.5.3
• It fails on pandas 2.0.3 + 2.1.4 + 2.2.2

is_unique is failing on first call, then giving an incorrect result on later calls.
is_monotonic_increasing gives an incorrect result.

I think we have two bugs at play here.

1. is_unique is obviously bugged, failing raising an exception.
2. is_monotonic_increasing depends on is_unique and they are both cached. There may be a bug in the caching code not handling is_unique raising an exception.

import pandas as pd
idx = pd.Index([2.0, 4.0, 6.0, 8.0, 10.0, 12.0, 14.0, 16.0], dtype='float128')
print("pandas version? ", pd.__version__)
print("monotonic? ", idx.is_monotonic_increasing)
print("unique? ", idx.is_unique)

Issue Description

First call of is_unique failing on pandas 2.2.2

# python debug_float128.py

pandas version?  2.2.2
monotonic?  False

Traceback (most recent call last):
  File "/users/is/rmorotti/debug_float128.py", line 5, in <module>
    print("unique? ", idx.is_unique)
                      ^^^^^^^^^^^^^
  File "properties.pyx", line 36, in pandas._libs.properties.CachedProperty.__get__
  File "/turbo/rmorotti/pyenvs/debugpandas/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 2346, in is_unique
    return self._engine.is_unique
           ^^^^^^^^^^^^^^^^^^^^^^
  File "index.pyx", line 266, in pandas._libs.index.IndexEngine.is_unique.__get__
  File "index.pyx", line 271, in pandas._libs.index.IndexEngine._do_unique_check
  File "index.pyx", line 333, in pandas._libs.index.IndexEngine._ensure_mapping_populated
  File "pandas/_libs/hashtable_class_helper.pxi", line 7105, in pandas._libs.hashtable.PyObjectHashTable.map_locations
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long double'

Expected Behavior

Works fine on pandas 1.5.3

# python debug_float128.py

pandas version?  1.5.3
monotonic?  True
unique?  True

Installed Versions

bug since pandas 2.0

[rhshadrach]

Thanks for the report. I don't believe pandas supports float128 in indexes. @jbrockmendel @mroeschke - should we? If not, we should likely raise a better error message here.

[jbrockmendel]

id be fine with adding a Float128Engine in index.pyx. There's probably a long tail of places where we have poor support for float128.

[morotti]

from the discussion, it seems float128 are not meant to be used/supported by pandas?
I talked to the guy who found the bug in my company and they found the bug by accident. they don't actually need float128 for their code.

for reference, there is almost no library/software supporting/using float128 because it is not supported by x64 CPU.
when it exists, the type is usually implemented as float 80 bits + 48 bits of padding, that is the highest precision available in hardware.
see numpy documentation: numpy.float128 and numpy.float96 are only 80 bits https://numpy.org/doc/stable/user/basics.types.html#extended-precision

it's fine by me if pandas doesn't support float128/float96. the issue is, it's accepting the type then returning invalid data when using the dataframe. would it possible to reject the types on creation with a more meaningful error message?

[rhshadrach]

@morotti - I believe dtype="float128" would be using NumPy's float128.

[rhshadrach]

Actually, without a PyArrow float128, I'm not sure we should be adding this to pandas. It would create more fragmentation between NumPy-backed and PyArrow-backed arrays.