BUG: boolean series .isin([pd.NA])] inconsistent for series length

[yoavrv]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
ser = pd.Series(True, index=range(10_000_000), dtype='boolean')
ser.iloc[0] = pd.NA
ser.iloc[1] =False
result_short = ser.iloc[:10].isin([False, pd.NA])
assert result_short.sum() == 2
result_long = ser.isin([False, pd.NA])

Issue Description

The behavior of the .isin() function on nullable boolean boolean series is inconsistent for pd.NA:
For short series, this returns a series with True for every null values in the original series

>>> pd.Series([True, False, pd.NA], dtype='boolean').isin([pd.NA])
0    False
1    False
2     True
dtype: boolean

For long series, this raises a TypeError: boolean value of NA is ambiguous

>>> pd.Series(True, index=range(1_000_000), dtype='boolean').isin([pd.NA])
0         False
1         False
2         False
3         False
4         False
          ...  
999995    False
999996    False
999997    False
999998    False
999999    False
Length: 1000000, dtype: boolean
>>> pd.Series(True, index=range(1_000_001), dtype='boolean').isin([pd.NA])
Traceback (most recent call last):
  File "<python-input-34>", line 1, in <module>
    pd.Series(True, index=range(1_000_001), dtype='boolean').isin([pd.NA])
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/homes/yoavr/.conda/envs/pandas_latest/lib/python3.13/site-packages/pandas/core/series.py", line 5559, in isin
    result = algorithms.isin(self._values, values)
  File "/homes/yoavr/.conda/envs/pandas_latest/lib/python3.13/site-packages/pandas/core/algorithms.py", line 505, in isin
    return comps_array.isin(values)
           ~~~~~~~~~~~~~~~~^^^^^^^^
  File "/homes/yoavr/.conda/envs/pandas_latest/lib/python3.13/site-packages/pandas/core/arrays/masked.py", line 971, in isin
    result = isin(self._data, values_arr)
  File "/homes/yoavr/.conda/envs/pandas_latest/lib/python3.13/site-packages/pandas/core/algorithms.py", line 545, in isin
    return f(comps_array, values)
  File "/homes/yoavr/.conda/envs/pandas_latest/lib/python3.13/site-packages/pandas/core/algorithms.py", line 534, in f
    return np.logical_or(np.isin(c, v).ravel(), np.isnan(c))
                         ~~~~~~~^^^^^^
  File "/homes/yoavr/.conda/envs/pandas_latest/lib/python3.13/site-packages/numpy/lib/_arraysetops_impl.py", line 1132, in isin
    return _in1d(element, test_elements, assume_unique=assume_unique,
           ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                 invert=invert, kind=kind).reshape(element.shape)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/homes/yoavr/.conda/envs/pandas_latest/lib/python3.13/site-packages/numpy/lib/_arraysetops_impl.py", line 982, in _in1d
    mask |= (ar1 == a)
             ^^^^^^^^
  File "missing.pyx", line 392, in pandas._libs.missing.NAType.__bool__
TypeError: boolean value of NA is ambiguous

Expected Behavior

.isin([pd.NA]) should either always work or it should always raise an error.

Installed Versions

INSTALLED VERSIONS

commit : 0691c5c
python : 3.13.1
python-bits : 64
OS : Linux
OS-release : 5.15.0-88-generic
Version : #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : C.UTF-8
LANG : en_US.UTF-8
LOCALE : C.UTF-8

pandas : 2.2.3
numpy : 2.2.1
pytz : 2024.1
dateutil : 2.9.0.post0
pip : 24.3.1
Cython : None
sphinx : None
IPython : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : None
lxml.etree : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : None
pyreadstat : None
pytest : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlsxwriter : None
zstandard : None
tzdata : 2024.2
qtpy : None
pyqt5 : None

[rhshadrach]

Thanks for the report! Confirmed on main, further investigations and PRs to fix are welcome.

[akj2018]

Hi @rhshadrach. I would like to take this issue. Please assign it to me. Thanks!

[snitish]

@akj2018 you can comment take to assign it to yourself

[akj2018]

take

[akj2018]

Hi everyone,

This is my first contribution to this repository and still learning. I’m open to constructive criticism and suggestions for improvement. If there are better ways to handle this scenario or additional areas where my fix could be improved, I’d greatly appreciate your insights!

No inconsistency due to pd.NA

After examining the error logs and debugging the reproducible example, I noticed

• Based on the length of comps_array, isin(comps: ListLike, values: ListLike) function in pandas\core\algorithms.py, has a choice between np.isin and htable.ismember
• With the threshold value of _MINIMUM_COMP_ARR_LEN = 1_000_000

def isin(comps: ListLike, values: ListLike) -> npt.NDArray[np.bool_]:
    """
    Compute the isin boolean array.

    Parameters
    ----------
    comps : list-like
    values : list-like

    Returns
    -------
    ndarray[bool]
        Same length as `comps`.
    """
...

 if (
        len(comps_array) > _MINIMUM_COMP_ARR_LEN
        and len(values) <= 26
        and comps_array.dtype != object
    ):
        # If the values include nan we need to check for nan explicitly
        # since np.nan it not equal to np.nan
        if isna(values).any():

            def f(c, v):
                return np.logical_or(np.isin(c, v).ravel(), np.isnan(c))

        else:
            f = lambda a, b: np.isin(a, b).ravel()

    else:
        common = np_find_common_type(values.dtype, comps_array.dtype)
        values = values.astype(common, copy=False)
        comps_array = comps_array.astype(common, copy=False)
        f = htable.ismember

    return f(comps_array, values)

What this means

1. result_short = ser.iloc[:10].isin([False, pd.NA]) triggered else part as the size of comps_array is 10 and used htable.ismember, which is an optimized hash-table-based function. Same case with pd.Series(True, index=range(1_000_000), dtype='boolean').isin([pd.NA]). else part has no issues with pd.NA

2. But result_long = ser.isin([False, pd.NA]) triggered 'if' part as size of comps_array is 10_000_000. This part caused TypeError: boolean value of NA is ambiguous whenever pd.NA was passed in the values

Conclusion

logic of isin() for pandas.Series differs based on length of input array passed, and datatype (mentioned in the if block). This proves there is no inconsistency due to pd.NA

Reason for TypeError: boolean value of NA is ambiguous

• When isin calls numpy.isin, it expects ndarray arrays. On reading algorithms.py, you will find that it converted inputs to ndarray from Line 518 to Line 524.

    elif isinstance(values, ABCMultiIndex): 
        # Avoid raising in extract_array
        values = np.array(values)
    else:
        values = extract_array(values, extract_numpy=True, extract_range=True)

    comps_array = _ensure_arraylike(comps, func_name="isin")

• Problem occurs inputs contains contains pd.NA which makes their dtype==object. The if part filters out cases with comp_array of dtype==object but doesn't care if values contains pd.NA

• Inside numpy.isin , it uses the _in1d function to perform element-wise comparisons. (code for _in1d)

• numpy tries to apply the result of the comparison (ar1 == a, which is pd.NA) in a boolean context (e.g., mask |= ...), triggering NAType.__bool__. And this throws TypeError("boolean value of NA is ambiguous")

What this means

If values contains incompatible types (e.g., pd.NA or mixed types), np.isin throws a TypeError or behaves unexpectedly because it cannot reliably handle comparisons involving object arrays.

Proposed Solution

To address this issue, I added a condition to exclude cases where values.dtype == object used np.isin optimization path:

if (
    len(comps_array) > _MINIMUM_COMP_ARR_LEN
    and len(values) <= 26
    and comps_array.dtype != object
    and values.dtype != object  # New condition to handle object dtype in values
)

Testing

I have not yet run the tests but plan to do so once this approach is reviewed and agreed upon. If there are any edge cases I should focus on or additional scenarios you’d like me to test, please let me know!

Looking forward to your feedback.
Thanks :)