BUG: Inconsistent behavior in comparison operators between scalar and series with arrow types

[rohanjain101]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

>>> i = pd.Series([1], dtype="int64[pyarrow]")
>>> i == True
0    True
dtype: bool[pyarrow]
>>>
>>> b = pd.Series([True], dtype="bool[pyarrow]")
>>> i == b
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\pd_202_env\Lib\site-packages\pandas\core\ops\common.py", line 81, in new_method
    return method(self, other)
           ^^^^^^^^^^^^^^^^^^^
  File "D:\pd_202_env\Lib\site-packages\pandas\core\arraylike.py", line 40, in __eq__
    return self._cmp_method(other, operator.eq)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\pd_202_env\Lib\site-packages\pandas\core\series.py", line 6096, in _cmp_method
    res_values = ops.comparison_op(lvalues, rvalues, op)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\pd_202_env\Lib\site-packages\pandas\core\ops\array_ops.py", line 279, in comparison_op
    res_values = op(lvalues, rvalues)
                 ^^^^^^^^^^^^^^^^^^^^
  File "D:\pd_202_env\Lib\site-packages\pandas\core\ops\common.py", line 81, in new_method
    return method(self, other)
           ^^^^^^^^^^^^^^^^^^^
  File "D:\pd_202_env\Lib\site-packages\pandas\core\arraylike.py", line 40, in __eq__
    return self._cmp_method(other, operator.eq)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\pd_202_env\Lib\site-packages\pandas\core\arrays\arrow\array.py", line 467, in _cmp_method
    result = pc_func(self._data, other._data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\pd_202_env\Lib\site-packages\pyarrow\compute.py", line 242, in wrapper
    return func.call(args, None, memory_pool)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow\_compute.pyx", line 367, in pyarrow._compute.Function.call
  File "pyarrow\error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow\error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Function 'equal' has no kernel matching input types (int64, bool)
>>>

Issue Description

The behavior is different when a scalar is provided vs a series.

Expected Behavior

I would expect the behavior to be the same, either both raise not implemented, or both succeed.

Installed Versions

pd.show_versions()
D:\pd_202_env\Lib\site-packages_distutils_hack_init_.py:33: UserWarning: Setuptools is replacing distutils.
warnings.warn("Setuptools is replacing distutils.")

INSTALLED VERSIONS

commit : 965ceca
python : 3.11.2.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.22621
machine : AMD64
processor : Intel64 Family 6 Model 85 Stepping 7, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252

pandas : 2.0.2
numpy : 1.24.3
pytz : 2023.3
dateutil : 2.8.2
setuptools : 67.7.2
pip : 23.1.2
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 12.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

[mroeschke]

Thanks for the report. This operation fails on the pyarrow side and ArrowExtensionArray has a fallback for scalar inputs. This is a little harder to generalize for array inputs due to behavior in cases like mismatched lengths

[rohanjain101]

Thanks @mroeschke for checking, would it be reasonable to assume the expected behavior of a scalar should match the behavior of a series containing the same data with the same index?

[mroeschke]

Yes that sounds reasonable

[rohanjain101]

Thanks, is the behavior of the fallback intentionally implemented? In other words, is the expected behavior that the series case should also have the fallback, or the scalar case should not have the fallback? I ask because when calling the equal method in arrow compute with an array/scalar, it still raises an error:

>>> a = pa.array([1,2,3])
>>> pa.compute.equal(a, pa.scalar(True))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\ca2_ps_311\Lib\site-packages\pyarrow\compute.py", line 242, in wrapper
    return func.call(args, None, memory_pool)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow\_compute.pyx", line 367, in pyarrow._compute.Function.call
  File "pyarrow\error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow\error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Function 'equal' has no kernel matching input types (int64, bool)
>>>

There's also no fallback if the scalar is an arrow scalar:

>>> pd.Series(["abc"], dtype=pd.ArrowDtype(pa.string())) == pa.scalar(True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\pd_202_env\Lib\site-packages\pandas\core\ops\common.py", line 81, in new_method
    return method(self, other)
           ^^^^^^^^^^^^^^^^^^^
  File "D:\pd_202_env\Lib\site-packages\pandas\core\arraylike.py", line 40, in __eq__
    return self._cmp_method(other, operator.eq)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\pd_202_env\Lib\site-packages\pandas\core\series.py", line 6096, in _cmp_method
    res_values = ops.comparison_op(lvalues, rvalues, op)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\pd_202_env\Lib\site-packages\pandas\core\ops\array_ops.py", line 279, in comparison_op
    res_values = op(lvalues, rvalues)
                 ^^^^^^^^^^^^^^^^^^^^
  File "D:\pd_202_env\Lib\site-packages\pandas\core\ops\common.py", line 81, in new_method
    return method(self, other)
           ^^^^^^^^^^^^^^^^^^^
  File "D:\pd_202_env\Lib\site-packages\pandas\core\arraylike.py", line 40, in __eq__
    return self._cmp_method(other, operator.eq)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\pd_202_env\Lib\site-packages\pandas\core\arrays\arrow\array.py", line 484, in _cmp_method
    raise NotImplementedError(
NotImplementedError: eq not implemented for <class 'pyarrow.lib.BooleanScalar'>
>>>

Looks like this is because is_scalar in the fallback returns false for a pyarrow scalar.

[mroeschke]

I think the fallback was inherited from StringArrowArray that was meant to be more permissive for strings, but I am not sure of the origins of why it was decided to be more permissive

[jorisvandenbossche]

FWIW, we now catch any error assuming that the comparison doesn't work and then (for ==) return False. So the below no longer gives an error, but it is still does give an inconsistent result for the comparison with scalar vs series:

In [5]: i = pd.Series([1], dtype="int64[pyarrow]")

In [6]: i == True
Out[6]: 
0    True
dtype: bool[pyarrow]

In [7]: b = pd.Series([True], dtype="bool[pyarrow]")

In [8]: i == b
Out[8]: 
0    False
dtype: bool[pyarrow]

Under the hood, we should probably try to cast the RHS to the dtype of self before trying the pyarrow compute kernel (because pyarrow allows not all casts in operations), like:

In [10]: i == pd.Series([True], dtype="bool[pyarrow]").astype("int64[pyarrow]")
Out[10]: 
0    True
dtype: bool[pyarrow]