Description
Pandas version checks
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
from pandas._libs import hashtable as ht
class testkey:
def __init__(self, val):
self._val = val;
self.throw = False
def __hash__(self):
if self.throw:
import sys
print("exiting")
sys.exit(0)
return self._val.__hash__()
def __eq__(self, other):
return self._val.__eq__(other._val)
def __repr__(self):
return self._val.__repr__()
y1 = testkey("hello")
y2 = testkey("hello2")
x = ht.PyObjectHashTable()
x.set_item(y1, 123)
x.set_item(y2, 456)
print(f"val1: {x.get_item(y2)}")
y2.throw = True
try:
print(f"val2: {x.get_item(y2)}")
except KeyError:
print("missed key")
y2.throw = False
print(f"val3: {x.get_item(y2)}")Prints:
val1: 456
exiting
missed key
val3: 456
or (I guess this depends on some random hash value used when the __hash__ throws)
val1: 456
exiting
val2: 456
val3: 456
While I would expect it to print
val1: 456
exiting
Issue Description
I'm running to an issue in prod where I have a custom signal handler for SIGINT, which, when raised, will raise a SystemExit exception. Now, sometimes this SystemExit does not result in the application stopping, and the application keeps running.
Debugging it so far, it seems to be this always happens when we're running some pandas code. It's a bit hard to debug, but I think the issue happens due to PyObjectHashTable's handling of exceptions during __hash__ and __eq__ calls. It looks like these are swallowed (or replaced by another exception) under some conditions. I'm not too familiar with the Pandas code, so it was a bit hard to follow where it exactly goes wrong. The example attached, however, tries to show this behavior when we throw in the __hash__ during PyObjectHashTable.get_item(). In this case the SystemExit exception gets replaced by a KeyError.
Expected Behavior
I expect exceptions thrown during __hash__ or __eq__ to not be swallowed.
Installed Versions
>>> import pandas
pand>>> pandas.show_versions(
... )
INSTALLED VERSIONS
------------------
commit : fd3f57170aa1af588ba877e8e28c158a20a4886d
python : 3.9.6.final.0
python-bits : 64
OS : Darwin
OS-release : 22.1.0
Version : Darwin Kernel Version 22.1.0: Sun Oct 9 20:19:12 PDT 2022; root:xnu-8792.41.9~3/RELEASE_ARM64_T6020
machine : arm64
processor : arm
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.UTF-8
pandas : 2.2.0
numpy : 1.26.3
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : 58.0.4
pip : 23.3.2
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 15.0.0
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.4
qtpy : None
pyqt5 : None