BUG: SIGBUS in test_append_frame_column_oriented, test_select_filter_corner on arm

[thesamesam]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

Run the pandas-2.0.3 testsuite on armv7.

Issue Description

See also #54391.

The pandas/tests/io/pytables/test_append.py::test_append_frame_column_oriented and pandas/tests/io/pytables/test_store.py::test_select_filter_corner tests SIGBUS for me on arm (arm64 host, armv7 chroot).

When building with UBSAN (-fsanitize=undefined), I get the following for pandas/tests/io/pytables/test_append.py::test_append_frame_column_oriented:

pandas/_libs/algos.c:172405:173: runtime error: load of misaligned address 0x056a1dd9 for type 'const __pyx_t_5numpy_float64_t', which requires 8 byte alignment
0x056a1dd9: note: pointer points here
 00 00 00  41 07 2c e2 69 91 c3 ef  3f 3b 18 13 11 66 2e e9  bf 93 05 1d b4 60 cd 03  40 34 84 74 9e
              ^
Fatal Python error: Bus error

Thread 0xf6dcb420 (most recent call first):
  File "/usr/lib/python3.11/site-packages/execnet/gateway_base.py", line 474 in read
  File "/usr/lib/python3.11/site-packages/execnet/gateway_base.py", line 507 in from_io
  File "/usr/lib/python3.11/site-packages/execnet/gateway_base.py", line 1049 in _thread_receiver
  File "/usr/lib/python3.11/site-packages/execnet/gateway_base.py", line 296 in run
  File "/usr/lib/python3.11/site-packages/execnet/gateway_base.py", line 361 in _perform_spawn

Current thread 0xf7bcc020 (most recent call first):
  File "/var/tmp/portage/dev-python/pandas-2.0.3/work/pandas-2.0.3-python3_11/install/usr/lib/python3.11/site-packages/pandas/core/array_algos/take.py", line 162 in _take_nd_ndarray
  File "/var/tmp/portage/dev-python/pandas-2.0.3/work/pandas-2.0.3-python3_11/install/usr/lib/python3.11/site-packages/pandas/core/array_algos/take.py", line 117 in take_nd
  File "/var/tmp/portage/dev-python/pandas-2.0.3/work/pandas-2.0.3-python3_11/install/usr/lib/python3.11/site-packages/pandas/core/internals/blocks.py", line 945 in take_nd
  File "/var/tmp/portage/dev-python/pandas-2.0.3/work/pandas-2.0.3-python3_11/install/usr/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 748 in <listcomp>
  File "/var/tmp/portage/dev-python/pandas-2.0.3/work/pandas-2.0.3-python3_11/install/usr/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 747 in reindex_indexer
  File "/var/tmp/portage/dev-python/pandas-2.0.3/work/pandas-2.0.3-python3_11/install/usr/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 963 in take
  File "/var/tmp/portage/dev-python/pandas-2.0.3/work/pandas-2.0.3-python3_11/install/usr/lib/python3.11/site-packages/pandas/core/generic.py", line 3932 in _take
  File "/var/tmp/portage/dev-python/pandas-2.0.3/work/pandas-2.0.3-python3_11/install/usr/lib/python3.11/site-packages/pandas/core/generic.py", line 3948 in _take_with_is_copy
  File "/var/tmp/portage/dev-python/pandas-2.0.3/work/pandas-2.0.3-python3_11/install/usr/lib/python3.11/site-packages/pandas/core/indexing.py", line 1123 in _getbool_axis
  File "/var/tmp/portage/dev-python/pandas-2.0.3/work/pandas-2.0.3-python3_11/install/usr/lib/python3.11/site-packages/pandas/core/indexing.py", line 1325 in _getitem_axis
  File "/var/tmp/portage/dev-python/pandas-2.0.3/work/pandas-2.0.3-python3_11/install/usr/lib/python3.11/site-packages/pandas/core/indexing.py", line 1103 in __getitem__
  File "/var/tmp/portage/dev-python/pandas-2.0.3/work/pandas-2.0.3-python3_11/install/usr/lib/python3.11/site-packages/pandas/io/pytables.py", line 4103 in process_filter
  File "/var/tmp/portage/dev-python/pandas-2.0.3/work/pandas-2.0.3-python3_11/install/usr/lib/python3.11/site-packages/pandas/io/pytables.py", line 4123 in process_axes
  File "/var/tmp/portage/dev-python/pandas-2.0.3/work/pandas-2.0.3-python3_11/install/usr/lib/python3.11/site-packages/pandas/io/pytables.py", line 4579 in read
  File "/var/tmp/portage/dev-python/pandas-2.0.3/work/pandas-2.0.3-python3_11/install/usr/lib/python3.11/site-packages/pandas/io/pytables.py", line 850 in func
  File "/var/tmp/portage/dev-python/pandas-2.0.3/work/pandas-2.0.3-python3_11/install/usr/lib/python3.11/site-packages/pandas/io/pytables.py", line 1937 in get_result
  File "/var/tmp/portage/dev-python/pandas-2.0.3/work/pandas-2.0.3-python3_11/install/usr/lib/python3.11/site-packages/pandas/io/pytables.py", line 866 in select
  File "/var/tmp/portage/dev-python/pandas-2.0.3/work/pandas-2.0.3-python3_11/install/usr/lib/python3.11/site-packages/pandas/tests/io/pytables/test_append.py", line 285 in test_append_frame_column_oriented
[...]

and for pandas/tests/io/pytables/test_store.py::test_select_filter_corner:

 pandas/_libs/algos.c:172405:17
3: runtime error: load of misaligned address 0x0708f763 for type 'const __pyx_t_5numpy_float64_t', which requires 8 byte alignment
0x0708f763: note: pointer points here
 00  20 20 30 7f 17 9e 9f b3  20 e6 3f 65 44 21 ce 7b  f1 ea bf 21 62 1c ea 6a  4f e6 bf 4f ab 7c 7f
              ^
Fatal Python error: Bus error

Thread 0xf6ffb420 (most recent call first):
  File "/usr/lib/python3.11/site-packages/execnet/gateway_base.py", line 474 in read
  File "/usr/lib/python3.11/site-packages/execnet/gateway_base.py", line 507 in from_io
  File "/usr/lib/python3.11/site-packages/execnet/gateway_base.py", line 1049 in _thread_receiver
  File "/usr/lib/python3.11/site-packages/execnet/gateway_base.py", line 296 in run
  File "/usr/lib/python3.11/site-packages/execnet/gateway_base.py", line 361 in _perform_spawn
[...]

Expected Behavior

All tests pass.

Installed Versions

# python3 -c 'import pandas as pd; pd.show_versions()' /usr/lib/python3.11/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils. warnings.warn("Setuptools is replacing distutils.")

INSTALLED VERSIONS

commit : 0f43794
python : 3.11.4.final.0
python-bits : 32
OS : Linux
OS-release : 5.15.117-gentoo-dist
Version : #1 SMP Wed Jun 14 13:14:49 -00 2023
machine : armv8l
processor : ARMv8 Processor rev 1 (v8l)
byteorder : little
LC_ALL : None
LANG : C.UTF8
LOCALE : en_US.UTF-8

pandas : 2.0.3
numpy : 1.25.2
pytz : 2023.3
dateutil : 2.8.2
setuptools : 68.0.0
pip : None
Cython : 3.0.0
pytest : 7.4.0
hypothesis : 6.82.0
sphinx : None
blosc : 1.11.1
feather : None
xlsxwriter : 3.1.2
lxml.etree : 4.9.3
html5lib : 1.1
pymysql : 1.4.6
psycopg2 : 2.9.4
jinja2 : 3.1.2
IPython : None
pandas_datareader: None
bs4 : 4.12.2
bottleneck : 1.3.7
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.7.2
numba : None
numexpr : 2.8.4
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.11.1
snappy : None
sqlalchemy : 2.0.19
tables : 3.8.0
tabulate : 0.9.0
xarray : 2023.7.0
xlrd : 2.0.1
zstandard : None
tzdata : None
qtpy : None
pyqt5 : None

[thesamesam]

I'm pretty sure tests passed on arm fairly recently, but if I try e.g. pandas-1.4.0 (Jan 2022), it fails now, so I suspect GCC got better at optimising instead. So, unlike the other bug, I can't give you the commit which caused it.

[thesamesam]

It blows up for me at https://github.com/pandas-dev/pandas/blob/main/pandas/_libs/algos_take_helper.pxi.in#L219 with e.g.

__pyx_t_8 = __pyx_v_k;
      __pyx_t_9 = __pyx_t_8;
      for (__pyx_t_10 = 0; __pyx_t_10 < __pyx_t_9; __pyx_t_10+=1) {
        __pyx_v_j = __pyx_t_10;

        /* "pandas/_libs/algos_take_helper.pxi":1998
 *         else:
 *             for j in range(k):
 *                 out[i, j] = values[idx, j]             # <<<<<<<<<<<<<<
 *
 *
 */
        __pyx_t_7 = __pyx_v_idx;
        __pyx_t_11 = __pyx_v_j;
        __pyx_t_12 = __pyx_v_i;
        __pyx_t_13 = __pyx_v_j;
        *((__pyx_t_5numpy_float64_t *) ( /* dim=1 */ (( /* dim=0 */ (__pyx_v_out.data + __pyx_t_12 * __pyx_v_out.strides[0]) ) + __pyx_t_13 * __pyx_v_out.strides[1]) )) = (*((__pyx_t_5numpy_float64_t const  *) ( /* dim=1 */ (( /* dim=0 */ (__pyx_v_values.data + __pyx_t_7 * __pyx_v_values.strides[0]) ) + __pyx_t_11 * __pyx_v_values.strides[1]) )));
      }

I don't know enough about cython or numpy to say why it's generating the load/store like this and what's to blame, though.

Advice very much welcome and required at this point, as I'm stuck.

[rebecca-palmer]

I'm pretty sure tests passed on arm fairly recently, but if I try e.g. pandas-1.4.0 (Jan 2022), it fails now, so I suspect GCC got better at optimising instead. So, unlike the other bug, I can't give you the commit which caused it.

This bug is much older than that - it's been known in Debian since pandas 0.16, but since the response was to disable the tests in question, we don't know whether it's always existed since then. We also don't know whether it's a pandas bug or a pytables bug.

I vaguely remember that unaligned access is a crash on only some ARM systems (depending on hardware and/or whether a "silently (but slowly) fix up unaligned access attempts" kernel setting is enabled).