`.loc[] = pd.Series({...})` segmentation fault

[kuraga]

Code Sample, a copy-pastable example if possible

test.csv:

"A","B"
"Давид Юрченко","Tosno"
"Евгений Чернов","Tosno"
"Раде Дугалич","Tosno"
"Виталий Шахов","Tosno"
"Аслан Дудиев","Tosno"
"Вагиз Галиулин","Tosno"
"Нуну Роша","Tosno"
"Георгий Мелкадзе","Tosno"

test.py:

import pandas as pd

df = pd.read_csv('test.csv', sep=',').set_index('A', drop=False)

for s in [
'Евгений Чернов',
'Раде Дугалич',
'Виталий Шахов',
'Аслан Дудиев',
'Александр Карницкий',
'Вагиз Галиулин',
'Нолан Ру'
    ]:
    df.loc[s] = pd.Series({ 'A': 'value' })

Problem description

This code sometimes causes a segmentation fault.

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.6.5.final.0 python-bits: 64 OS: Linux OS-release: 4.14.39-calculate machine: x86_64 processor: Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz byteorder: little LC_ALL: None LANG: ru_RU.utf8 LOCALE: ru_RU.UTF-8

pandas: 0.23.0
pytest: None
pip: 9.0.3
setuptools: 39.0.1
Cython: None
numpy: 1.14.3
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 6.4.0
sphinx: None
patsy: 0.5.0
dateutil: 2.7.3
pytz: 2018.4
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.2.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.6.0
html5lib: 0.9999999
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

cc @TomAugspurger

[phofl]

Can not reproduce this. Someone else?

[kuraga]

@phofl in what environment?

[phofl]

Master

[kuraga]

Reproduced in 1.1.3: https://gist.github.com/kuraga/9aa46967ee7ab3fac93214331a404356 on 3.7.9.

[kuraga]

Created https://bugs.python.org/issue42372.

[kuraga]

Created https://bugs.python.org/issue42372.

Christian Heimes said:

The problem is probably a bug in Pandas or another C extension. The stacktrace does not contain enough information to see the type and value of v and w.

[mzeitlin11]

I can reproduce on current master with macOS. Based on the backtrace, segfault looks caused by some issue in our hash table logic

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xc8)
  * frame #0: 0x0000000100095f2e python`PyObject_RichCompare + 94
    frame #1: 0x0000000100096221 python`PyObject_RichCompareBool + 17
    frame #2: 0x0000000170ebabd5 hashtable.cpython-38-darwin.so`pyobject_cmp + 181
    frame #3: 0x0000000170eba062 hashtable.cpython-38-darwin.so`kh_get_pymap + 146
    frame #4: 0x0000000170f40977 hashtable.cpython-38-darwin.so`__pyx_f_6pandas_5_libs_9hashtable_17PyObjectHashTable_get_item + 871
    frame #5: 0x0000000170f8b731 hashtable.cpython-38-darwin.so`__pyx_pf_6pandas_5_libs_9hashtable_17PyObjectHashTable_12get_item + 81
    frame #6: 0x0000000170f83dc5 hashtable.cpython-38-darwin.so`__pyx_pw_6pandas_5_libs_9hashtable_17PyObjectHashTable_13get_item + 37
...

Contributions welcome to look into this further - would be helpful to know if the example can be shrunk - eg is the read_csv or set_index causing a somehow broken df or can this be reproduced purely through indexing?

[roberthdevries]

This looks like another use after free bug.
In khash_python.h on line 248 it says:

	int result = PyObject_RichCompareBool(a, b, Py_EQ);

Printing the contents of a and b reveals the following:

(gdb) print *a
$3 = {ob_refcnt = 140659575290544, ob_type = 0x7feddc03fd70}
(gdb) print *b
$4 = {ob_refcnt = 14, ob_type = 0x55a33d18af60 <PyUnicode_Type>}

It is obvious (to me at least) that object a has been deallocated some time before this line. This is typically caused by a Py_DECREF call.