Seqfault on creation of dataframe with np.empty_like

[flatberg]

This code segfaults:

import numpy as np
import pandas as pd
df = pd.DataFrame(np.random.randn(3, 3), index='A B C'.split())
pd.DataFrame(np.empty_like(df.index))

INSTALLED VERSIONS

commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 3.13.0-91-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.18.1
nose: 1.3.7
pip: 8.1.2
setuptools: 23.0.0
Cython: 0.24
numpy: 1.11.1
scipy: 0.17.1
statsmodels: 0.6.1
xarray: 0.7.2
IPython: 5.0.0
sphinx: None
patsy: 0.4.1
dateutil: 2.3
pytz: 2016.4
blosc: None
bottleneck: 1.1.0
tables: 3.2.2
numexpr: 2.6.0
matplotlib: 1.5.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.4.1
html5lib: 0.999
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
boto: None
pandas_datareader: None

[jreback]

In [1]: pd.DataFrame(np.empty_like(np.array([1,2])))
Out[1]:
     0
0  109
1    0

In [2]: pd.DataFrame(np.empty_like(np.array(['a', 'b'])))
Out[2]:
  0
0
1

# setfault
In [3]: pd.DataFrame(np.empty_like(Index(['a', 'b'])))
Out[3]:

not really sure what np.empty_like actually does under the hood.

cc @charris
cc @gfyoung

[gfyoung]

I suspect it is a case of memory corruption on the numpy end when creating a None for the object dtype, as evidenced below:

>>> import numpy as np
>>> import pandas as pd
>>> arr = np.empty_like(pd.Index(['a', 'b']))
>>> arr
array([None, None], dtype=object)
>>> arr2 = np.array([None, None], dtype=object)
>>> arr == arr2
True
>>> arr.data.tobytes()    # segfault
b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
>>> arr2.data.tobytes()   # no segfault
b'0\x91\xad\x8b\xd8\x7f\x00\x000\x91\xad\x8b\xd8\x7f\x00\x00'

You can see here that the data is not corrupted when you have integers:

>>> import numpy as np
>>> import pandas as pd
>>> arr = np.empty_like(pd.Index([1, 2]))
>>> arr
array([140568033367712, 140568033367744], dtype=int64)
>>> arr2 = np.array([140568033367712, 140568033367744], dtype=np.int64)
>>> arr == arr2
True
>>> arr.data.tobytes()    # no segfault
b'\xa0\x16\xb2\x8b\xd8\x7f\x00\x00\xc0\x16\xb2\x8b\xd8\x7f\x00\x00'
>>> arr2.data.tobytes()   # no segfault
b'\xa0\x16\xb2\x8b\xd8\x7f\x00\x00\xc0\x16\xb2\x8b\xd8\x7f\x00\x00'

Also, you don't need an Index to trigger the segfault:

>>> import numpy as np
>>> import pandas as pd
>>> pd.DataFrame(np.empty_like([None]))

[bashtage]

You can do more wierd tricks with this:

np.empty_like(np.array([None, None], dtype=object)).data.tobytes()
Out[41]: b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'

np.empty_like(np.array([None, None], dtype=object)).astype(np.int64).data.tobytes()
Out[42]: b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'

np.empty_like(np.array([None, None], dtype=object)).astype(np.int64).astype(np.object).data.tobytes()
Out[43]: b'\xd0\x01\x9ac\x00\x00\x00\x00\xd0\x01\x9ac\x00\x00\x00\x00'

This looks like a NumPy bug.

I suspect that an object array initialized using None should contain the value of id(None) if correct. 0-filled "Nones" don't make any sense, and seem to an incorrect treatment of NumPy.

>>> np.array([None]).data.tobytes()[::-1].hex()
'0000000063979420'

>>> hex(id(None))
'0x63979420'

[jreback]

agree with @bashtage here - some memory aliasing going on

want to open a numpy side issue?

[gfyoung]

@jreback : I opened a PR now in numpy that should fix the issue. However, even if it doesn't get merged, technically, this issue could be closed since the bug is on the numpy side.

[jreback]

running this in the debugger actually works. I think there is a reference issue to the Nones; an element-wise copy would fix it I suppose (from pandas perspective). but yes this is a numpy-bug.

[bashtage]

It is probably, technically, a Pandas bug. NumPy is returns an empty array filled with NULL. Pandas doesn't seems to know what to do when an object array contains NULLs. This said, it would be better if NumPy addressed this by never returning object arrays containing NULLs, and in instead returned the address of None.