Repeated unpickling of DataFrame results in different data each time

[chrisjbillington]

I have a MultiIndexed DataFrame which, when pickled and unpickled, results in some of the data changing.

Specifically, some of its columns contain data which is a tuple of integers, and these integers are different each unpickling. Here's a DataFrame which shows this problem:

http://bec.physics.monash.edu/docs/dataframe.pickle

It was pickled with pandas 0.9.0.dev-667220a (current git-master) and Python 2.7.3 64 bit.

Try the following to reproduce:

import pickle
df = pickle.load(open('dataframe.pickle'))
print df['top','roi0']

Repeating this gives me different data each time.

Which integers I get seems slightly platform dependent. The tuples contain four integers, and on Linux most rows (but not all) in the DataFrame end up getting three zeros and one nonzero value, and all the nonzero values are somewhat similar, and in the tens of millions. On Windows the data seems to be random integers that are less regular, and without most of them being zeros. The data that was pickled should have four different values, between 0 and 2048 or so (these are definitions of region of interest rects on a CCD camera).

I am unable to reproduce the problem with non-herarchical index DataFrames.

Any clue as to why this might be, or any suggested workarounds? We use different serialisation formats for on-disk persistence, but we've been using pickle for slinging dataframes across networks.

[chrisjbillington]

Also can't seem to reproduce with a heirarchical indexed (both columns and rows) dataframe, like this:

import pandas
import pickle
from dateutil.parser import parse

import numpy

data = {('x','y'): [(1,2,3,4),(4,5,6,7),(7,8,9,10)],
        ('sequence',''): [parse('20120920120000'), parse('20120920120000'), parse('20120920130000')],
        ('run time',''): [parse('20120920121000'), parse('20120920122000'), parse('20120920131000')]}

# convert to numpy.int32s:
for i, data_tuple in enumerate(data['x','y'][:]):
    data['x','y'][i] = tuple(numpy.int32(n) for n in data_tuple)

index = pandas.MultiIndex.from_tuples(sorted(data.keys()))
df = pandas.DataFrame(data,columns=index)
df.set_index(['sequence','run time'], inplace=True, drop=False)

df2 = pickle.loads(pickle.dumps(df))

print df['x','y']
print df2['x','y']

I'm not sure what it is about my dataframe that distinguishes it from this test-case. Both use tuples of numpy.int32s and have a datetime multiindex. They're pretty similar really, not sure why one shows the bug and the other doesn't.

[wesm]

Well, this isn't great. I'm looking at it now and will try to figure out what's wrong

[wesm]

Well. You aren't going to like it:

See this structured array:

arr = np.array([(0, 0, 811532298, 687868416), (0, 0, 58785802, 0),
       (0, 0, 58064906, 0), (0, 0, 58130442, 0), (0, 0, 58523658, 0),
       (0, 0, 58130442, 0), (0, 0, 58195978, 0), (0, 0, 58064906, 0),
       (0, 0, 58130442, 0), (0, 0, 58589194, 0), (0, 0, 58195978, 0),
       (0, 0, 58064906, 0), (0, 0, 58720266, 0), (0, 0, 58589194, 0),
       (0, 0, 1443954698, 32692), (0, 0, 1443954698, 32692),
       (0, 0, 1443954698, 32692), (0, 0, 1443954698, 32692),
       (0, 0, 1443954698, 32692), (0, 0, 1443954698, 32692),
       (0, 0, 1443954698, 32692), (0, 0, 1443954698, 32692),
       (0, 0, 1443954698, 32692), (0, 0, 1443954698, 32692),
       (0, 0, 1443954698, 32692), (0, 0, 1443954698, 32692),
       (0, 0, 1443954698, 32692), (0, 0, 1443954698, 32692),
       (0, 0, 1443954698, 32692), (0, 0, 1443954698, 32692),
       (0, 0, 1443954698, 32692), (0, 0, 1443954698, 32692),
       (0, 0, 1443954698, 32692), (0, 0, 1443954698, 32692),
       (0, 0, 1443954698, 32692), (0, 0, 1443954698, 32692),
       (0, 0, 1443954698, 32692), (0, 0, 1443954698, 32692),
       (0, 0, 1443954698, 32692), (0, 0, 1443954698, 32692),
       (0, 0, 1443954698, 32692), (0, 0, 1443954698, 32692),
       (0, 0, 1443954698, 32692), (0, 0, 1443954698, 32692),
       (0, 0, 1443954698, 32692), (0, 0, 1443954698, 32692),
       (0, 0, 1443954698, 32692), (0, 0, 1443954698, 32692),
       (0, 0, 1443954698, 32692), (0, 0, 1443954698, 32692),
       (0, 0, 1443954698, 32692), (0, 0, 1443954698, 32692),
       (0, 0, 1443954698, 32692), (0, 0, 1443954698, 32692),
       (0, 0, 1443954698, 32692), (0, 0, 1443954698, 32692),
       (0, 0, 1443954698, 32692), (0, 0, 1443954698, 32692),
       (0, 0, 1443954698, 32692), (0, 0, 1443954698, 32692),
       (0, 0, 1443954698, 32692), (0, 0, 1443954698, 32692)], 
      dtype=[('left', '<i4'), ('top', '<i4'), ('right', '<i4'), ('bottom', '<i4')])

If you convert this to an array of numpy.void objects as you have in your case, it won't survive the pickle roundtrip:

In [151]: pickle.loads(pickle.dumps(np.array(list(arr), dtype=object)))
Out[151]: 
array([(0, 0, 56421320, 0), (1443987992, 32692, 1443987992, 32692),
       (1443988136, 32692, 1443988136, 32692), (0, 0, 60521864, 0),
       (0, 0, 60519736, 0), (0, 0, 60521808, 0), (0, 0, 58343584, 0),
       (0, 0, 58344424, 0), (0, 0, 58344144, 0), (0, 0, 58344648, 0),
       (0, 0, 872947192, 32692), (0, 0, 59774320, 0),
       (0, 0, 872947136, 32692), (0, 0, 872947640, 32692),
       (0, 0, 58003984, 0), (0, 0, 872947528, 32692),
       (0, 0, 872947304, 32692), (1443988024, 32692, 60157744, 0),
       (1443987992, 32692, 1443987992, 32692),
       (1443987992, 32692, 1443987992, 32692), (0, 0, 872947416, 32692),
       (0, 0, 872947584, 32692), (1443987992, 32692, 1443987992, 32692),
       (0, 0, 873103464, 32692), (0, 0, 873103520, 32692),
       (1443987992, 32692, 1443987992, 32692), (0, 0, 873103632, 32692),
       (0, 0, 873103688, 32692), (0, 0, 55044944, 0),
       (0, 0, 873103800, 32692), (0, 0, 873103856, 32692),
       (0, 0, 58536576, 0), (0, 0, 873103968, 32692),
       (0, 0, 873104024, 32692), (0, 0, 59227184, 0),
       (0, 0, 873104136, 32692), (0, 0, 1443988088, 32692),
       (0, 0, 873104248, 32692), (0, 0, 873104304, 32692),
       (0, 0, 873104360, 32692), (0, 0, 873104416, 32692),
       (0, 0, 873104472, 32692), (0, 0, 873104528, 32692),
       (0, 0, 873104584, 32692), (0, 0, 873104640, 32692),
       (0, 0, 873104696, 32692), (0, 0, 873104752, 32692),
       (0, 0, 58699600, 0), (0, 0, 873104864, 32692),
       (0, 0, 873104920, 32692), (0, 0, 873104976, 32692),
       (0, 0, 873105032, 32692), (0, 0, 873105088, 32692),
       (0, 0, 873105144, 32692), (0, 0, 873105200, 32692),
       (0, 0, 873105256, 32692), (0, 0, 873105312, 32692),
       (0, 0, 873105368, 32692), (0, 0, 873105424, 32692),
       (0, 0, 873105480, 32692), (0, 0, 873105536, 32692),
       (0, 0, 873105592, 32692)], dtype=object)

In [152]: pickle.loads(pickle.dumps(np.array(list(arr), dtype=object)))
Out[152]: 
array([(0, 0, 56418800, 0), (0, 0, 55244256, 0),
       (1443987992, 32692, 1443987992, 32692), (0, 0, 60521864, 0),
       (1443987992, 32692, 1443987992, 32692), (0, 0, 60521808, 0),
       (1443987992, 32692, 1443987992, 32692), (0, 0, 58343584, 0),
       (1443987992, 32692, 1443987992, 32692),
       (1443988024, 32692, 58917296, 0),
       (1443988024, 32692, 1443988024, 32692), (0, 0, 872947472, 32692),
       (0, 0, 872947360, 32692), (0, 0, 872947640, 32692),
       (0, 0, 872947416, 32692), (0, 0, 872947136, 32692),
       (0, 0, 872947024, 32692), (1443988024, 32692, 58179680, 0),
       (1443987992, 32692, 1443987992, 32692),
       (1443987992, 32692, 1443987992, 32692), (0, 0, 872947304, 32692),
       (1443987992, 32692, 1443987992, 32692), (0, 0, 873104864, 32692),
       (1443987992, 32692, 1443987992, 32692), (0, 0, 873103856, 32692),
       (1443987992, 32692, 1443987992, 32692), (0, 0, 1443988088, 32692),
       (0, 0, 873105424, 32692), (0, 0, 873105200, 32692),
       (0, 0, 873105592, 32692), (0, 0, 873105032, 32692),
       (0, 0, 873105144, 32692), (0, 0, 873105256, 32692),
       (0, 0, 873104192, 32692), (0, 0, 873103688, 32692),
       (0, 0, 873104528, 32692), (0, 0, 873105368, 32692),
       (0, 0, 873103800, 32692), (0, 0, 873105088, 32692),
       (0, 0, 873105648, 32692), (0, 0, 873104024, 32692),
       (0, 0, 873104752, 32692), (0, 0, 873104248, 32692),
       (0, 0, 873104304, 32692), (0, 0, 873103968, 32692),
       (0, 0, 873104360, 32692), (0, 0, 59774848, 0),
       (0, 0, 1443988136, 32692), (0, 0, 873103912, 32692),
       (0, 0, 873103744, 32692), (0, 0, 873103408, 32692),
       (0, 0, 873103520, 32692), (0, 0, 873103576, 32692),
       (0, 0, 873104080, 32692), (0, 0, 873105312, 32692),
       (0, 0, 873104920, 32692), (0, 0, 873104136, 32692),
       (0, 0, 873104696, 32692), (0, 0, 873104976, 32692),
       (0, 0, 873104472, 32692), (0, 0, 873103464, 32692),
       (0, 0, 873104640, 32692)], dtype=object)

Basically, pickling numpy.void objects, at least in NumPy 1.6.1, is horribly broken. I honestly don't know how it could be screwed up because all you have to do is serialize the bytes and the dtype:

In [190]: arr[-1]
Out[190]: (0, 0, 1443954698, 32692)

In [191]: type(arr[-1])
Out[191]: numpy.void

In [192]: arr[-1].dtype
Out[192]: dtype([('left', '<i4'), ('top', '<i4'), ('right', '<i4'), ('bottom', '<i4')])

In [193]: arr[-1].tostring()
Out[193]: '\x00\x00\x00\x00\x00\x00\x00\x00\n\x00\x11V\xb4\x7f\x00\x00'

But I digress. Not sure what to tell you

[chrisjbillington]

Thanks for your help Wes, I didn't realise the objects were a numpy.void type, and hadn't heard of that bug. It does seem pretty silly that the bug still exists after four years, but it doesn't seem to have gotten much attention, maybe I'll try to draw some to it.

I'm just going to workaround it for the moment by converting any numpy.void types found during construction of the DataFrame to a custom class which implements mostly the same functionality, ie named and ordered fields. My code now looks like this, for anyone who's interested (For context, each row in the DataFrame comes from a dictionary which in turn comes from attributes in a HDF5 file. Each row in the DataFrame represents one HDF5 file.):

class Fields(object):
    """A workaraound for the fact that numpy.void objects cannot be
    correctly unpickled (a bug in numpy) and therefore cannot be sent
    to other processes over the network. This class implements the same
    functionality mostly. Basically the thing you get back looks like a
    tuple but can be indexed with either names of the fields or integers,
    much like a single row of a numpy structured array. Whenever this
    module encounters a numpy.void type when reading attributes from a
    HDF5 file, it converts it to one of these."""

    def __init__(self, data):
        self.data_by_name = {}
        self.data_by_index = tuple(data)
        self.dtype = data.dtype
        for name in data.dtype.names:
            self.data_by_name[name] = data[name]

    def __getitem__(self, key):
        if isinstance(key,int):
            return self.data_by_index[key]
        else:
            return self.data_by_name[key]

    def __repr__(self):
        return str(self.data_by_index)

def workaround_numpy_void_bug(dictionary):
    # numpy.void objects undergo data corruption when pickled and
    # unpickled.  h5py returns numpy.void objects for attributes
    # which are its 'compound' datatype.  We'll convert any we find to our
    # home-cooked Fields class (defined above), which provides mostly
    # the same functionality. This will be removed if and when numpy fix their bug.
    for key, value in dictionary.items():
        if isinstance(value, void):
            dictionary[key] = Fields(value)

The clients always have access to this module, so they can unpickle just fine.