Create efficient binary storage format alternative to pickle

[wesm]

Ideally it should support compression! Possibly using blosc or some other method

[mcobzarenco]

As a first order approximation, what do you think about:

import pickle, bz2

def pickle_compress(obj, path):
    pickle.dump(obj, bz2.BZ2File(path, 'w'), pickle.HIGHEST_PROTOCOL)

def unpickle_compress(obj, path):
    return pickle.load(bz2.BZ2File(path, 'r'))

[wesm]

Unfortunately, this doesn't solve the "pickle problem" (i.e. that classes can't move to different modules).

[dalejung]

@wesm have you considered making HDF5 the preferred backend? You would get compression baked in. Plus, the random access support would make on-disk dataframe operations possible.

[wesm]

I have, but the main problem is the deserialization speed for lots of small objects (this turns out to be a fairly important use case to a lot of users). I think using msgpack along with snappy or blosc (which uses the fastlz algorithm) for fast in-memory compression might be a good way to go

I should point out that using HDF5 (or at least PyTables) adds quite a bit of overhead for loading small objects

[dalejung]

When you talk about loading many small objects, what do you mean exactly? I've definitely run into a lot of performance cul-de-sacs when it comes to pytables, so I'm not sure which one you're referring to :/

I've come to having a DatetimeIndex in memory and translating the bool arrays to list of slices since pytables dies on long list of indexes. Really quick but bypassing most pytables features like in-kernel searches.

[wesm]

As a benchmark, try loading 1000 Series objects each containing 100 values and random string indexes with 10 or fewer characters. So we're talking roughly 8K of data per Series

related #3151

The metadata discussion in #2485 reached an impasse about having metadata
propegate through data ops, it still makes sense to allow a metadata descriptor
to be serialized with an object and available initially upon deserialization for things
like date of measurements, source of data information, etc'.

The pickle code in pandas is not my favorite bit, would be nice to tackle it
as part of a new binary format if/when though.

[jreback]

I think in 0.12 we could add a generic .meta that attaches to all objects and can propogate it. Easy to save/recreate in HDF (see http://pandas.pydata.org/pandas-docs/dev/cookbook.html#hdfstore) and pickle, csv/excel would prob need some work to do that though. Adding direct .name also should be straightforward (on frames/panel).

How do you mean propogate? you have a way to do #2485 in a clean way across operations? or
just adding it to the current pickle serialization?

[jreback]

pickle/hdf is easy, I would do it like name propogates in series, essentially its an add to the constructor (and maybe move name and make it a property of meta). I think can prop across most common operations.

of course not exactly sure what to do in a case like this:

df1.meta = dict(foo = 1)
df2.meta = dict(bar = 1)

df = df1 + df2

df.meta = dict(foo = 1, bar = 1)

and maybe a warning for clobbering? could also

#2485 (possibly the longest discussion ever?) convinced me there be dragons on the propagation
front, too much complexity for something that really doesn't mesh with the data model.

Adding metadata to store/load only is indeed fairly straightforward, and useful for archiving. That's
why I brought it up.