COMPAT: reading generic PyTables Table format fails with sub-selection

[rabernat]

I created a file using pytables, and now I would like to read it into pandas. My (naive) expectation was that these two tools were compatible. But I am getting an error

pd.read_hdf(output_fname, '/floats/trajectories', start=0, stop=10)

gives

ValueError: Shape of passed values is (1, 10), indices imply (1, 753664023)

The file is 44 GB, so I can't really post it. I would be happy to post the h5dump --head metadata if that would help.

(Cross post with dask/dask#747.)

[jreback]

@rabernat

pandas can read a Table from PyTables. Their is simply not enough meta data to generally read other HDF5 constructs. Sure it could be done in some way, but is not a very easy/nice use case. The pandas written formats can be read by PyTables, but might not be semantically readable.

[rabernat]

pandas can read a Table from PyTables

My file was created by pytables. That's why I was surprised it didn't work.

[jreback]

well you would have to show a specific case then

[rabernat]

The example below reproduces the error. The problem is with the start and stop kwargs. Without those, it works fine. But they are necessary if dask is going to be able to chunk the file.

import tables
import pandas as pd
import numpy as np

output_fname = 'test.h5'
class LFloat(tables.IsDescription):
    npart   = tables.Int32Col(pos=1)   # float id number, starts at 1
    time    = tables.Float32Col(pos=2)  # time of the datapoint
    x       = tables.Float32Col(pos=3)  # x position
    y       = tables.Float32Col(pos=4)  # y position
    z       = tables.Float32Col(pos=5)  # z position

dtype = tables.description.dtype_from_descr(LFloat)

nrecs = 10
with tables.openFile(output_fname, mode='w', title='Float Data') as h5file:
    group = h5file.createGroup("/", 'floats', 'Float Data')
    table = h5file.createTable(group, 'trajectories', LFloat,
                                "Float Trajectories", expectedrows=nrecs)
    for n in range(nrecs):
        d = np.empty(1, dtype)
        d['npart'] = n
        table.append(d)

    table.cols.npart.createIndex()
    table.flush()

df = pd.read_hdf('test.h5', '/floats/trajectories', start=0, stop=5)

[jreback]

ok, this is a bug in the reading this tables. never had a need to do this, so its untested. So if you want to submit a pull-request fix would be great. The issue is here I think.