Allow custom metadata to be attached to panel/df/series?

related:
#39 (column descriptions)
#686 (serialization concerns)
#447 (comment) (Feature request, implementation variant)

Ideas and issues:

• is pickle stable enough?
• serialization req. potentially couples this with storage format
• allow arbitrary python objects?
• impl. option: JSONables-only + type tag + load/save callback hook
• getter/setter or attribute style access (limits naming)?
• flat or hirarchical?

[jreback]

storage of this data is is pretty easy to implement in HDFStore.
(not pushing HDFStor as a general mechanism!)

general thoughts on meta data:

• A single meta attribute avoids lots of issues
• vote for 'meta'
• allow hierarchical
• allow anything (eg no checking)
• do u keep copying it?, just shallow I think
• in serialization make json only

specific to HDFStore:

• serialization -(PyTables already has this requirement), maybe have this raise/warn user that they r going to lose data
• in HDFStore: what to do when appending (and not overwriting) - maybe ignore it, and force user to pass an option, meta=True?

pytables it is a very good fit in terms of features, but:

• pytables is not currently a hard dependency of pandas.
• is not currently available for python3?
• does not support in memory databases (?)
• alsom cannot itself be pickled or jsoned easily? so you have to embed a binary hdfs file.

[jreback]

oh - was not suggesting we use this as a backend for specific storage
of meta deta in general (the above points were my comments in general on meta
data - reading it again it DOES look like I am pushing HDFStore)

was just pointing out that HDFStore can support meta deta if pandas structures do

to answer your questions

• not a hard dependency - nor should pandas make it one
• not yet py3 (being worked on now I believe)
• not in memory capable
• hdf is meant to be an external format
  (so would have to include a separate file)

+1 for all meta data living under a single attribute

I'm against allowing non-serializable objects as metadata, at all. But not sure
if that should be a constraint on the objects or the serialization format.

in any case, a hook+type tag mechanism would allow users to plant ids of external
objects and reconstruct things at load-time.
I've been thinking of suggesting a hooking mechanism elsewhere (for custom representations
of dataframes - viz, html and so on).

[gerigk]

what do you mean by not in memory capable?

HDF5 has an in memory+ stdout writer and pytables support has been added
recently.

(PyTables/PyTables#173)

On Tue, Dec 11, 2012 at 8:31 AM, jreback notifications@github.com wrote:

oh - was not suggesting we use this as a backend for specific storage
of meta deta in general
just that HDFStore can support meta deta if pandas structures do

to answer your questions

• not a hard dependency - nor should pandas make it one

• not yet py3 (being worked on now I believe)

• not in memory capable

  —
  Reply to this email directly or view it on GitHubhttps://github.com/Allow custom metadata to be attached to panel/df/series? #2485#issuecomment-11233812.

oh. I wasn't aware of that and didn't find anythig in the online docs.
This seems to have been added after the latest 2.4.0 pytables release and so is not
yet available off pypi or the distros.

[hughesadam87]

Thanks for including me on this request y-p.

IMO, it seems like we should not try to prohibit objects as metadata based on their serialization capacity. I only say this because how would one account for every possible object? For example, Chaco plots from the Enthought Tool Suite don't serialize easily, but who would know that unless they tried. I think that it's best to let users put anything as metadata, and if it can't serialize, then they'll know when an error is thrown. It is also possible to have the program serialize everything but the metadata, and then just alert the user that this aspect has been lost.

Does anyone here know the pandas source code well enough to understand how to implement something like this? I really don't have a clue, but hope this isn't asking too much of the developers.

Also, I think this addition will be a nice way to appease people who are always looking to subclass a dataframe.

*up vote for adding attribute being called 'meta'
*up vote for putting it on Index classes as well as Series, DataFrame and Panel

[dalejung]

Last time I checked, HDF5 has a limit on the size of the AttributeSet. I had to get around it by having my store object encapsulate a directory, with .h5 and pickled meta objects.

[dalejung]

I think that adding metadata to the DataFrame object requires that it serialize and work with all backends (pickle, hdf5, etc). Which probably means restricting the type of metadata that can be added. There are corner cases to pickling custom classes that would become pandas problems.

[hughesadam87]

Hi guys. I'm a bit curious about something. This fix is currently addressing adding custom attributes to a dataframe. The values of these attributes, they can be python functions no? If so, this might be a workaround to adding custom instance methods to a dataframe. I know some people way back when were interested in this possibility.

I think the way this could work is the dataframe should have a new method, call it... I dunno, add_custom_method(). This would take in a function, then add the function to the 'meta' attribute dictionary, with some sort of traceback to let the program know it is special.

When the proposed new machinery assigns custom attributes to the new dataframe, it also may be neat to automatically promote such a function to an instance method. If it could do that, then we would have a way to effectively subclass a DataFrame without actually doing so.

This is likely overkill for the first first go around, but maybe something to think about down the road.

[jreback]

@dalejung do you have a link to the AttributeSet limit?
@hugadams you can simply monkey-patch if you want custom instance methods

import pandas
def my_func(self, **kwargs):
    return self * 2
pandas.DataFrame.my_func = myfunc

[hughesadam87]

@jreback: Thanks for pointing this out man. I've heard of monkeypatching instance methods, but always thought it was more of a colloquialism for something more difficult.

Thanks for showing me this.

[dalejung]

@jreback http://www.hdfgroup.org/HDF5/doc/UG/13_Attributes.html#SpecIssues maybe? It's been awhile and it could be that pytables hasn't implemented new HDF5 features.

Personally, I had a dataset with ~40k items of metadata. Nothing complicated, just large. It was much easier to just pickle that stuff separately and use HDF for the actual data.