DOC/API: document how to use metadata

[jreback]

from SO
xref #2485
xref #7868

• documentation (maybe in the cookbook) to start.
• add _metadata to Index
• top-level metadata control (maybe)

def unit_meta(self, other, method=None):
     # combine and return a new unit for this

      if self.unit == other.unit:
          return self.unit

      # coerce unit
      return coerced_unit

pd.set_metadata({ 'unit' : unit_meta })

This last will require a bit of change in __finalize__ to handle a metadata finalizer for a specific name (but straightforward)

[shoyer]

I think this is a good idea, but we'll need to do it carefully.

I'm guessing the method argument is there to provide some description of the context? The difficulties with propagating metadata is that there are some many edge cases. For example, to handle units, it's not enough to know that both units are the same -- you also need to what sort of operation is being performed (e.g., + vs * vs ==). And perhaps you might prefer to raise rather than propagate the wrong units, etc.

When I implemented this in xray, I didn't want to deal with these issues, so I took the more conservative approach of dropping custom metadata (other than name) in all binary arithmetic and aggregations. See http://xray.readthedocs.org/en/stable/faq.html#what-is-your-approach-to-metadata

I think this sort of hook system is a good idea to let someone else deal with the complexity. __numpy_ufunc__ (arriving in numpy 1.10) may provide a useful reference design -- I know astropy is planning on using it for their Quantities class.

RE: your specific design. I would rather make users define a custom subclass, e.g., UnitsDataFrame = pd.add_metadata(pd.DataFrame, {'unit': unit_meta}), then use some sort of global state.

[jreback]

@shoyer the design for this was for users to subclass / monkey-patch DataFrame and replace __finalize__ with a completely new routine that handles whatever behavior the user wants. I am proposing an extension to allow a method by method interaction to be specified, e.g. method='merge'|'add' are passed, and then let the user provide a routine to handle it.

This is just a dispatch to user interaction, like: hey I have these 2 objects which I am adding, how do you want to combine the meta-data. It will drop by default, this just allows a 'plug-in' type of mechanism.

[hughesadam87]

Hi,

Really glad to see the subclassing and finalize behavior come into fruition.

I'm not sure I fully understand the scope of this general metadata problem, but in my experience, the relation of metadata to the results of __finalize__ can get very complex, so any solution you come up with should error on the side of conservative as Stephan mentioned. As a user of such functionality, I'd prefer my metadata to cause errors in operations that are not directly supported rather than having a general schema in place that might lead to convoluted, hard-to-find bugs. For example, in a spectroscopy library, if the user adds two objects that have different unit systems, I'd rather an error get raised rather than a decision be made for me based on Pandas' metadata handling system.

From studying GeoPandas as well as pyuvvis, I get the impression that most pandas-subclassing libraries are going to use a relatively limited set of operations from the API, so maybe it's a good idea to make them work out all of the usecases for metadata as they go, and just provide context and suggestions in the docs?

Sorry if this is not germane to the issue at hand ;0

[jreback]

@hugadams

this modification / doc update is for the USER to really do all of the work. pandas provides a framework if you will. but the USER decides ALL interactions with metadata (otherwise they are NOT propogated). So it is basically wide open for the USER to provide a mechanism to propogate some / raise an error if needed etc. I guess some docs are in order!

[hughesadam87]

Sounds great