New data format: ROOT files

[ibab]

Hi,
I've recently written a tiny python package for loading/saving ROOT files as pandas DataFrames: root_pandas.
ROOT is the main data format used by particle physicists.

Do you think it might be worth adding read_root and to_root functions to pandas itself?
This could convince a lot of physicists to give pandas a try.
The dependencies it would add could be handled like it's currently done with hdf5.

ROOT performs well in comparison with hdf5 and could be a useful addition to pandas, even ignoring the fact that it is very popular in physics.

If there's interest, I would polish my code and create a pull request.

[jreback]

this is a format that the hdf5 files are setup in?

kind of like a convention for naming and structure?

[ibab]

It's a completely different format that was designed from the ground up for physics experiments. I think it might actually predate HDF5.

Here's an info page on the file format: https://root.cern.ch/drupal/content/root-files-1

[jreback]

So I think a nice way to do this would be to create pandas-io (like we are doing in #8961)

where we can put 'non-core' IO packages handling. So that an import would inject them into the pandas namespace but it would be optional (and updated externally to pandas), maybe something like

import pandas-io import root_format
df.to_root()
pd.from_root()

would work

[ibab]

That sounds like the right place to put a package like this.
It might be even more convenient if pandas would detect the presence of pandas-io and import all new formats automatically.

[jreback]

@ibab if you'd be interested in this, then I can setup the repo and you can port things.

@jorisvandenbossche ?

[ibab]

@jreback Sure, thanks! I'll be happy to port it over.

[TomAugspurger]

We should also make sure the new methods are into aware. I don't believe it will be too difficult since anything we read will go through a DataFrame, so there should be just one interface (or two if you split out chunked).

[shoyer]

It might be even more convenient if pandas would detect the presence of pandas-io and import all new formats automatically.

This is an intriguing idea, and certainly would be a win for interactive use. And I really like supporting method chaining. But there are a few tradeoffs to consider here:

1. I don't see anyway to add methods like to_root dynamically (e.g., only if ROOT is installed) without triggering potentially expensive imports (CLN: make sure that we don't have extraneous imports #9482) at import time for pandas.
2. It's slightly un-Pythonic to have implicit imports from different modules. On the other hand, adding methods to pandas.DataFrame when root pandas is imported is also not very explicit.
3. The bigger issue is that injected methods are hard to discover or track down if they're not in pandas itself. At the very least, users expect that DataFrame methods should be documented in our API docs.

Considering these factors, it seems like creating stubs like to_root in pandas itself would be ideal. We could outsource the actual implementation to pandas-io or whatever, and refer to those external sources in the pandas documentation. pandas-io could even replace entirely generic stubs like DataFrame.to_root(*args, **kwargs) with an appropriate signature at import time.

[ibab]

Adding stubs directly to pandas looks like a good idea.
The only problem I see is that pandas-io would then have to wait for new pandas releases to add new data formats.

[KonstantinSchubert]

Hi,
I would be interested to know if there are any updates to this?

[jreback]

as discussed above, this could be added to a pandas-io type of package if @ibab were interested in doing this

[tunnell]

+1. Is @ibab interested in help?