Support exporting to JSON Table Schema

[rgbkrk]

For Jupyter based frontends, we would love to see a common tabular format in JSON that we can render (in addition to or in lieu of the current HTML). This would provide us the flexibility to style and format according to data type, as well as have better hooks for theming of tabular data on frontends. Everyone has an opinion, let's give them flexibility to apply it.

It's important to us to support a common JSON format so that for R, Julia, and other languages also can display their DataFrames with similar formatting and styling out of the box.

The best one I've seen so far, with a great amount of discussion and collaboration, is the JSON Table Schema.

Update: In order to include both data + schema, we're using data resource which has media type application/vnd.dataresource+json.

/cc @captainsafia @ellisonbg @jreback @TomAugspurger

[jreback]

xref #9146, #9166

[TomAugspurger]

I'll dig into the schema later, but just to make sure: the basic idea is for pandas to publish multiple outputs (application/html, application/json) wherever we publish just the HTML right now?
More concretely, what changes do we need to make to Series / DataFrames / Indexes to support this? IIRC there isn't a _repr_json_ equivalent of _repr_html_.

[rgbkrk]

Interesting - I just noticed they wrote a wrapper for pandas: https://github.com/frictionlessdata/jsontableschema-pandas-py

On the JupyterLab, notebook, and nteract side, we'd have https://github.com/frictionlessdata/jsontableschema-js to lean on.

[rgbkrk]

the basic idea is for pandas to publish multiple outputs (application/html, application/json) wherever we publish just the HTML right now?

Yes. The media type (mime type in Jupyter parlance) would be something like application/vnd.table-schema.v1+json.

[rgbkrk]

While there's not a repr for arbitrary media types in IPython (we can evolve that as a result of this discussion), there is a way to display raw messages with IPython.display.display:

IPython.display.display({
    'application/json': releases
}, raw=True)

Which shows up in nteract as:

[pwalsh]

Hi. I'm one of the authors of JSON Table Schema, and also part of the team working on reference implementations for this and the related family of specs. The JavaScript implementation is just a little behind the Python one, and probably also of relevance here.

Happy to help.

edit: added link to the JavaScript implementation, in addition to the Python one previously linked.

[rgbkrk]

By the way, on the nteract and jupyterlab side, it's pretty easy for us to iterate with new renderers and media types.

[TomAugspurger]

I don't really see a reason not to add this in pandas; The additional code shouldn't be too much of a burden.

Would clients expect to receive the entire DataFrame, and do their own truncation? I worry a bit about the overhead of publishing huge DataFrames. I would say follow the options in pd.options.display.max_rows, etc. and only ship over some of the DataFrame (but need some way of saying that there's more...)

A few things directly related to the spec that pandas might have trouble with:

• field descriptors: in principal _metadata should carry this, but IIRC we don't have a good story on propagating that though operations, so it's liable to be dropped
• field types: shouldn't have any problems here
• primary key: Typically this would be the (multi)Index, but we don't require uniqueness on that.
• field names: Somewhat rare, but we can have MultiIndexes in the columns, so we could have "multiple rows" of field names; These can be collapsed down to tuples.

[pwalsh]

We are very happy to make any changes needed to https://github.com/frictionlessdata/jsontableschema-pandas-py in order to support this smoothly, and especially in reference to things like streaming data out of a DataFrame, or limiting the rows from a frame for preview, and so forth.