Provide a template-engine based way of rendering pandas data objects

to_string(), to_html(), to_latex(), formatters, float_format, etc.

related #459

• The existing methods are not as flexible as users sometimes wish,
  see Possible interactions with taldcroft/asciitable? #167, Float format syntax #2502, ENH: add escape parameter to to_html() #2919 (comment), Multilevel indexing with .to_latex() method merges two index columns #2942, Multilevel indexing with .to_latex() method merges two index columns #2924,HTMLFormatter.to_html() does not look at show_index_names #3195, Update DataFrame.to_latex() to have more flexibility #3196,ENH: update DataFrame to_latex for nicer typesetting #3264, BUG: adjust to_latex column format when no index #3467, Update DataFrame.to_latex() to have more flexibility #3197,ENH: Remove the hardcoded border=1 in the to_html dataframe export. #4578,API: Would it be useful for everyone if DataFrame.to_html (and to_string) had an argument to specify justify settings for each column? #4315, ml1., BUG: formatters argument to DataFrame.to_latex() is broken #6052
• Pandas' core competence is it's data structures and data ops DSL.
  the output formats were added for convenience, but don't see much
  ongoing work (formatting tables isn't sexy).
• Users roll their own using custom code, there's no uniform way to do it
  in a way that naturally extends pandas.

Template engines [jinja2, mako] are the accepted way of converting data into documents,
and the web dev community has provided great libraries to do it.
We should piggyback.

This would be good:

• Give users the power to scratch their own itch, avoiding the need for pandas devs
  do it, or ignoring "fringe" use cases.
• Provides users with a readable baseline , which is easy to modify and adapt to their needs.
• Having a standard way to do it (by customizing a template), users can more easily
  have their output blend into python, and they can contribute / manage / source-control
  document templates in a clean, self-contained way.
• Escape the "%d" not "{:d}" silliness and format strings in general, which is a relic of
  olden 2.5 support days. No more parameter hell necessary to make the existing methods
  more expressive. I never ever want to format html tables using function arguments.
• Use a better way to express formatting logic via a DSL, rather then piecing together
  python strings.
• Unify the handling of textual representations, not different code, only different template files.
• Can refactor this out of core (but not really, see below)
• Can provide features such as conditional formatting, styling.

However,

• Can't really remove existing methods.
• Adding a template engine dependency to pandas is clearly feature-creep (IMO).

So,

• Make it a standalone project out of core (or optional dependency?)
• Translate the existing styles into template form.
• Provide hooks in pandas to allow users to optionally install the
  library and use it for rendeing
• Let users know it's available for their use.

[cpcloud]

@y-p What kind of API are you thinking about here?

For 1) existing to_* methods something like:

df.to_html(template='/path/to/template')

and 2) New to_* methods, maybe something ilke:

df.to_template(template='/path/to/template')

[cpcloud]

Or for 1) just keeping the current API (probably better because of back compat) and "freezing" the keyword parameters (and using them in the template).

import pandas.export_templates as et
et.register_template("/tmp/html_with_shocking_pink_headings.tmpl","HoTPInK!")
df.render("HoTPInK!",output_file="/tmp/output.html") ; otherwise stdout.
df.render("html",output_file="/tmp/output.html") # bundled template
df.render("latex",output_file="/tmp/output.html") # bundled template
# conditional formatting
et.Stylist(df).cmap("midnight_bluez").zebra([1,1,2],axis=0).topk(rows=[("Foo","Bar"),cols=["ColA"],n=10,axis=0,style="heatmap").render("html")
# or the more general select/apply
et.Stylist(df).select(lambda r,c: r>c).apply("some generalized style DSL here: bg_color=grey20").
  select(lambda r,c: r %2 ==0).apply(font_bold=True).render("latex")

and so on...

The style DSL and all the bells and whistles you could put into this make me fear
feature creep if it's placed within pandas. The library I developed and shelved sort
of balooned into a monstrosity that wasn't very appealing to use.

The idea was to use pandas objects as fundamental data containers, let the user
use some selector language (predicate over position and value/textual mini-dsl/perhaps
even ripping off css3 constructs) to express selections and then provide style "tags" to those cells.
The rendering stage compiles that data structure into a context object for the template engine
in a standardized way, and then the templates themselves use that context to generate
the specific output text that realizes that presentation in a given output format.

I was excited about the prospects but my initial attempt to implement did not end in a system
that's lovely to use. leaky abstractions and ugly templates full of logic were the fundamental
result. The output tables were really pretty though (ignore the failed selector mini-language):

If you drop conditional formating and just provide boilerplate templates for html and latex, which
users can modify in an editor (indent this, double border that) it can be simpler, but may not
be appealing to users if learning a new templating language is required.

That doesn't cover the value formatters problem ("%3.6f", translate factors levels into labels, etc').
There's lots to solve there, which is why I think it makes sense to do it as a project on top of pandas.

I'm not going to try again for the forseeable future.

I'm working on this again now. typical.

@olgabot, can you recommend a few alternatives for colormaps to bake into this?
Specifically, heatmaps by value (top10 values in column, for example).

Nailed it. claiming this for 0.14.

If there are any watchers out there who dabble in design and want to sling some css to style
the way tables look by default for thousands of pandas users out there, raise your hand.

[jreback]

you might want to consider using df.eval/query parsing machinery (instead of lambdas)

Reduced scope considerably, beautifully simple now and trivial to implement.