ENH: Add Rich formatting support to Pandas object displays with `__rich_console__` method

[dwreeves]

Feature Type

• Adding new functionality to pandas

• Changing existing functionality in pandas

• Removing existing functionality in pandas

Problem Description

Rich is a very popular and, at this point, a standard tool in the Python ecosystem for beautiful rendering of text in terminals. Pandas's rich rendering via _repr_html_ is very optimized for the Jupyter ecoystem, but lags behind in the terminal space. As someone who primarily codes in a terminal with a Rich patched REPL, I find this unfortunate and unnecessary. Hence I am adding this feature request in hopes that the situation can be improved. I can work on this if it is agreed that this feature would be handy and useful.

Feature Description

This is a very rough solution that does not cover all the appropriate challenges, requirements and complexities that would need to be met. It is just to give you an idea that it is possible:

import pandas as pd

class DataFrame(pd.DataFrame):

    def __rich_console__(self, console: "rich.console.Console", console_options: "rich.console.Console"):
        from rich.table import Table
        table = Table()
        table.add_column(self.index.name if self.index.name is not None else "")
        for c in self.columns:
            table.add_column(str(c))
        for r in self.itertuples():
            table.add_row(*map(str, r))
        yield from table.__rich_console__(console, console_options)

from rich.console import Console

console = Console()

df = DataFrame([
    {"name": "John Doe", "phone": "555-555-5555", "last_purchase_amount": 1.23},
    {"name": "Mary Sue", "phone": "800-123-4567", "last_purchase_amount": 100.00}
])

console.print(df)

A few things are missing from this solution. Notably, it needs to hook into the Pandas options; it needs to be able to truncate the rows and columns based on the relevant display options. Series objects also need to have a __rich_console__. The Styler object should, as well, and this should hook into the Styler object configuration. There should probably be a way to set additional display options for this somewhere (e.g. display.rich.[...] and styler.rich.[...]). You'd need to hook into the try-except import stuff that Pandas does on the backend for optional dependencies. Lastly, of course, this would all need to be documented.

Alternative Solutions

• Users can just do it themselves, I suppose.
• You can use the __rich__ method, but I think you'd want to have access to the console parameters when fleshing out the code.

Additional Context

I could not find any related issues.

[Delengowski]

Can you show examples?

Specifically with and without a pandas.MultiIndex both axis 0 and axis 1? Without and without axis 0 names? What kind of support does rich provide for multi indexes?

[mroeschke]

Looks like there's this external library that supports rich for pandas objects: https://pypi.org/project/rich-dataframe/

[dwreeves]

@Delengowski

Can you show examples?

The example code I provided in the issue outputs like this:

One can certainly do additional work however to implement more styling options. Rich has a bunch.

Specifically with and without a pandas.MultiIndex both axis 0 and axis 1? Without and without axis 0 names? What kind of support does rich provide for multi indexes?

Multiindexes are a good point. Rich does not "provide support for multindexes" per se, it's is just a visualization library. The way you render them depends on how you want to render them.

With my code example above the following dataframe:

df = DataFrame(
    [
        {"name": "John Doe", "phone": "555-555-5555", "last_purchase_amount": 1.23},
        {"name": "Mary Sue", "phone": "800-123-4567", "last_purchase_amount": 100.00},
    ],
    index=pd.MultiIndex.from_tuples([(1, 2), (3, 4)], names=[None, None])
)

You get this:

You can modify the code to support more functionality, though:

import pandas as pd

# todo: replace with actual pandas settings
settings__display_rich_combine_multindex = False

class DataFrame(pd.DataFrame):

    def __rich_console__(self, console: "rich.console.Console", console_options: "rich.console.Console"):
        from rich.table import Table
        table = Table()
        split_multiindices = False
        # Todo: check if there is a better way to check for multi indices
        if self.index.names and not settings__display_rich_combine_multindex:
            split_multiindices = True
            for name in self.index.names:
                table.add_column(name if name is not None else "")
        else:
            if self.index.names:
                name = ",".join(self.index.names)
            else:
                name = self.index.name
            table.add_column(name if name is not None else "")

        for c in self.columns:
            table.add_column(str(c))
        for r in self.itertuples():
            if split_multiindices:
                row_data = map(str, [*r[0], *r[1:]])
            else:
                row_data = map(str, r)
            table.add_row(*row_data)

        yield from table.__rich_console__(console, console_options)

from rich.console import Console

console = Console(color_system="truecolor")

df = DataFrame(
    [
        {"name": "John Doe", "phone": "555-555-5555", "last_purchase_amount": 1.23},
        {"name": "Mary Sue", "phone": "800-123-4567", "last_purchase_amount": 100.00},
    ],
    index=pd.MultiIndex.from_tuples([(1, 2), (3, 4)], names=["foo", "bar"])
)

console.print(df)

Output:

You can do the thing where you create a new row just for the index names, if you want, as well. Again, it's very composable.

---

Note that all of this code is very rough. Implementing this would take a lot, lot more effort than I put into this code right here. I'm just trying to see first whether the Pandas dev team agrees that this should be implemented.

[dwreeves]

@mroeschke

Looks like there's this external library that supports rich for pandas objects: https://pypi.org/project/rich-dataframe/

I'm aware, and the owners of that library should definitely be consulted and should be brought into this discussion if we agree that this should exist as a native feature, as they have relevant knowledge and I'm sure they'd love to see this become a native feature + work on it.

But, I do think there is a lot of value in making this native, and not just relying on third-party support, even if said third-party support exists.

I would put it like this: if there was a library called jupyter-dataframe, it would still have been a good idea for Pandas to implement _repr_html_ natively. It would not be a good user experience to tell people, "well, just use this third party library and wrap everything with jupyter_dataframe.prettify(df). Arguably, Pandas may not have become nearly as popular as it is today if _repr_html_ wasn't natively implemented.

Some of the benefits of adding this natively:

• Just type >>> df and get your pretty repr, instead of needing to wrap in prettify(df). That's a massive difference in quality of life for exploratory workflows.
• User does not need to install another library.
• Likely to receive more functionality, bugfixes, and community support. For example, the linked library does not seem to support MultiIndexes, which I was just asked about above.
• People are way more likely to use this in general, I think, if it is officially supported.

Again, Rich is popular (46k Github stars), modern, and people just really enjoy it. I'm not proposing pandas-dev creates a new optional dependency on some random no-name library. I'm proposing that terminal-based Pandas workflows should get a modern, native, built-in facelift, in the same spirit that Pandas already supports a quality native experience for Jupyter-based workflows.

Also to clarify, I don't necessarily see pointing this out as a tacit disagreement with my argument that it should be native. I do just want to state my position here that for purposes of native support, I don't see it as relevant that a third-party library exists, other than that we could @ the creator of that library and get them involved in this, if we agree it should be native 😄

[dwreeves]

It looks like all the unit-tests for Pandas display related stuff already are:

• For dataframes reprs: https://github.com/pandas-dev/pandas/blob/main/pandas/tests/frame/test_repr.py
• For series reprs + other reprs: https://github.com/pandas-dev/pandas/blob/main/pandas/tests/series/test_formats.py
• For HTML, console, etc. outputs: https://github.com/pandas-dev/pandas/blob/main/pandas/tests/io/formats (HTML tests seem to be the most comprehensive.)

Obviously, reasonable acceptance criteria would be that a Rich hook can implement all these behaviors but as Rich-formatted tables. There are a few stylistic decisions that one would still have to make even within this constraint, however, but reimplementing the spirit of these tests but for Rich would really narrow down the scope of the problem.

[Delengowski]

Note that all of this code is very rough. Implementing this would take a lot, lot more effort than I put into this code right here. I'm just trying to see first whether the Pandas dev team agrees that this should be implemented.

Thanks for this. The first thought that came to mind was merging of cells for multi index. I would think that unless Rich can do that, it's a non starter because displaying rich bases repr for a data frame with multi index that looks different from the regular repr would be incredibly confusing.

That's why I was asking, to see if it has out of box support or if it would be a bit of work.

[Delengowski]

It looks like spans are desired but not yet implemented nor a high priority.

Textualize/rich#164

[dwreeves]

You mean to support this test case?

https://github.com/pandas-dev/pandas/blob/e14a9bd41d8cd8ac52c5c958b735623fe0eae064/pandas/tests/io/formats/data/html/index_5.html

If you do not include horizontal lines to split out the rows (which is the default behavior of Rich, i.e. show_lines=False, and also is likely a reasonable default for Pandas too), then the reasonable 'work around' is just to display a blank string. Because there are no horizontal dividers, this will visually look correct.

This only gets finicky when show_lines=True, and then you have empty cells. This would not look very good.

An option here is (pseudocode):

if not config.show_lines and index_is_same_as_above:
   # don't repeat the row as above
   multi_index_row_label = ""
else:
   # if `show_lines` is on, stylistically the best option is to repeat the value over again.
  multi_index_row_label = row.index

This does not 100% meet the exact same style for MultiIndex support that HTML + base repr do, but only specifically in the case of show_lines=True. When show_lines=False then it should look correct to users.

Another option: Disable support for show_lines=True. I don't love this, as users would probably expect a bit of customizability for these Rich Table objects, but it is an option to enforce this style decision for multi-indexes.

---

If the main blocker to adoption of this feature is a concern over support for this and a few other things, rather than e.g. this feature being out of scope in general for Pandas, we can potentially make changes to Rich itself.

My question is: If Rich had every feature you'd reasonably want to render the reprs in a way that is consistent with the current repr behavior (which I think it does), would Rich support still be considered out of scope for Pandas? Or would it be potentially in-scope? Because if the latter, i.e. Rich support is in-scope, it's possible we can try to prioritize any missing functionality on Textualize/Rich's end; I'm sure they'd be interested in unblocking a major library like Pandas adopting Rich.

[dwreeves]

Here is an example of the snippet code modified to emulate the rowspan-like visualization that the base repr provides:

import pandas as pd

# todo: replace with actual pandas settings
settings__display_rich_combine_multindex = False

class DataFrame(pd.DataFrame):

    def __rich_console__(self, console: "rich.console.Console", console_options: "rich.console.Console"):
        from rich.table import Table
        table = Table()
        split_multiindices = False
        # Todo: check if there is a better way to check for multi indices
        if self.index.names and not settings__display_rich_combine_multindex:
            split_multiindices = True
            for name in self.index.names:
                table.add_column(name if name is not None else "")
        else:
            if self.index.names:
                name = ",".join(self.index.names)
            else:
                name = self.index.name
            table.add_column(name if name is not None else "")

        _prev_index = None

        for c in self.columns:
            table.add_column(str(c))
        for r in self.itertuples():
            if split_multiindices:
                if _prev_index is not None:
                    new_index = [
                        new if prev != new else ""
                        for prev, new in zip(_prev_index, r[0])
                    ]
                else:
                    new_index = r[0]
                _prev_index = r[0]
                row_data = map(str, [*new_index, *r[1:]])
            else:
                row_data = map(str, r)
            table.add_row(*row_data)

        yield from table.__rich_console__(console, console_options)

from rich.console import Console

console = Console(color_system="truecolor")

df = DataFrame(
    [
        {"name": "John Doe", "phone": "555-555-5555", "last_purchase_amount": 1.23},
        {"name": "Mary Sue", "phone": "800-123-4567", "last_purchase_amount": 100.00},
        {"name": "Bob James", "phone": "800-234-5678", "last_purchase_amount": 50.00},
    ],
    index=pd.MultiIndex.from_tuples([(1, 1), (1, 2), (2, 1)], names=["foo", "bar"])
)

console.print(df)

This is the output:

Looks decent if you ask me.

However, you can see that it looks bad when I replace table = Table() with table = Table(show_lines=True):

The 'work around' I propose is, when show_lines=True, we display everything:

In the sloppy draft code above, you can achieve this by replacing this line: if _prev_index is not None: with: if _prev_index is not None and not table.show_lines:.

This is not ideal, but it's the best idea I can come up with that doesn't involve significant rewrites of Rich's internals and/or getting Rich maintainers involved (again, not that this would be an impossible ask if a library as big as Pandas were to ask for a feature).