ENH: Display IntEnums by name rather then value

[dzimmanck]

Is your feature request related to a problem?

When displaying dataframes that contain IntEnums in the index, columns, or data, it would be nice if they displayed similar to Enum types by using the Enum names that correspond to the value rather then the integer values, which somewhat defeats the prupose of using IntEnums over just ints with alias names.

Describe the solution you'd like

I would like pandas to display IntEnums in the same way as Enums.

API breaking implications

Non that I am aware of, as this request is only for a display change.

Describe alternatives you've considered

I have considered two solution which both have drawbacks. First was to monkey patch Int Array Formatter. This ended up not working as Pandas appears to case IntEnums to regulat ints as soon as they are put into the frame. The second was to change the IntEnums to be categorical, which makes logical sense, but to get them to display properly I need to map the data to the names of the enums which changes the underlying data to strings.

Additional context

Here is a simple example.

from enum import Enum, IntEnum
import pandas as pd

class Colors(Enum):
    red = 1
    blue = 2

class IntColors(IntEnum):
    red = 1
    blue = 2


df = pd.DataFrame({Colors.red:[1,2,3], Colors.blue:[5,6,6]})
int_df = pd.DataFrame({IntColors.red:[1,2,3], IntColors.blue:[5,6,6]})

print('This is how I want the dataframe to display')
print(df)
print()
print('This is how it is displayed')
print(int_df)

Which outputs:

This is how I want the dataframe to display
   Colors.red  Colors.blue
0           1            5
1           2            6
2           3            6

This is how it is displayed
   1  2
0  1  5
1  2  6
2  3  6

[TomAugspurger]

Can you provide an example along with your expected output?

[dzimmanck]

@TomAugspurger

I updated my comment with a simple example.

[TomAugspurger]

Seems that the issue isn't in the repr, it's that pandas converts the IntEnum to integers

In [67]: int_df.columns
Out[67]: Int64Index([1, 2], dtype='int64')

In [70]: pd.Index([IntColors.red])
Out[70]: Int64Index([1], dtype='int64')

[dzimmanck]

@TomAugspurger, I think that is part of the issue, but they still don't print correctly even when the columns are cast as objects explicitly.

from enum import Enum, IntEnum
import pandas as pd


class IntColors(IntEnum):
    red = 1
    blue = 2

# lets create an index explicitly called out as objects
columns = pd.Index([IntColors.red, IntColors.blue], dtype=object)

print('I see colors')
print(columns)
print()
# now lets build a dataframe with those columns
df = pd.DataFrame([[1,2], [3,4]], columns=columns)

print('I still see colors, so the columns are the correct type')
print(df.columns)
print()
print('But not when then entire dataframe is printed')
print(df)

Which returns:

I see colors
Index([IntColors.red, IntColors.blue], dtype='object')

I still see colors, so the columns are the correct type
Index([IntColors.red, IntColors.blue], dtype='object')

But not when then entire dataframe is printed
   1  2
0  1  2
1  3  4

[TomAugspurger]

Thanks, that might be a bug. Can you check the formatting code in io/formats/format.py to see where it's converted?

Can you also edit your original post to just include the relevant details (construct dataframe, show actual output, show expected ouptut).

[asishm]

the main issue seems to be because for IntEnums - IntColors.red == 1 evaluates to True

which causes two things:

1. without specifying dtype = object, pandas infers the dtype to be integer (lib.infer_dtype)
   that explains

In [70]: pd.Index([IntColors.red])
Out[70]: Int64Index([1], dtype='int64')

2. when formatting colums.format is called - which calls lib.maybe_convert_objects(values, safe=1) which converts to int, presumably again because IntColors.red == 1 evaluates to True

In [41]: lib.infer_dtype([IntColors.red, IntColors.blue])
Out[41]: 'integer'

In [43]: lib.maybe_convert_objects(columns._values)
Out[43]: array([1, 2])

[TomAugspurger]

Thanks. I think let's separate the two issues as much as possible. Changing the behavior of Index seems harder since it's potentially API-breaking. Let's focus just on the output formatting here when you get an object-dtype index with these enums.

[dzimmanck]

@TomAugspurger,

I think the issue is in the TableFormatter. The _get_formatter() method re-casts the data as integers if is_integer() evaluates as true before passing to the formatter for display. This explains why all that work I did to try and MonekeyPatch the integer formatter failed. It could no longer differentiate integers from Int Enums:

def _get_formatter(self, i: Union[str, int]) -> Optional[Callable]:
        if isinstance(self.formatters, (list, tuple)):
            if is_integer(i):
                i = cast(int, i)
                return self.formatters[i]
            else:
                return None
        else:
            if is_integer(i) and i not in self.columns:
                i = self.columns[i]
            return self.formatters.get(i, None)

I don't really think the formatter should be re-casting in this manner, but in order to avoid any API breaking changes, we could simply avoid this re-cast JUST for IntEnum types. Something like this:

def _get_formatter(self, i: Union[str, int]) -> Optional[Callable]:
        if isinstance(self.formatters, (list, tuple)):
            if is_integer(i):

                # IntEnums will be displyed differently then ints, so do not re-cast
                if not instance(i, IntEnum):
                    i = cast(int, i)
                    
                return self.formatters[i]
            else:
                return None
        else:
            if is_integer(i) and i not in self.columns:
                i = self.columns[i]
            return self.formatters.get(i, None)

I don't have a pandas dev environment setup to test this, but should be easy to check if you have one with my simple example above. If no one takes that on by tomorrow I will setup a dev environment and do it myself. I do a lot of Cython work so I think I have most of the dependencies already. Would be cool to get a PR in pandas under my name, even such a mundane one.