BUG: Series/Dataframe has bad repr for Right-To-Left (RTL) data

[David-Alkalai]

This code

df=pd.DataFrame([['שלום','עליכם', 'מלאכי השלום', 'ברוכים הבאים']])
df.columns=['A','B','C','D']
df

does this:

See that the data row is close to the right?

Half a workaround is to prepend the unicode LTR character to each row:

print u"\n\u200e".join(unicode(df).split("\n"))

but if you look you can see the data appears in reverse order, since:

In [49]: print df.iloc[0,0]
שלום

The same occurs even if the text is decoded into unicode strings.

I tried to understand the repr code in pandas but it's too complex
for me to follow, full of special cases. I hope the developers can fix this.

[cpcloud]

I don't think this is a pandas issue, because just using a list of the characters above yields the same behavior:

In [35]: x = ['שלום','עליכם', 'מלאכי השלום', 'ברוכים הבאים']

x[0] == 'שלום'
Out[57]: True

[cpcloud]

xref: ipython/ipython#5960

[cpcloud]

I think this should be tracked at IPython because this isn't a pandas issue.

[David-Alkalai]

What "same behavior" do you mean, exactly?

The problem is not that x[0] == 'שלום'. I included that only to show that
even though the data value is in location (0,0) in the dataframe, it appears in the
4th column in the repr when using the unicode LTR trick I suggested.

ipython/ipython#5960, which I opened earlier today is about a seperate issue having to do
with RTL support.

[cpcloud]

What "same behavior" do you mean, exactly?

By "same behavior" I mean that the slice semantics are the same for list, ndarray, Series and DataFrame objects and probably any other sequence. There might be some kind of global modification to __getitem__.

The problem is not that x[0] == 'שלום'. I included that only to show that
even though the data value is in location (0,0) in the dataframe, it appears in the
4th column in the repr when using the unicode LTR trick I suggested.

Check out some operations on the Series and the underlying ndarray, they're reversed:

In [96]: original = pd.Series(['שלום','עליכם', 'מלאכי השלום', 'ברוכים הבאים'])

In [97]: df = DataFrame([original.values])

In [98]: print(df)
      0      1            2             3
0  שלום  עליכם  מלאכי השלום  ברוכים הבאים

In [99]: print(df.loc[0])
0            שלום
1           עליכם
2     מלאכי השלום
3    ברוכים הבאים
Name: 0, dtype: object

In [100]: values = original.values

In [101]: map(lambda x: print(x.decode('utf8')), values)
שלום
עליכם
מלאכי השלום
ברוכים הבאים
Out[101]: [None, None, None, None]

This is not something wrong with pandas; it looks like a feature of IPython. Your LTR trick works for the repr, but doesn't work for the iteration most likely because they're two separate mechanisms for dealing with RTL locales. I'm speculating a lot here, but none of these issues is something wrong with pandas. I'm happy to be proven wrong, just need an example that's specific to pandas.

[David-Alkalai]

By "same behavior" I mean that the slice semantics are the same for list, ndarray, Series and
DataFrame objects and probably any other sequence.

Yes, what about it?

There might be some kind of global modification to getitem.

I don't understand exactly what you mean by this. If you suspect that ipython changes
the meaning of slice indexes if the contents are RTL text... obiously that's not a
reasonable assumption.

Whether it's specific to pandas or not depends on your point of view.
You could change IPython or pandas to fix it (maybe), I was suggesting that
pandas make use facilities from the unicode specification to ensure
the repr is displayed correctly.

If that's "specific to pandas" enough, I can't say.

The issue I think is that a dataframe is tabular in nature, but the way
the text repr is constructed it gets rendered like a text string and
unicode bidi kicks in where it shouldn't. So I'm suggesting pandas
might want to be explicit about it.

Your LTR trick works for the repr, but doesn't work for the iteration most likely
because they're two separate mechanisms for dealing with RTL locales.

Sorry, I don't understand your point about iteration. What two mechanisms
are they that you mean?

[cpcloud]

Yes, what about it?

Simply answering your question.

I don't know why those slice semantics are like that, but that RTL iteration seemingly goes in the opposite direction for an arbitrary sequence smells a bit funny to my LTR centric brain. Again, I was simply pointing out that that's why it shows up like it does in the DataFrame repr. Lotta thats there. Sorry about that 😄

You could change IPython or pandas to fix it, I was suggesting that
pandas make use facilities from the unicode specification to ensure
the repr is displayed correctly.

We merge pull requests liberally around here, so if you'd like to put one up, we're happy to assist you in navigating the hell 🔥 that is the repr-ing code. I know I've done 🙀 ➡️ 😡 many a time in the past when trying to grok that code.

I don't know what the unicode spec is for RTL languages, but it sounds like you do. I'd love to learn about it from you in the context of some real code that actually uses it.

What two mechanisms are they that you mean?

What I meant what that iterating from RTL doesn't change to iterating from LTR when you prefix the characters with a LTR character (not sure why I thought it should, but then I don't understand why it goes from RTL in the first place). So, just forget what I said above about the two mechanisms.

[cpcloud]

Maybe I was a bit hasty in closing this.

[David-Alkalai]

Thanks for reopening.

Really there's no difference in how iteration or slicing work like you suggest(ed)
It's just the way the text is displayed. Very resonable that it would seem bizzare,
hebrew/arabic/farsi/urdu speakers are just used to it.

We merge pull requests liberally around here

and use emoji!

[dsm054]

Out of curiosity, do Hebrew tables normally have columns in RTL order? Would you typically see "3 2 1 0", or "dalet gimel bet alef", reading left to right on the page?