BUG: DataFrame.shift(axis=1) with multiple blocks of same type

[bwillers]

Shows up whenver you have multiple blocks of the same type in a dataframe.

Int example:

In [1]: a = pd.Series([1,2,3])   
In [2]: b = pd.Series([4,5,6])
In [3]: df1 = pd.DataFrame({'a': a, 'b': b})
In [4]: df2 = pd.DataFrame({'a': a}).assign(b=b)

In [5]: df1._data.blocks
Out[5]: (IntBlock: slice(0, 2, 1), 2 x 3, dtype: int64,)

In [6]: df2._data.blocks
Out[6]:
(IntBlock: slice(0, 1, 1), 1 x 3, dtype: int64,
 IntBlock: slice(1, 2, 1), 1 x 3, dtype: int64)

In [7]: df1
Out[7]:
   a  b
0  1  4
1  2  5
2  3  6

In [8]: df2
Out[8]:
   a  b
0  1  4
1  2  5
2  3  6

In [9]: df1.shift(1, axis=1)
Out[9]:
    a  b
0 NaN  1
1 NaN  2
2 NaN  3

In [10]: df2.shift(1, axis=1)
Out[10]:
    a   b
0 NaN NaN
1 NaN NaN
2 NaN NaN

Or with different data types:

In [2]: df = pd.DataFrame({'a': [1,2,3], 'b': ['A', 'B', 'C']})

In [3]: df
Out[3]:
   a  b
0  1  A
1  2  B
2  3  C

In [4]: df.shift(1, axis=1)
Out[4]:
    a    b
0 NaN  NaN
1 NaN  NaN
2 NaN  NaN

I expect a similar issue pops up with panels and ndframes when shifting across the axis where dtypes can change.

Perhaps this can be fixed by having shift move the column labels rather than the data here?

[jreback]

just needs to have a call to consolidate before it goes to the block manager

[bwillers]

@jreback sorry, i somehow pasted the same example twice, original comment updated. Consolidate would solve the two integer block case, but not the different data dtype case, unless I'm fundamentally misunderstanding what consolidate does?

[jreback]

yes could be 2 different bugs

[evanpw]

The problem is that if you have multiple blocks, then you can't compute shift with axis=1 by shifting each block separately and then putting them together, which is what the current implementation tries to do.

[ryscet]

This is still a remaining issue. It was only solved for data frame made of blocks of the same type. For mixed types shift on columns does not work properly. See in the example by changing to a = pd.Series([1,2,3]).astype(float)