BUG: concat on axis with both different and duplicate labels raising error

[jorisvandenbossche]

When concatting two dataframes where there are a) there are duplicate columns in one of the dataframes, and b) there are non-overlapping column names in both, then you get a IndexError:

In [9]: df1 = pd.DataFrame(np.random.randn(3,3), columns=['A', 'A', 'B1'])
   ...: df2 = pd.DataFrame(np.random.randn(3,3), columns=['A', 'A', 'B2'])

In [10]: pd.concat([df1, df2])

Traceback (most recent call last):
  File "<ipython-input-10-f61a1ab4009e>", line 1, in <module>
    pd.concat([df1, df2])
...
  File "c:\users\vdbosscj\scipy\pandas-joris\pandas\core\index.py", line 765, in take
    taken = self.view(np.ndarray).take(indexer)
IndexError: index 3 is out of bounds for axis 0 with size 3

I don't know if it should work (although I suppose it should, as with only the duplicate columns it does work), but at least the error message is not really helpfull.

[jreback]

cc @immerrr

I think after #6745 this will be straightforward to fix

[immerrr]

The main issue is how to align indices that both have duplicate items, as of now, indexing with dupes does strange things:

In [1]: pd.Index([1,1,2])
Out[1]: Int64Index([1, 1, 2], dtype='int64')

In [2]: _1.get_indexer_for(_1)
Out[2]: Int64Index([0, 1, 0, 1, 2], dtype='int64')

Apparently, for each non-unique element found in destination, get_indexer tries to insert all locs of this element. I can hardly thinkg of a use case when I'd want to do a reindex(['x', 'y']) and getting (['x', 'x', 'x', 'y']) instead would do.

[jreback]

The dup indexers came out of having duplicates indexers on a unique index, so you have to dup

In [1]: df = DataFrame(np.arange(10).reshape(5,2))

In [2]: df
Out[2]: 
   0  1
0  0  1
1  2  3
2  4  5
3  6  7
4  8  9

[5 rows x 2 columns]

In [5]: df.loc[:,[0,1,1,0]]
Out[5]: 
   0  1  1  0
0  0  1  1  0
1  2  3  3  2
2  4  5  5  4
3  6  7  7  6
4  8  9  9  8

[5 rows x 4 columns]

Now maybe outlaw that I suppose

[immerrr]

Maybe it would make more sense to require destination index to have the same count of duplicate entries for each element present in source, i.e.:

# e.g. these should be ok
pd.Index([1,1,2]).get_indexer_for([1,1,2]) # should be ok and return [0, 1, 2]
pd.Index([1,1,2]).get_indexer_for([2]) # return [2]
pd.Index([1,1,2]).get_indexer_for([1,2,1]) # return [0, 2, 1]

# but these should be forbidden
pd.Index([1,1,2]).get_indexer_for([1,2]) # which one of `1` did you want?
pd.Index([1,1,2]).get_indexer_for([1,1,1,2]) # which `1` should be duplicated?

UPD: or maybe cycle over duplicate elements like np.putmask does...

x = np.arange(5)
np.putmask(x, x>1, [-33, -44])
print x
array([  0,   1, -33, -44, -33])

# so that
pd.Index([1,1,2]).get_indexer_for([1,1,2,1,1]) # return [0,1,2,0,1] ?

[jreback]

right, so that's a duplicate of a duplicate; yes, that would need to be handled differently