BUG: Index set operations issues

[pijucha]

Issues:

• Index.difference and symmetric_difference raise for mixed types (solved by BUG/PERF: Sort mixed-int in Py3, fix Index.difference #13514)
• Index._get_consensus_name incorrectly returns an empty index (solved by BUG/PERF: Sort mixed-int in Py3, fix Index.difference #13514)
• bug in Index.union for non-unique indexes (see comment)
• bug in Index.intersection for non-unique indexes (same comment)
• inconsistent sorting in of mixed-int Indexes in python2 (see comment)

---

Index.difference and symmetric_difference raise for mixed types

Code Sample, a copy-pastable example if possible

idx1 = pd.Index([0, 1, 'A', 'B'])
idx2 = pd.Index([0, 2, 'A', 'C'])

idx1.difference(idx2)
...
  File "/usr/local/lib64/python3.5/site-packages/pandas/indexes/base.py", line 1861, in difference
    theDiff = sorted(set(self) - set(other))
TypeError: unorderable types: str() < int()

idx1.symmetric_difference(idx2)
...
  File "/usr/local/lib64/python3.5/site-packages/pandas/indexes/base.py", line 1861, in difference
    theDiff = sorted(set(self) - set(other))
TypeError: unorderable types: str() < int()

But union and intersection work:

idx1.union(idx2)
Out[14]: Index([0, 1, 'A', 'B', 2, 'C'], dtype='object')
idx1.intersection(idx2)
Out[15]: Index([0, 'A'], dtype='object')

Expected Output

idx1.difference(idx2)
Out[]: Index([1, 'B'], dtype='object')

idx1.symmetric_difference(idx2)
Out[]: Index([1, 'B', 2, 'C'], dtype='object')

output of pd.show_versions()

INSTALLED VERSIONS
------------------
commit: 4a6621fcaa9a3b98172b90a69de574ec94b108df
python: 3.5.1.final.0
python-bits: 64
OS: Linux
machine: x86_64
byteorder: little

pandas: 0.18.1
...

[max-sixty]

The issue is with the sorting:

In [8]: pd.Index([0, 1, 'A', 'B']).sort_values()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-8-cd271f2d641b> in <module>()
----> 1 pd.Index([0, 1, 'A', 'B']).sort_values()

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pandas/indexes/base.py in sort_values(self, return_indexer, ascending)
   1565         Return sorted copy of Index
   1566         """
-> 1567         _as = self.argsort()
   1568         if not ascending:
   1569             _as = _as[::-1]

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pandas/indexes/base.py in argsort(self, *args, **kwargs)
   1639         if result is None:
   1640             result = np.array(self)
-> 1641         return result.argsort(*args, **kwargs)
   1642 
   1643     def __add__(self, other):

TypeError: unorderable types: str() > int()

We could try removing the sorting

[jreback]

These should be catching the TypeError as numpy doesn't handle this very well. You can slightly update the guarantee to sort if possible.

[pijucha]

I thought the both difference functions could be rewritten in the style of union (and sort only if it is possible), especially since there is a performance issue #12044.

BTW, not directly related but still in a set function.
There's an internal _get_consensus_name(), the purpose of which is not clear to me, and something is not quite right:

idx1 = pd.Index([1, 2], name='A')
idx2 = pd.Index([], name='B')
idx1.union(idx2)
Out[26]: Index([], dtype='object')
idx2.union(idx1)
Out[27]: Index([], dtype='object')

[jreback]

consensus is the must agree or 1 is None, otherwise it is None.

changing sorting is a big deal, I suspect lots of things will break; further it provides a nice guarantee on the result set. These are essentially set operations into an ordered list.

[pijucha]

consensus

Yes, this is what is written in the docstring. But it should affect only the name, not the values of the output. Am I right?

[pijucha]

I should have mentioned that groupby calls Index.difference internally and can trigger this exception:

df = pd.DataFrame([[0, 1, 2]], columns = ['A', 'B', 0])
df.groupby('A').sum()
Traceback (most recent call last):
  File "/usr/share/python3.5/site-packages/IPython/core/interactiveshell.py", line 3066, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-225-c27b3941f9c8>", line 1, in <module>
    df.groupby('A').sum()
  File "/usr/local/lib64/python3.5/site-packages/pandas/core/groupby.py", line 97, in f
    self._set_selection_from_grouper()
  File "/usr/local/lib64/python3.5/site-packages/pandas/core/groupby.py", line 469, in _set_selection_from_grouper
    self._group_selection = ax.difference(Index(groupers)).tolist()
  File "/usr/local/lib64/python3.5/site-packages/pandas/indexes/base.py", line 1861, in difference
    theDiff = sorted(set(self) - set(other))
TypeError: unorderable types: str() < int()

[jreback]

@pijucha yeah, guess should guard a bit better against this. Note that this is a safe sorter that I wrote a while back here. Could/should just strip this out to a separate function (and use it here).

[pijucha]

Bugs in union and intersection for non-unique indexes

# This is OK
pd.Index([1, 2, 2]).union(pd.Index([2, 3]))
Out[24]: Int64Index([1, 2, 2, 3], dtype='int64')

# but fails with a non-increasing rhs:
pd.Index([1, 2, 2]).union(pd.Index([3, 2]))
pandas.indexes.base.InvalidIndexError: Reindexing only valid with uniquely valued Index objects

# lost 2
pd.Index([3, 2]).union(pd.Index([1, 2, 2]))
Out[26]: Int64Index([1, 2, 3], dtype='int64')

# too many 2's 
pd.Index([1, 2, 2]).union(pd.Index([2, 2, 3]))
Out[27]: Int64Index([1, 2, 2, 2, 3], dtype='int64')

# too many 2's
pd.Index([1, 2, 2]).intersection(pd.Index([2, 3]))
Out[30]: Int64Index([2, 2], dtype='int64')

# too many 2's
pd.Index([1, 2, 2]).intersection(pd.Index([2, 2, 3]))
Out[32]: Int64Index([2, 2, 2], dtype='int64')

I think these "too many 2's" are manifestations of a bug in the algorithms in pd.algos.*_join_indexer_*:

pd.algos.outer_join_indexer_int64(np.array([1, 2, 2]), np.array([2, 2, 3]))
Out[37]: 
(array([1, 2, 2, 2, 3]),
 array([ 0,  1,  1,  2, -1]),
 array([-1,  0,  1,  1,  2]))

[jreback]

@pijucha yeah these duplicates are prob not tested much if at all.