Unify index and multindex (and possibly others) API

Have you ever written code that looks like this:

if isinstance(d.index, MultiIndex):
    results = []
    for l in d.index.levels:
       for x in baz(l):
          results.append(foo)
elif  isinstance(d.index, Index):
    for x in d.index:
       foo

I've had to special case the handling of index vs. multindex several times in the past.
Conceptually, I should be able to treat index as a private case of MultIndex
with nlevels =1, and supporting that in the API would make things nicer.

---

Edit by @cpcloud:
Tasks :

API Unification

Method unification is relatively simple:

• Index.from_tuples and Index.from_arrays are just MultiIndex.from_tuples and MultiIndex.from_arrays moved to classmethods of Index.
• droplevel, ` just raises on Index (right? what would it mean?): API: implement droplevels() for flat index #21115
• has_duplicates is straightforward
• truncate should be equivalent to slicing
• reorder_levels raises if not level=0 or name of index
• equal_levels - straightforward
• levshape - (len(ind),)
• sortorder - None
• get_loc_level - I think meaningless with tuple, raises whatever if not 0 or index name
• is_lexsorted - doesn't need to change
• is_lexosrted_tuple - doesn't need to change
• is_monotonic_*
• lexsort_depth - doesn't need to be changed at all
• searchsorted
• repeat
• levels and labels property for Index - question on whether it should be sorted.
• change to rename behavior: Index will accept either string or single-element list; MI continues to handle only list

[wesm]

Much agreed and this would lead to a lot of code simplification in places.

Great, glad you think so. Do you think this should be done as new stub methods
on Index, or should there really be a deeper unification of Index into MultiIndex at
the code level?

[wesm]

It would be great to do at a deeper level. Maybe needs to wait until after Index is made "not an ndarray" and we push the code to Cython. To do that we first need to fix the serialization / "pickle problem".

[jreback]

@jtratner want to assign yourself?

[jtratner]

sure. @wesm (or others) what do you mean by "Maybe needs to wait until after Index is made 'not an ndarray' and we push code to Cython"? Block Manager is still fundamentally using ndarray right? If we're just thinking that the ndarray becomes the equivalent of the _data attribute of NDFrame, then I understand.

[jreback]

@jtratner

right now Index subclasses FrozenNDarray which is an ndarray subclass. The idea would be to do a has-a,similar to what we did with Series. So make the Index have an attribute, prob _data which holds the data. Would need to add some compat methods and delegate other methods to _data.

The pickle problem is solved same way as solved the Series problem (via pickle_compat.py, where we fake the read back class if needed and just recreate in the new scheme).

[jtratner]

@jreback okay, that makes sense.

[jtratner]

Just running through this in my head - is this what you'd expect @y-p or others?

Handling Levels and Labels

Are you thinking that the outputted levels and labels are supposed to be equivalent? (given that MI levels appear to be sorted and labels are 0-indexed)?

>>> arr = [2, 3, 2, 7]
>>> ind = Index(arr)
>>> mi = MultiIndex.from_arrays([arr, [1, 1, 1, 1]])
>>> assert all(mi.levels[0] == ind.levels[0])
>>> assert all(mi.labels[0] == ind.labels[0])

>>> ind = Index(list('abcde'))
>>> ind.levels
[['a', 'b', 'c', 'd', 'e']]
>>> ind.labels
[[0, 1, 2, 3, 4, 5]]

This particularly applies when you have duplicates.

>>> ind = Index(list('abcaadec'))
>>> ind.levels
[['a', 'b', 'c', 'd', 'e']]
>>> ind.labels
[[0, 1, 2, 0, 0, 3, 4, 2]]

and for Int64Index, levels and labels may be the same

>>> ind = Index(range(10))
>>> ind.levels
[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]]
>>> ind.labels
[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]]

but again, ambiguity with duplicates and ordering

>>> ind = Index([10, 0, 1, 2])
>>> ind.levels
[[0, 1, 2, 10]]
>>> ind.labels
[[3, 0, 1, 2]]

[jreback]

@jtratner
I would move this to the top of this issue (easier to see).

[jtratner]

@jreback I will, but can you tell me whether I'm right about how this should work? :P

[jreback]

I think easiest way to do this would be to copy all/bunch of multi-index tests and just substitue an Index there to see what happens/breaks. the idea would be that from a duck perspective it would walk the same as far as methods go

(which I think you have enumerated)

I think levels/labels are wrong on an Index now, in that levels should be treated as if it has 1 level (but not the case now)

[jtratner]

@jreback right, but question is about sorting... probably the stopgap solution is to roundtrip through MultiIndex constructor when levels and labels are asked for.

what jeff said.