API: capabilities of df.set_index

[h-vetinari]

This is coming out of a discussion that has stalled #22225 (which is about adding .set_index to Series, see #21684). The discussion has shifted away from what capabilities a putative Series.set_index should have, but what capabilities df.set_index has currently.

The main issue (for @jreback) is that df.set_index takes arrays:

@jreback: There were several attempts to have DataFrame.set_index take an array as well, but these never got off the ground.

@h-vetinari: I'm not sure when, but they certainly did get off the ground:

>>> import pandas as pd
>>> import numpy as np
>>> pd.__version__
'0.23.4'
>>>
>>> df = pd.DataFrame(np.random.randint(0, 10, (4, 4)), columns=list('abcd'))
>>> df.set_index(['a',          # label
...               df.index,     # Index
...               df.b ** 2,    # Series
...               df.b.values,  # ndarray
...               list('ABCD'), # list
...               'c'])         # label again
              b  d
a   b      c
0 0 0  2 A 1  0  2
8 1 1  4 B 4  1  4
3 2 25 5 C 8  5  5
0 3 9  7 D 2  3  7

Further on:

@jreback: @h-vetinari you are confusing the purpose of .set_axis. [...] The problem with .set_index on a DataFrame with an array is that it technically can work with an array and not keys. (meaning its not unambiguous)

I don't think I am confusing them. If I want to set the .index-attribute of a Series/DataFrame, then using .set_index is the most reasonable name by far. If anything, set_axis should be a superset of set_index (and a putative set_columns), that just switches between the two based on the axis-kwarg.

More than that, the current capabilities of df.set_index are a proper superset of df.set_axis(axis=0)**, in that it's possible to fill keys with only Series/Index/ndarray/list etc.:

>>> df.set_index(pd.Index(df.a))  # same result as Series directly below
>>> df.set_index(df.a) 
   a  b  c  d
a
0  0  0  1  2
8  8  1  4  4
3  3  5  8  5
0  0  3  2  7
>>> df.set_index(df.a.values)  # same result as list directly below
>>> df.set_index([[0, 8, 3, 0]])
   a  b  c  d
0  0  0  1  2
8  8  1  4  4
3  3  5  8  5
0  0  3  2  7

** there is one caveat, in that lists (and only lists; out of all containers) need to be wrapped in another list, i.e. df.set_index([[0, 8, 3, 0]]) instead of df.set_index([0, 8, 3, 0]). This is the heart of the ambiguity that @jreback mentioned above (because a list is interpreted as a list of column keys).

Summing up:

• set_index is the most natural name for setting the .index-attribute
• df.set_index should be able to process list-likes (as it currently does; this is the source of the ambiguity of the list case).
• df.set_axis should be able to do everything that df.set_index does, and just switch between operating on index/columns based on the axis-kwarg (after all, index and columns are the two axes of a DF).
  • it could be considered to add a method set_columns on a DataFrame
  • The axis-kwarg of set_axis should just switch between the behaviour of set_index (i.e. dealing with keys and array-likes) and set_columns.
• Series.set_index should support the same signature as df.set_index, with the exception of the drop-keyword (which only makes sense for column labels).
• For Series, the set_index and set_axis methods should be exactly the same.

Since I can't tag @pandas-dev/pandas-core, here are a few individual tags: @jreback @TomAugspurger @jorisvandenbossche @gfyoung @WillAyd @jbrockmendel @jschendel @toobaz.

EDIT: Forgot to add an xref from @jreback:

@h-vetinari we had quite some discussion about this: #14829
and never reached resolution. This is an API question.

In that issue, there's discussion largely around .rename, and how to make that method more consistent. Also discussed was potentially introducing .relabel, as well as .set_columns.

[jbrockmendel]

@h-vetinari should list('ABC') in the first example be list(ABCD')? If not then I am confused in several directions.

[h-vetinari]

@jbrockmendel
That was indeed an artefact from merging together several things from the other thread to make this issue...

[h-vetinari]

@jreback Any comments here or in #22225?

[h-vetinari]

@jreback
I am honestly stunned by you closing #22225 and then locking it after I objected. So much for my motivation to work on some big PRs today.

@h-vetinari you are not listening. If you want to raise an issue or comment feel free.

I've opened this issue here for exactly this purpose (discussing your objections to existing capabilities of DataFrame.set_index) over a month ago.

[toobaz]

@h-vetinari While I think locking #22225 was an unnecessary move from @jreback , you have to realize that the "''overruling approving reviews''" thing is not a good argument to raise in such a discussion. True, in pandas we look for devs consensus, but in the end we prefer to do so by argumenting rather than by any form of proper vote. So that comment was not very useful, to use an euphemism.

Back to the topic of the discussion: if I understand correctly, @jreback is argumenting that df.set_index is already ambiguous enough to make it a bad idea to sponsor its use when passing anything but keys (which would be the only sensible use in Series.set_index); and at the same time, you are suggesting that

[ ] For Series, the set_index and set_axis methods should be exactly the same.

If this summary of the discussion is correct, then I think I am also against introducing Series.set_index.

And actually, I'm probably in favor of deprecating df.set_index to pass actual values, if df.set_axis is able to fulfill exactly the same task (didn't check).

I understand you argument about df.set_index being what one expects to use to set the .index (to some values)... but set_axis is simply too long established to be removed, and I find duplication in the API a worse problem than the maybe sub-optimal naming. Or in other words, I think

The axis-kwarg of set_axis should just switch between the behaviour of set_index (i.e. dealing with keys and array-likes) and set_columns.

is a bad idea. ''There should be one - and preferably only one - obvious way to do it."

EDIT: By the way: sorry for not reacting before to the ping - busy period.

[jorisvandenbossche]

@h-vetinari In my opinion, the locking was not the best way to handle the discussion in the PR, so sorry about that. In the meantime, Jeff has unlocked the conversation there, but let's continue the discussion here.

---

On the topic: given the behaviour of DataFrame.set_index (supporting setting a full array-like in addition of a list of column names), I personally don't have problems with adding a similar behaviour for Series.set_index.

I'm probably in favor of deprecating df.set_index to pass actual values, if df.set_axis is able to fulfill exactly the same task .....
.... but set_axis is simply too long established to be removed, and I find duplication in the API a worse problem than the maybe sub-optimal naming

@toobaz I personally almost never seen someone use set_axis (and never used it myself), but I do regularly see people use set_index with a full Series/array (but that's subjective of course).
So personally, I would rather deprecate set_axis and only keep set_index (but: set_axis also has the ability to set the columns, which set_index cannot do, so it is not fully duplicative and therefore probably cannot easily be deprecated).

[TomAugspurger]

Agreed that locking was not appropriate.

---

On the issue itself, to me it's pretty clear that Series.set_index(sequence) is a limiting case of DataFrame.set_index(Sequece[sequence])

In [1]: import pandas as pd

In [2]: df = pd.DataFrame({"A": [1, 2, 3]})

In [3]: df.set_index([['a', 'b', 'c']])
Out[3]:
   A
a  1
b  2
c  3

Since In[3] works, I would expect that

In [4]: df.A.set_index([['a', 'b', 'c']])

work as well.

[WillAyd]

I don't think continuing to post here or in the associated PR is an effective use of anyone's time. Why don't we just add it as a discussion point to the next dev chat?

[TomAugspurger]

The implementation is fine, and deserves to go in 0.24.0 if we can agree on the desired behavior. No need to delay I think.

…

On Mon, Jan 7, 2019 at 10:09 AM William Ayd ***@***.***> wrote: I don't think continuing to post here or in the associated PR is an effective use of anyone's time. Why don't we just add it as a discussion point to the next dev chat? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#24046 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIrAEcEJoHmYf7ZMXGa7rvVpAlYUFks5vA3EtgaJpZM4Y9ZRa> .