API: behaviour of label indexing with floats on integer index

[jreback]

from #12246

@jorisvandenbossche this looks odd

In [11]: s = pd.Series([1,2,3])

In [12]: s.loc[0.0] = 10

In [13]: s.loc[0.0]
TypeError: cannot do label indexing on <class 'pandas.indexes.range.RangeIndex'>
 with these indexers [0.0] of <type 'float'>

[jorisvandenbossche]

I think there is clear agreement about the following:

• when doing integer positional indexing, using floats are not allowed.
  Eg:

  In [22]: s = pd.Series([1,2,3])
  
  In [23]: s.iloc[0.0]
  TypeError: cannot do positional indexing on <class 'pandas.indexes.range.RangeIn
  dex'> with these indexers [0.0] of <type 'float'>
  
  In [24]: s2 = pd.Series([1,2,3], index=['a', 'b', 'c'])
  
  In [25]: s2[0.0]
  TypeError: cannot do label indexing on <class 'pandas.indexes.base.Index'> with
  these indexers [0.0] of <type 'float'>
  

  This was deprecated and now errors after Jeff's PR.

• when doing label indexing on a non-numerical index (this is logical, but just to be complete)

There is less agreement on when floats should be allowed when doing label indexing on a numerical but not-float index (so integer index or RangeIndex) in case the float can be interpreted as an integer:

• Previously, these were interpreted as integers, but raised a deprecation warning

• In current master, when setting, the float will be coerced to an integer when possible:

  In [38]: s = pd.Series([1,2,3])
  
  In [39]: s[1.0] = 10
  
  In [40]: s
  Out[40]:
  0     1
  1    10
  2     3
  dtype: int64
  

• But (also in current master), when getting a value, this still raises, also for coercables floats:

  In [42]: s[1.0]
  TypeError: cannot do label indexing on <class 'pandas.indexes.range.RangeIndex'>
  with these indexers [1.0] of <type 'float'>
  

  giving the strange contradiction between s[1.0] = 10 and s[1.0] raising

I think for setting, the rule is if the label you want to set evaluates as equal to one of the labels in the index, then it works without changing the index (1 == 1.0 and 1 != '1'). If it does not evaluates to equal, then it is seen as a new label and the index is expanded.

I would argue that the same rule should apply for getting as for setting.
On the other hand, setting is already more flexible (as for example s[1.5] = 10 is allowed and will upcast the index, while getting with this s[1.5] obviously raises).

cc @shoyer @TomAugspurger

[jreback]

The issue I have with @jorisvandenbossche last

I think for setting, the rule is if the label you want to set evaluates as equal to one of the labels in the index, then it works without changing the index (1 == 1.0 and 1 != '1'). If it does not evaluates to equal, then it is seen as a new label and the index is expanded.

as this would only apply to [], .ix and NOT .loc as .loc is strict about labels while the others can do positional AND/OR labels.

But since this in an integer index we are talking about we have the odd fallback where it is treated like a label (.ix), so this would be inconsistent then? (of course I may be misrepresenting things here as fallback is simply confusing).

[TomAugspurger]

as this would only apply to [], .ix and NOT .loc as .loc is strict about labels while the others can do positional AND/OR labels.

We could say that a python dict is also strict about labels, but {1: 'one'}[1.0] goes through fine.

I think I'm in agreement with Joris; haven't though how it affects fallback indexing entirely yet.

[jorisvandenbossche]

AFAIK, integer indexes don't have any fallback to positional, and the indexer is always interpreted as label. So I don't see how this could mess up with the fallback indexing.

.loc is strict about labels while the others can do positional AND/OR labels.

I don't see why that is a reason to make .loc have different behaviour, as we are speaking here only about label indexing, not positional.

[jorisvandenbossche]

You also have this discrepancy:

In [71]: 1.0 in s.index
Out[71]: True

In [72]: s[1.0]
TypeError: cannot do label indexing on <class 'pandas.indexes.numeric.Int64Index
'> with these indexers [1.0] of <type 'float'>

(to be honest, it would only be a full discrepancy if this gave a "KeyError 1.0 not found in index")

[jreback]

ok, so @jorisvandenbossche what do you think should work then. give me an example for setting/getting with the various indexers.

[shoyer]

I agree with @jorisvandenbossche, as I stated in the last PR:

• __setitem__ and __getitem__ should be entirely consistent
• We should use equality (like the Python dict) to determine whether or not a key matches
• If inserting a key that doesn't exist, then the index type is upcast if necessary

[jreback]

so @shoyer then essentially you want to have full support back for float indexers? (excluding .iloc)

thats what your conditions imply.

[shoyer]

@jreback Yes, I guess so. If pandas were stricter about not upcasting types (or even better, if we did not support reindexing with .loc at all), then it could make sense to make the distinction. But as is, I think we should be OK with float indexers. Obviously float indexers that don't match an integer will still result in KeyError, though

[jreback]

I suppose this means we are reverting this as well.

In [1]: s = Series(range(3))

In [2]: s
Out[2]: 
0    0
1    1
2    2
dtype: int64

In [3]: s[1.0:2]
TypeError: cannot do slice start value indexing on <class 'pandas.indexes.range.RangeIndex'> with these indexers [1.0] of <type 'float'>

and have it return equiv of s[1:2] (and allow setting).

Note that this will only apply to integer-like indices as string index will still raise TypeError.