REGR: __setitem__ with integer slices on Int/RangeIndex is broken (label instead of positional)

[amueller]

There's an backward incompatible change in pandas 1.0 that I didn't find in the changelog. I might have just overlooked it though.

import numpy as np
X = pd.DataFrame(np.zeros((100, 1)))
X[-4:] = 1
X

In pandas 0.25.3 or lower, this results in the last four entries of X to be 1 and all the others zero. In pandas 1.0, it results in all entries of X being 1.
I assume it's a change of indexing axis 0 or axis 1?

[amueller]

I wonder if it's related to #31449 but I'm not using a multi-index.

[MarcoGorelli]

Thanks for the report.

Seems this doesn't affect .iloc:

In [26]: import numpy as np 
    ...: X = pd.DataFrame(np.zeros((5, 1))) 
    ...: X.iloc[-4:] = 1 
    ...: X                                                                      
Out[26]: 
     0
0  0.0
1  1.0
2  1.0
3  1.0
4  1.0

will look into it

[jreback]

you are label indexing with a slice with loc
since none of the labels exist nothing is set

did this actually work previously?

this should never have worked with .loc

it might have with [] which has fallback integer indexing

[jorisvandenbossche]

I do not remember any specific discussion about this, so I think it is definitely a regression.

Slicing rows in [] has always worked positional if there is an integer index (surprising, yes, but longstanding behaviour, see eg my summary of this of 5 years ago #9595)

[MarcoGorelli]

since none of the labels exist nothing is set

@jreback if I've understood correctly, the issue is that everything is being set

>>> import numpy as np
>>> import pandas as pd

>>> X = pd.DataFrame(np.zeros((5, 1)))
>>> X                                                                     
     0
0  0.0
1  0.0
2  0.0
3  0.0
4  0.0

>>> X[-4:]  # only prints the last 4 rows of X...
     0
1  0.0
2  0.0
3  0.0
4  0.0

>>> X[-4:] = 1
>>> X  # ...but everything (including the first row) has now been set
     0
0  1.0
1  1.0
2  1.0
3  1.0
4  1.0

[jreback]

I do not remember any specific discussion about this, so I think it is definitely a regression.

Slicing rows in [] has always worked positional if there is an integer index (surprising, yes, but longstanding behaviour, see eg my summary of this of 5 years ago #9595)

maybe but indexing with an out or range label on both sides should return nothing

so the results are correct

[jreback]

since none of the labels exist nothing is set

@jreback if I've understood correctly, the issue is that everything is being set

>>> import numpy as np
>>> import pandas as pd

>>> X = pd.DataFrame(np.zeros((5, 1)))
>>> X                                                                     
     0
0  0.0
1  0.0
2  0.0
3  0.0
4  0.0

>>> X[-4:]  # only prints the last 4 rows of X...
     0
1  0.0
2  0.0
3  0.0
4  0.0

>>> X[-4:] = 1
>>> X  # ...but everything (including the first row) has now been set
     0
0  1.0
1  1.0
2  1.0
3  1.0
4  1.0

ahh ok that is not correct; i would expect this indexer to return noting

[jreback]

might be #31393

[amueller]

ahh ok that is not correct; i would expect this indexer to return noting

Asking for the shape, both in 0.25 and 1.0, you get

>>> X[-4:].shape
(4, 1)

but assignment in version 1.0 assigns to everything.