BUG: IndexError: positional indexers are out-of-bounds iloc boolean indexing

[gooney47]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

import pandas as pd
df = pd.DataFrame([[0, 1, 2]], columns=['a', 'b', 'c'])
mask = pd.DataFrame([[False, True, False]], columns=['a', 'b', 'c'])
df.iloc[mask] = 3 # Works fine with assignment
print(df.iloc[mask]) # Throws IndexError, but should give similar result as df.values[mask]

Problem description

Documentation of Pandas says I can use boolean array with iloc. The iloc call with the assignment works fine, but without assignment fails. Just as a note here: In pandas 1.0.1 not even the assignment works (it starts working again if there are more than 1 rows of data).

Expected Output

3 (similar as df.values[mask])

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit : 3e89b4c python : 3.8.5.final.0 python-bits : 64 OS : Linux OS-release : 5.4.0-58-generic Version : #64-Ubuntu SMP Wed Dec 9 08:16:25 UTC 2020 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8 pandas : 1.2.0 numpy : 1.18.4 pytz : 2019.3 dateutil : 2.7.3 pip : 20.2 setuptools : 46.2.0 Cython : None pytest : None hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : 4.5.0 html5lib : 1.0.1 pymysql : None psycopg2 : 2.8.5 (dt dec pq3 ext lo64) jinja2 : 2.11.2 IPython : 7.19.0 pandas_datareader: None bs4 : 4.8.2 bottleneck : None fsspec : 0.7.3 fastparquet : None gcsfs : None matplotlib : 3.2.1 numexpr : None odfpy : None openpyxl : None pandas_gbq : None pyarrow : None pyxlsb : None s3fs : None scipy : 1.4.1 sqlalchemy : None tables : None tabulate : None xarray : None xlrd : 1.2.0 xlwt : None numba : 0.49.1

[phofl]

Hi, thanks for your report.

Why are you using a DataFrame as mask? I am not sure if this is intended to work

[gooney47]

I'm using a dataframe mask because I have a 2D boolean mask that marks all the places I want to change in my 2D float value dataframe. I was hoping there would be a function for this that prevents me from writing a loop (not for performance, just for convenience). In the documentation it is says it is possible to use "A boolean array (any NA values will be treated as False)." to index via iloc. What I found so confusing about this, is that the indexing actually works if I use an assignment, but it breaks if I only evaluate the indexing operation. As the assigment is something that is done on top of the indexing it makes no sense for me that it would work.

[phofl]

Our fallback for assignments is to operate on arrays. Hence it works.

iloc is two dimensional, but I am not quite sure what you want to achieve. Something like

mask = pd.DataFrame(
[
    [False, True, False],
    [True, False, True],
], columns=['a', 'b', 'c'])

should not be possible. You have to specify rows to select and columns to select.

@jbrockmendel thoughts here? I understood array as numpy array or pandas array not DataFrame.

[gooney47]

@phofl It does work though, check out this:

import pandas as pd
df = pd.DataFrame([
    [0, 1, 2], 
    [3, 4, 5]
], columns=['a', 'b', 'c'])
mask = pd.DataFrame(
[
    [False, True, False],
    [True, False, True],
], columns=['a', 'b', 'c'])
df.iloc[mask] = -1 # Works fine
print(df)

I don't have to specify rows and columns seperatively.

[phofl]

Sorry, was not clear enough. I did not expect

df.iloc[mask]

to work. I don't even know what this should return in your case. Meaning getitem

[gooney47]

I think what numpy does is pretty good, so returning the elements, selected by the mask, in a Series sounds appropriate to me. That would solve two things at once, my concern about it being weird that something is only working if you do an assignment on top of it, plus it provides more functionality.

[jreback]

this is not intended to work at all

iloc does not align and esp does not on a Dataframe

instead this gets turned into a ndarray and can easily be the wrong shape

[phofl]

Maybe we should raise when a DataFrame is given as indexer?

[jbrockmendel]

Maybe we should raise when a DataFrame is given as indexer?

I think this makes sense. Definitely shouldnt be working for iloc.setitem but not iloc.getitem

[jreback]

yep agreed