Masking broken (or at least different) for Int64 Series

[vanschelven]

Code Sample, a copy-pastable example if possible

>>> import pandas as pd                                                                                         
>>> df = pd.DataFrame({"a": [3, 4], "b": [5, 6]}).astype({"a": "int64", "b": "Int64"})                          

>>> df["a"]                                                                                                     
0    3
1    4
Name: a, dtype: int64

>>> df["b"]                                                                                                     
0    5
1    6
Name: b, dtype: Int64

>>> mask = pd.Series(False, index=df.index)                                                                     
>>> df.loc[mask, "a"] = df["a"]                                                                                 
>>> df.loc[mask, "b"] = df["b"]  
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/lib/python3.6/site-packages/pandas/core/indexing.py", line 190, in __setitem__
    self._setitem_with_indexer(indexer, value)
  File "/lib/python3.6/site-packages/pandas/core/indexing.py", line 605, in _setitem_with_indexer
    setter(labels[0], value)
  File "/lib/python3.6/site-packages/pandas/core/indexing.py", line 539, in setter
    s._data = s._data.setitem(indexer=pi, value=v)
  File "/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 510, in setitem
    return self.apply('setitem', **kwargs)
  File "/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 395, in apply
    applied = getattr(b, f)(**kwargs)
  File "/lib/python3.6/site-packages/pandas/core/internals/blocks.py", line 1751, in setitem
    check_setitem_lengths(indexer, value, self.values)
  File "/lib/python3.6/site-packages/pandas/core/indexing.py", line 2534, in check_setitem_lengths
    raise ValueError("cannot set using a list-like indexer "
ValueError: cannot set using a list-like indexer with a different length than the value

Problem description

I would not expect the type of the Series to affect the masking behavior.

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line]
INSTALLED VERSIONS

commit: None
python: 3.6.8.final.0
python-bits: 64
OS: Linux
OS-release: 4.19.43
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.2
pytest: 4.2.1
pip: None
setuptools: 40.8.0
Cython: None
numpy: 1.16.4
scipy: 1.2.1
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.8.0
pytz: 2018.9
blosc: None
bottleneck: 1.2.1
tables: 3.4.4
numexpr: 2.6.9
feather: None
matplotlib: None
openpyxl: 2.6.0
xlrd: 1.2.0
xlwt: 1.3.0
xlsxwriter: None
lxml.etree: 4.3.1
bs4: 4.7.1
html5lib: 1.0.1
sqlalchemy: 1.2.14
pymysql: None
psycopg2: None
jinja2: 2.10.1
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

[vanschelven]

On pandas 0.25.1 I'm having exactly the same problem

[jorisvandenbossche]

@vanschelven very late reply, but thanks a lot for the clear report!
That's indeed a bug in indexing with the new nullable type (or maybe with extension dtypes in general)

[TomAugspurger]

Seems to be working on master. A test in pandas/tests/frame/indexing/test_setitem.py would be welcome.

[Marvzinc]

When diving into this found the following bug.
Running:

import pandas as pd

df = pd.DataFrame({"a": [3,4]}).astype("Int64")
mask = pd.Series(False, index=df.index)                                                                     
df.loc[mask, "a"] = df["a"]

Gives:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pandas/pandas/core/indexing.py", line 668, in __setitem__
    iloc._setitem_with_indexer(indexer, value)
  File "/home/pandas/pandas/core/indexing.py", line 1792, in _setitem_with_indexer
    self.obj._mgr = self.obj._mgr.setitem(indexer=indexer, value=value)
  File "/home/pandas/pandas/core/internals/managers.py", line 530, in setitem
    return self.apply("setitem", indexer=indexer, value=value)
  File "/home/pandas/pandas/core/internals/managers.py", line 402, in apply
    applied = getattr(b, f)(**kwargs)
  File "/home/pandas/pandas/core/internals/blocks.py", line 1685, in setitem
    check_setitem_lengths(indexer, value, self.values)
  File "/home/pandas/pandas/core/indexers.py", line 152, in check_setitem_lengths
    "cannot set using a list-like indexer "
ValueError: cannot set using a list-like indexer with a different length than the value

While the same example, but with an additional column, b:

df = pd.DataFrame({"a": [3,4], "b": [5,6]}).astype("Int64")
mask = pd.Series(False, index=df.index)                                                                     
df.loc[mask, "a"] = df["a"]    

Does not give any error, which is the desired behaviour.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 80ba4c4
python : 3.7.7.final.0
python-bits : 64
OS : Linux
OS-release : 4.19.76-linuxkit
Version : #1 SMP Tue May 26 11:42:35 UTC 2020
machine : x86_64
processor :
byteorder : little
LC_ALL : C.UTF-8
LANG : C.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.1.0.dev0+1900.g80ba4c429
numpy : 1.18.5
pytz : 2020.1
dateutil : 2.8.1
pip : 20.1.1
setuptools : 45.2.0.post20200210
Cython : 0.29.20
pytest : 5.4.3
hypothesis : 5.16.2
sphinx : 3.1.1
blosc : None
feather : None
xlsxwriter : 1.2.9
lxml.etree : 4.4.1
html5lib : 1.0.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.15.0
pandas_datareader: None
bs4 : 4.9.1
bottleneck : 1.3.2
fastparquet : 0.4.0
gcsfs : None
matplotlib : 3.2.1
numexpr : 2.7.1
odfpy : None
openpyxl : 3.0.3
pandas_gbq : None
pyarrow : 0.16.0
pytables : None
pyxlsb : None
s3fs : 0.4.2
scipy : 1.4.1
sqlalchemy : 1.3.17
tables : 3.6.1
tabulate : 0.8.7
xarray : 0.15.1
xlrd : 1.2.0
xlwt : 1.3.0
numba : 0.48.0

[phofl]

cc @jbrockmendel this is a split path problem. Will probably be fixed with your upcoming pr concerning this

[jbrockmendel]

At first glance it looks like an EA2D problem that is hidden because of separate split-path logic.