BUG: assignment fails with copy_on_write = True

[kameamea]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

pd.options.mode.copy_on_write = True
# pd.options.mode.copy_on_write = "warn"
dftest = pd.DataFrame({"A":[1,4,1,5], "B":[2,5,2,6], "C":[3,6,1,7]})
df=dftest[["B","C"]]
df.iloc[[1,3],:] = [[2, 2],[2 ,2]]

Issue Description

The result is the following error output:

Traceback (most recent call last):
File "/home/rolf/jupyter/venv/lib/python3.12/site-packages/pandas/core/internals/blocks.py", line 1429, in setitem
values[indexer] = casted
~~~~~~^^^^^^^^^
ValueError: shape mismatch: value array of shape (2,2) could not be broadcast to indexing result of shape (2,)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/rolf/code/python/bug.py", line 7, in
df.iloc[[1,3],:] = [[2, 2],[2 ,2]]
~~~~~~~^^^^^^^^^
File "/home/rolf/jupyter/venv/lib/python3.12/site-packages/pandas/core/indexing.py", line 911, in setitem
iloc._setitem_with_indexer(indexer, value, self.name)
File "/home/rolf/jupyter/venv/lib/python3.12/site-packages/pandas/core/indexing.py", line 1944, in _setitem_with_indexer
self._setitem_single_block(indexer, value, name)
File "/home/rolf/jupyter/venv/lib/python3.12/site-packages/pandas/core/indexing.py", line 2218, in _setitem_single_block
self.obj._mgr = self.obj._mgr.setitem(indexer=indexer, value=value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rolf/jupyter/venv/lib/python3.12/site-packages/pandas/core/internals/managers.py", line 409, in setitem
self.blocks[0].setitem((indexer[0], np.arange(len(blk_loc))), value)
File "/home/rolf/jupyter/venv/lib/python3.12/site-packages/pandas/core/internals/blocks.py", line 1432, in setitem
raise ValueError(
ValueError: setting an array element with a sequence.

Expected Behavior

Should not give an error. When uncommenting the copy_on_write="warn" line the code runs with no error message as expected (I'd expect a warning in this case, but thats a separate issue)

Installed Versions

INSTALLED VERSIONS ------------------ commit : 0691c5c python : 3.12.3 python-bits : 64 OS : Linux OS-release : 6.8.0-47-generic Version : #47-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 27 21:40:26 UTC 2024 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : de_DE.UTF-8 LOCALE : de_DE.UTF-8

pandas : 2.2.3
numpy : 2.0.2
pytz : 2024.2
dateutil : 2.9.0.post0
pip : 24.0
Cython : None
sphinx : None
IPython : 8.27.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.4
lxml.etree : None
matplotlib : 3.9.2
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : None
pyreadstat : None
pytest : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.14.1
sqlalchemy : 2.0.36
tables : None
tabulate : None
xarray : None
xlrd : None
xlsxwriter : None
zstandard : None
tzdata : 2024.2
qtpy : None
pyqt5 : None

[sooooooing]

take

[jorisvandenbossche]

@kameamea thanks a lot for testing the CoW mode and for the report! I can confirm that I see the same for 2.2 (it works fine without enabling CoW, but gives that error when enabled), and it also appears on the main development branch for 3.0. So something we should definitely fix.

[jorisvandenbossche]

FWIW I think is related to #51435 (comment), i.e. the fact that we pass integer indices instead of a slice(None) (:) to the underlying method, which ends up using this for indexing the numpy array, which fails in this case.

[bobcorthals]

FYI. Remarkably, the reported error is raised only on the first call. Demonstration of this behavior (pandas 2.2.3):

import pandas as pd
# 2.2.3

pd.options.mode.copy_on_write = True
dftest = pd.DataFrame({"A":[1,4,1,5], "B":[2,5,2,6], "C":[3,6,1,7]})
df=dftest[["B","C"]]
try:
    print('try: I failed\n')
    df.iloc[[1,3],:] = [[2, 2],[2 ,2]]
except:
    df.iloc[[1,3],:] = [[2, 2],[2 ,2]]
    print('except: I worked\n')
    
print(df)

Prints:

try: I failed

except: I worked

   B  C
0  2  3
1  2  2
2  2  1
3  2  2

[jorisvandenbossche]

@bobcorthals I think the reason for that is because the initial dataframes starts under the hood with the three columns stored in a single "block" (a single 2d numpy array). Then in the first assignment (which fails) it splits the block, and then the second time you try the assignment, because the block is already split at that point, it takes a different code path that does not have the bug.

[chilin0525]

Hi @jorisvandenbossche and @bobcorthals, I retested on the main branch and got the same error message output when executing with and without the pd.options.mode.copy_on_write = True setting. It seems like we should pass the slice(None) parameter for BlockManager?

pandas/pandas/core/internals/managers.py

Line 575 in 6bcd303

self.blocks[0].setitem((indexer[0], np.arange(len(blk_loc))), value)

The following is some testing result in main branch codebase:

• Without pd.options.mode.copy_on_write = True

  import pandas as pd
  
  # pd.options.mode.copy_on_write = True
  dftest = pd.DataFrame({"A":[1,4,1,5], "B":[2,5,2,6], "C":[3,6,1,7]})
  df = dftest[["B","C"]]
  
  df.iloc[[1,3], :] = [[2, 2],[2 ,2]]
  print(dftest)
  print(df)

  • output:

    Traceback (most recent call last):
      File "/home/pandas/pandas/core/internals/blocks.py", line 1135, in setitem
        values[indexer] = casted
    ValueError: shape mismatch: value array of shape (2,2) could not be broadcast to indexing result of shape (2,)
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/home/pandas/test.py", line 9, in <module>
        df.iloc[[1,3], :] = [[2, 2],[2 ,2]]
      File "/home/pandas/pandas/core/indexing.py", line 920, in __setitem__
        iloc._setitem_with_indexer(indexer, value, self.name)
      File "/home/pandas/pandas/core/indexing.py", line 1940, in _setitem_with_indexer
        self._setitem_single_block(indexer, value, name)
      File "/home/pandas/pandas/core/indexing.py", line 2204, in _setitem_single_block
        self.obj._mgr = self.obj._mgr.setitem(indexer=indexer, value=value)
      File "/home/pandas/pandas/core/internals/managers.py", line 578, in setitem
        self.blocks[0].setitem((indexer[0], np.arange(len(blk_loc))), value)
      File "/home/pandas/pandas/core/internals/blocks.py", line 1138, in setitem
        raise ValueError(
    ValueError: setting an array element with a sequence.
    

• With pd.options.mode.copy_on_write = True

  import pandas as pd
  
  pd.options.mode.copy_on_write = True
  dftest = pd.DataFrame({"A":[1,4,1,5], "B":[2,5,2,6], "C":[3,6,1,7]})
  df = dftest[["B","C"]]
  
  df.iloc[[1,3], :] = [[2, 2],[2 ,2]]
  print(dftest)
  print(df)

  • output:

    Traceback (most recent call last):
      File "/home/pandas/pandas/core/internals/blocks.py", line 1135, in setitem
        values[indexer] = casted
    ValueError: shape mismatch: value array of shape (2,2) could not be broadcast to indexing result of shape (2,)
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/home/pandas/test.py", line 9, in <module>
        df.iloc[[1,3], :] = [[2, 2],[2 ,2]]
      File "/home/pandas/pandas/core/indexing.py", line 920, in __setitem__
        iloc._setitem_with_indexer(indexer, value, self.name)
      File "/home/pandas/pandas/core/indexing.py", line 1940, in _setitem_with_indexer
        self._setitem_single_block(indexer, value, name)
      File "/home/pandas/pandas/core/indexing.py", line 2204, in _setitem_single_block
        self.obj._mgr = self.obj._mgr.setitem(indexer=indexer, value=value)
      File "/home/pandas/pandas/core/internals/managers.py", line 578, in setitem
        self.blocks[0].setitem((indexer[0], np.arange(len(blk_loc))), value)
      File "/home/pandas/pandas/core/internals/blocks.py", line 1138, in setitem
        raise ValueError(
    ValueError: setting an array element with a sequence.