PDEP6 implementation pt 2: eablock.setitem, eablock.putmask (#53405)

* pt1

* fixup test collection

* fixup warnings

* add comments

* fixup warnings

* fixup test_indexing

* fixup test_set_value

* fixup test_where

* fixup test_asof

* add one more explicit upcast

* fixup test_update

* fixup test_constructors

* fixup test_stack_unstack

* catch warnings in test_update_dtypes

* fixup all test_update

* start fixing up setitem

* finish fixing up test_setitem

* more fixups

* catch numpy-dev warning

* fixup some more

* fixup test_indexing

* fixup test_function

* fixup test_multi;

* fixup test_base

* fixup test_impl

* fixup multiindex/test_setitem

* fixup test_scalar

* fixup test_loc

* fixup test_iloc

* fixup test_at

* fixup test_groupby

* fixup some doc warnings

* post-merge fixup

* change dtype in doctest

* fixup doctest

* explicit cast in test

* fixup test for COW

* fixup COW

* catch warnings in testsetitemcastingequivalents

* wip

* fixup setitem test int key!

* getting there!

* fixup test_setitem

* getting there

* fixup remaining warnings

* fix test_update

* fixup some failing test

* one more

* simplify

* simplify and remove some false-positives

* clean up

* remove final filterwarnings

* undo unrelated change

* fixup raises_chained_assignment_error

* remove another filterwarnings

* fixup interchange test

* better parametrisation

* okwarning => codeblock

* okwarning => codeblock in v1.3.0

* one more codeblock

* avoid upcast

* post-merge fixup

* docs fixup;

* post-merge fixup

* remove more upcasts

* adapt test from EA types

* move test to series/indexing

* add tests about warnings

* fixup tests

* add dataframe tests too

* fixup tests

* simplify

* try-fix docs build

* post-merge fixup

* raise assertionerror if self.dtype equals new_dtype

* add todo for test case which should warn

* add more todos

* post-merge fixup

* fixup setitem

* fixup

* wip fixup

* wip fixup

* another fixup

* add whatsnew

* list examples of operations

---------

Co-authored-by: MarcoGorelli <>

Author: MarcoGorelli

doc/source/whatsnew/v0.21.0.rst

@@ -783,10 +783,15 @@ non-datetime-like item being assigned (:issue:`14145`).
 
 These now coerce to ``object`` dtype.
 
-.. ipython:: python
+.. code-block:: python
 
-   s[1] = 1
-   s
+   In [1]: s[1] = 1
+
+   In [2]: s
+   Out[2]:
+   0    2011-01-01 00:00:00
+   1                      1
+   dtype: object
 
 - Inconsistent behavior in ``.where()`` with datetimelikes which would raise rather than coerce to ``object`` (:issue:`16402`)
 - Bug in assignment against ``int64`` data with ``np.ndarray`` with ``float64`` dtype may keep ``int64`` dtype (:issue:`14001`)

doc/source/whatsnew/v2.1.0.rst

@@ -296,8 +296,99 @@ Other API changes
 .. ---------------------------------------------------------------------------
 .. _whatsnew_210.deprecations:
 
-Deprecate parsing datetimes with mixed time zones
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Deprecations
+~~~~~~~~~~~~
+
+Deprecated silent upcasting in setitem-like Series operations
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Setitem-like operations on Series (or DataFrame columns) which silently upcast the dtype are
+deprecated and show a warning. Examples of affected operations are:
+
+  - ``ser.fillna('foo', inplace=True)``
+  - ``ser.where(ser.isna(), 'foo', inplace=True)``
+  - ``ser.iloc[indexer] = 'foo'``
+  - ``ser.loc[indexer] = 'foo'``
+  - ``df.iloc[indexer, 0] = 'foo'``
+  - ``df.loc[indexer, 'a'] = 'foo'``
+  - ``ser[indexer] = 'foo'``
+
+where ``ser`` is a :class:`Series`, ``df`` is a :class:`DataFrame`, and ``indexer``
+could be a slice, a mask, a single value, a list or array of values, or any other
+allowed indexer.
+
+In a future version, these will raise an error and you should cast to a common dtype first.
+
+*Previous behavior*:
+
+.. code-block:: ipython
+
+  In [1]: ser = pd.Series([1, 2, 3])
+
+  In [2]: ser
+  Out[2]:
+  0    1
+  1    2
+  2    3
+  dtype: int64
+
+  In [3]: ser[0] = 'not an int64'
+
+  In [4]: ser
+  Out[4]:
+  0    not an int64
+  1               2
+  2               3
+  dtype: object
+
+*New behavior*:
+
+.. code-block:: ipython
+
+  In [1]: ser = pd.Series([1, 2, 3])
+
+  In [2]: ser
+  Out[2]:
+  0    1
+  1    2
+  2    3
+  dtype: int64
+
+  In [3]: ser[0] = 'not an int64'
+  FutureWarning:
+    Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas.
+    Value 'not an int64' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
+
+  In [4]: ser
+  Out[4]:
+  0    not an int64
+  1               2
+  2               3
+  dtype: object
+
+To retain the current behaviour, in the case above you could cast ``ser`` to ``object`` dtype first:
+
+.. ipython:: python
+
+  ser = pd.Series([1, 2, 3])
+  ser = ser.astype('object')
+  ser[0] = 'not an int64'
+  ser
+
+Depending on the use-case, it might be more appropriate to cast to a different dtype.
+In the following, for example, we cast to ``float64``:
+
+.. ipython:: python
+
+  ser = pd.Series([1, 2, 3])
+  ser = ser.astype('float64')
+  ser[0] = 1.1
+  ser
+
+For further reading, please see https://pandas.pydata.org/pdeps/0006-ban-upcasting.html.
+
+Deprecated parsing datetimes with mixed time zones
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Parsing datetimes with mixed time zones is deprecated and shows a warning unless user passes ``utc=True`` to :func:`to_datetime` (:issue:`50887`)
 
@@ -342,7 +433,7 @@ and ``datetime.datetime.strptime``:
     pd.Series(data).apply(lambda x: dt.datetime.strptime(x, '%Y-%m-%d %H:%M:%S%z'))
 
 Other Deprecations
-~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^
 - Deprecated 'broadcast_axis' keyword in :meth:`Series.align` and :meth:`DataFrame.align`, upcast before calling ``align`` with ``left = DataFrame({col: left for col in right.columns}, index=right.index)`` (:issue:`51856`)
 - Deprecated 'downcast' keyword in :meth:`Index.fillna` (:issue:`53956`)
 - Deprecated 'fill_method' and 'limit' keywords in :meth:`DataFrame.pct_change`, :meth:`Series.pct_change`, :meth:`DataFrameGroupBy.pct_change`, and :meth:`SeriesGroupBy.pct_change`, explicitly call ``ffill`` or ``bfill`` before calling ``pct_change`` instead (:issue:`53491`)

pandas/core/internals/blocks.py

@@ -461,7 +461,7 @@ def coerce_to_target_dtype(self, other, warn_on_upcast: bool = False) -> Block:
                 FutureWarning,
                 stacklevel=find_stack_level(),
             )
-        if self.dtype == new_dtype:
+        if self.values.dtype == new_dtype:
             raise AssertionError(
                 f"Did not expect new dtype {new_dtype} to equal self.dtype "
                 f"{self.values.dtype}. Please report a bug at "
@@ -1723,11 +1723,11 @@ def setitem(self, indexer, value, using_cow: bool = False):
 
             if isinstance(self.dtype, IntervalDtype):
                 # see TestSetitemFloatIntervalWithIntIntervalValues
-                nb = self.coerce_to_target_dtype(orig_value)
+                nb = self.coerce_to_target_dtype(orig_value, warn_on_upcast=True)
                 return nb.setitem(orig_indexer, orig_value)
 
             elif isinstance(self, NDArrayBackedExtensionBlock):
-                nb = self.coerce_to_target_dtype(orig_value)
+                nb = self.coerce_to_target_dtype(orig_value, warn_on_upcast=True)
                 return nb.setitem(orig_indexer, orig_value)
 
             else:
@@ -1841,13 +1841,13 @@ def putmask(self, mask, new, using_cow: bool = False) -> list[Block]:
                 if isinstance(self.dtype, IntervalDtype):
                     # Discussion about what we want to support in the general
                     #  case GH#39584
-                    blk = self.coerce_to_target_dtype(orig_new)
+                    blk = self.coerce_to_target_dtype(orig_new, warn_on_upcast=True)
                     return blk.putmask(orig_mask, orig_new)
 
                 elif isinstance(self, NDArrayBackedExtensionBlock):
                     # NB: not (yet) the same as
                     #  isinstance(values, NDArrayBackedExtensionArray)
-                    blk = self.coerce_to_target_dtype(orig_new)
+                    blk = self.coerce_to_target_dtype(orig_new, warn_on_upcast=True)
                     return blk.putmask(orig_mask, orig_new)
 
                 else:

pandas/tests/frame/indexing/test_indexing.py

@@ -831,7 +831,10 @@ def test_setitem_single_column_mixed_datetime(self):
         tm.assert_series_equal(result, expected)
 
         # GH#16674 iNaT is treated as an integer when given by the user
-        df.loc["b", "timestamp"] = iNaT
+        with tm.assert_produces_warning(
+            FutureWarning, match="Setting an item of incompatible dtype"
+        ):
+            df.loc["b", "timestamp"] = iNaT
         assert not isna(df.loc["b", "timestamp"])
         assert df["timestamp"].dtype == np.object_
         assert df.loc["b", "timestamp"] == iNaT
@@ -862,7 +865,10 @@ def test_setitem_mixed_datetime(self):
         df = DataFrame(0, columns=list("ab"), index=range(6))
         df["b"] = pd.NaT
         df.loc[0, "b"] = datetime(2012, 1, 1)
-        df.loc[1, "b"] = 1
+        with tm.assert_produces_warning(
+            FutureWarning, match="Setting an item of incompatible dtype"
+        ):
+            df.loc[1, "b"] = 1
         df.loc[[2, 3], "b"] = "x", "y"
         A = np.array(
             [

pandas/tests/frame/indexing/test_where.py

@@ -735,7 +735,10 @@ def test_where_interval_fullop_downcast(self, frame_or_series):
         tm.assert_equal(res, other.astype(np.int64))
 
         # unlike where, Block.putmask does not downcast
-        obj.mask(obj.notna(), other, inplace=True)
+        with tm.assert_produces_warning(
+            FutureWarning, match="Setting an item of incompatible dtype"
+        ):
+            obj.mask(obj.notna(), other, inplace=True)
         tm.assert_equal(obj, other.astype(object))
 
     @pytest.mark.parametrize(
@@ -775,7 +778,10 @@ def test_where_datetimelike_noop(self, dtype):
         tm.assert_frame_equal(res5, expected)
 
         # unlike where, Block.putmask does not downcast
-        df.mask(~mask2, 4, inplace=True)
+        with tm.assert_produces_warning(
+            FutureWarning, match="Setting an item of incompatible dtype"
+        ):
+            df.mask(~mask2, 4, inplace=True)
         tm.assert_frame_equal(df, expected.astype(object))
 
 
@@ -930,7 +936,10 @@ def test_where_period_invalid_na(frame_or_series, as_cat, request):
         result = obj.mask(mask, tdnat)
         tm.assert_equal(result, expected)
 
-        obj.mask(mask, tdnat, inplace=True)
+        with tm.assert_produces_warning(
+            FutureWarning, match="Setting an item of incompatible dtype"
+        ):
+            obj.mask(mask, tdnat, inplace=True)
         tm.assert_equal(obj, expected)
 
 
@@ -1006,7 +1015,10 @@ def test_where_dt64_2d():
 
     # setting all of one column, none of the other
     expected = DataFrame({"A": other[:, 0], "B": dta[:, 1]})
-    _check_where_equivalences(df, mask, other, expected)
+    with tm.assert_produces_warning(
+        FutureWarning, match="Setting an item of incompatible dtype"
+    ):
+        _check_where_equivalences(df, mask, other, expected)
 
     # setting part of one column, none of the other
     mask[1, 0] = True
@@ -1016,7 +1028,10 @@ def test_where_dt64_2d():
             "B": dta[:, 1],
         }
     )
-    _check_where_equivalences(df, mask, other, expected)
+    with tm.assert_produces_warning(
+        FutureWarning, match="Setting an item of incompatible dtype"
+    ):
+        _check_where_equivalences(df, mask, other, expected)
 
     # setting nothing in either column
     mask[:] = True

pandas/tests/frame/test_constructors.py

@@ -2549,8 +2549,7 @@ def check_views(c_only: bool = False):
             check_views()
 
         # TODO: most of the rest of this test belongs in indexing tests
-        # TODO: 'm' and 'M' should warn
-        if lib.is_np_dtype(df.dtypes.iloc[0], "fciuOmM"):
+        if lib.is_np_dtype(df.dtypes.iloc[0], "fciuO"):
             warn = None
         else:
             warn = FutureWarning

pandas/tests/indexing/test_at.py

@@ -23,7 +23,8 @@
 def test_at_timezone():
     # https://github.com/pandas-dev/pandas/issues/33544
     result = DataFrame({"foo": [datetime(2000, 1, 1)]})
-    result.at[0, "foo"] = datetime(2000, 1, 2, tzinfo=timezone.utc)
+    with tm.assert_produces_warning(FutureWarning, match="incompatible dtype"):
+        result.at[0, "foo"] = datetime(2000, 1, 2, tzinfo=timezone.utc)
     expected = DataFrame(
         {"foo": [datetime(2000, 1, 2, tzinfo=timezone.utc)]}, dtype=object
     )

pandas/tests/indexing/test_loc.py

@@ -1449,7 +1449,8 @@ def test_loc_setitem_datetime_coercion(self):
         df.loc[0:1, "c"] = np.datetime64("2008-08-08")
         assert Timestamp("2008-08-08") == df.loc[0, "c"]
         assert Timestamp("2008-08-08") == df.loc[1, "c"]
-        df.loc[2, "c"] = date(2005, 5, 5)
+        with tm.assert_produces_warning(FutureWarning, match="incompatible dtype"):
+            df.loc[2, "c"] = date(2005, 5, 5)
         assert Timestamp("2005-05-05").date() == df.loc[2, "c"]
 
     @pytest.mark.parametrize("idxer", ["var", ["var"]])

pandas/tests/internals/test_internals.py

@@ -1312,17 +1312,20 @@ def test_interval_can_hold_element(self, dtype, element):
         # `elem` to not have the same length as `arr`
         ii2 = IntervalIndex.from_breaks(arr[:-1], closed="neither")
         elem = element(ii2)
-        self.check_series_setitem(elem, ii, False)
+        with tm.assert_produces_warning(FutureWarning):
+            self.check_series_setitem(elem, ii, False)
         assert not blk._can_hold_element(elem)
 
         ii3 = IntervalIndex.from_breaks([Timestamp(1), Timestamp(3), Timestamp(4)])
         elem = element(ii3)
-        self.check_series_setitem(elem, ii, False)
+        with tm.assert_produces_warning(FutureWarning):
+            self.check_series_setitem(elem, ii, False)
         assert not blk._can_hold_element(elem)
 
         ii4 = IntervalIndex.from_breaks([Timedelta(1), Timedelta(3), Timedelta(4)])
         elem = element(ii4)
-        self.check_series_setitem(elem, ii, False)
+        with tm.assert_produces_warning(FutureWarning):
+            self.check_series_setitem(elem, ii, False)
         assert not blk._can_hold_element(elem)
 
     def test_period_can_hold_element_emptylist(self):
@@ -1341,11 +1344,13 @@ def test_period_can_hold_element(self, element):
         # `elem` to not have the same length as `arr`
         pi2 = pi.asfreq("D")[:-1]
         elem = element(pi2)
-        self.check_series_setitem(elem, pi, False)
+        with tm.assert_produces_warning(FutureWarning):
+            self.check_series_setitem(elem, pi, False)
 
         dti = pi.to_timestamp("S")[:-1]
         elem = element(dti)
-        self.check_series_setitem(elem, pi, False)
+        with tm.assert_produces_warning(FutureWarning):
+            self.check_series_setitem(elem, pi, False)
 
     def check_can_hold_element(self, obj, elem, inplace: bool):
         blk = obj._mgr.blocks[0]