DEPR: Deprecate downcast keyword for fillna

[phofl]

This came up recently in #40861
Depending on the way forward this would allow us to get downcast out from Block or be stricter about it

Thougts?

cc @jbrockmendel @jreback

[jbrockmendel]

The more I look at this, the wonkier it seems. #40861 implemented a test case:

df = pd.DataFrame({"col1": [1, np.nan]})

df.fillna({"col1": 2}, downcast={"col1": "int64"})

But similar calls still raise:

>>> df.fillna({"col1": 2}, downcast={"col1": np.int64})
[...]
TypeError: <class 'numpy.int64'>

>>> df.fillna(2, downcast={"col1": "int64"})
[...]
AssertionError: dtypes as dict is not supported yet

>>> df.fillna(2, downcast="int64")
[...]
ValueError: downcast must have a dictionary or 'infer' as its argument

Moreover, there are several places in NDFrame.fillna and Block.fillna where we ignore the 'downcast' keyword so end up using None. There's also a couple of layers of special-casing in Block._maybe_downcast and Block.downcast.

Notwithstanding the places where the downcast keyword is ignored, the status quo is roughly:

if downcast is None:
    if self.dtype == object:
        return datetime_centric_special_case()
    elif self.dtype.kind in ["f", "m", "M"]:
        return no_op()
    elif self.ndim == 1:
        downcast  = "infer"
        return do_inference()
    else:
        return no_op()

else:
    if self.dtype == object:
        return no_op()
    elif self.ndim == 1:
        return one_case_where_we_actually_respect_downcast_keyword()
    elif downcast != "infer":
          raise ValueErrorOrAssertionError
     else:
        return do_inference()

Some of this we can improve by actually respecting the keyword in more places and call that a bugfix. But the rest of the special-casing (object dtype, ndim==1 vs ndim==2) will likely change things and need a deprecation.

AFAICT the useful cases for downcasting after fillna (and interpolate, the other places from which _maybe_downcast is called) are 1) float->int and 2) object->not_object.

If this were greenfield, my inclination would be to make downcast take a bool, with False meaning no downcasting and True meaning "infer". (we could also take a dict but it would get unpacked before making it to the Block methods)

[jbrockmendel]

xref #11537

[jbrockmendel]

More questionable behavior: 1) we do the downcast even if the fillna is a no-op, 2) we ignore inplace=True and downcast anyway.

[jbrockmendel]

Based on a few days of trying to figure this out, along with #11537, I propose we deprecate the 'downcast' keyword in 'fillna', 'interpolate', 'ffill', 'bfill', with the future behavior being to always downcast for object dtype and never for anything else (the only relevant case for "anything else" being floats)

[jreback]

sounds ok to me

[jbrockmendel]

Having some trouble with the "do X instead" part because we don't have a method for downcasting floats (do we?)

[jreback]

we don't downcast floats as a general rule

[jbrockmendel]

update: deprecating the argument seems increasingly invasive. Leaning towards chipping away at the inconsistencies and calling them bugfixes.