Skip to content

API: Series[EA].fillna fallback behavior with incompatible value #45153

Open
@jbrockmendel

Description

@jbrockmendel
dti = pd.DatetimeIndex([pd.NaT, "2016-01-01"], tz="UTC")
tdi = dti - dti[1]
pi = dti.tz_localize(None).to_period("D")
ci = pi.astype("category")
ii = pd.IntervalIndex([None])

pd.Series(dti).fillna("foo")  # <- casts
pd.Series(tdi).fillna("foo")  # <- raises
pd.Series(pi).fillna("foo")  # <- casts, but untested
pd.Series(ci).fillna("foo")  # <- raises
pd.Series(ii).fillna("foo")  # <- raises, untested

ATM fillna behavior with incompatible values is pretty inconsistent. With numpy dtypes (except dt64 and td64) we always cast. With dt64 and dt64tz we always cast.

With td64 we always raise, but we only have tests for the value being an integer. Back circa 2018 we used to coerce integers to timedelta, so pd.Series(tdi).fillna(1) would interpret the 1 as pd.Timedelta(seconds=1) (not nanos!). We deprecated that coercion and now raise, but I don't think there was much thought given to whether to raise versus cast to object.

With Categorical we raise. We have 2 tests specific to that.

With PeriodDtype we cast and with IntervalDtype we raise, but we have tests for neither.

I don't have a particularly strong opinion on this, but would like to be consistent.

Metadata

Metadata

Assignees

No one assigned

    Labels

    BugExtensionArrayExtending pandas with custom dtypes or arrays.Missing-datanp.nan, pd.NaT, pd.NA, dropna, isnull, interpolatePDEP missing valuesIssues that would be addressed by the Ice Cream Agreement from the Aug 2023 sprintSeriesSeries data structure

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions