API: setting np.nan into Series[bool]

[jbrockmendel]

ser = pd.Series([True, False, True, False])
ser[0] = np.nan

>>> ser
0    NaN
1    0.0
2    1.0
3    0.0
dtype: float64

It would be more consistent (and would simplify the code) if this cast to object and did not convert True/False to 1.0/0.0. ATM we have 2 tests that such a change would break.

[jorisvandenbossche]

Isn't this actually consistent with how other numpy dtypes work? We generally upcast on setitem-like operations, and for boolean this means upcasting to float (like integers also get upcasted to float)?

[jbrockmendel]

I think it is consistent with how numpy works, but bool-casting is one of the cases where we often behave differently from numpy. Ideally we would consistently use find_common_type:

>>> find_common_type([np.float64, np.bool_])
dtype('O')

[jreback]

i agree with @jbrockmendel here

this is something we should do correctly

numpy does this because it for backwards compatibility not because it's the right thing to do

we want to preserve dtypes whenever possible

bools are a wart when converted to float (ideally we would actually just convert to Boolean though)

[jorisvandenbossche]

Where is this find_common_type used? Since it are not extension dtypes, for example concat also doesn't follow that:

In [17]: pd.concat([pd.Series([True, False]), pd.Series([0.1, 2])])
Out[17]: 
0    1.0
1    0.0
0    0.1
1    2.0
dtype: float64

[jorisvandenbossche]

bools are a wart when converted to float

Bools are just as much a wart when converted to object dtype, though.
There is no such thing as "correct" IMO, just choices with trade-offs (and for sure, we should discuss those trade-offs and make a choice, but converting to object dtype being "correct" is not an argument IMO)

[jbrockmendel]

Where is this find_common_type used?

If we were to change this behavior, the change would look like #38709, which uses coerce_to_target_dtype in place of the custom logic currently in Block.setitem. coerce_to_target_dtype now uses find_common_type.

for example concat also doesn't follow that:

I haven't looked at concat all that closely, but based on the example you gave id like that to be object too.

[jreback]

what if we convert directly to Boolean here? that is the long term type we want anyhow.