API: how to handle NA in conversion to numpy arrays

[jorisvandenbossche]

In #29964 and #29961 NA in IntegerArray and BooleanArray), the question comes up how to handle pd.NA's in conversion to numpy arrays.

Such conversion occurs mainly in __array__ (for np.(as)array(..)) and .astype(). For example:

In [3]: arr = pd.array([1, 2, pd.NA], dtype="Int64")  

In [4]: np.asarray(arr) 
Out[4]: array([1, 2, None/pd.NA/..?], dtype=object)

In [5]: arr.astype(float)  
Out[5]: array([ 1.,  2., nan])  # <--- allow automatic NA to NaN conversion?

Questions that come up here:

• By default, when converting to object dtype, what "NA value" should be used? Before this was NaN or None, now it could logically be pd.NA.
  A possible reason to choose None instead of pd.NA is that third party code that needs a numpy array will typically not be able to handle pd.NA while None is much more normal. On the other hand, there is also still time for such third party code to adapt. And it will probably be good to keep list(arr) (iteration/getitem) and np.array(arr, dtype=object) consisetnt.

• When converting to a float dtype, are we fine to automatically convert pd.NA to np.nan ? Or do we think the user should explicitly opt in for this?

We will probably want to add a to_numpy to those Integer/BooleanArray to be able to make those choices explicit, eg with following signature:

def to_numpy(self, dtype=object, na_value=...):
    ... 

where you can explicitly say which value to use for the NAs in the final numpy array (and the Series.numpy can then forward such keyword).
That way, a user can do arr.to_numpy(dtype=object, na_value=None) to get a numpy array with None instead of pd.NA, or arr.to_numpy(dtype=float, na_value=np.nan) to get a float array with NaNs.

But even if we have that function (which I think we should), the above questions about the defaults are still to be answered (eg for __array__ we cannot have such a na_value keyword, so we need to make a default choice).

cc @TomAugspurger @Dr-Irv

[TomAugspurger]

+1 for a standard array.to_numpy(dtype=None, na_value=None) method. Perhaps we'll want to align this with Series.to_numpy, and make the signature .to_numpy(dtype=None, copy=False, na_value=None).

I'm really uncertain about np.asarray(arr, dtype=float) / array.astype(float). On the one hand, how annoying will it be to have np.asarray(arr) be essentially useless for some timespan, as projects won't really know what to do? However, it really is inconsistent to have .astype(float) imply NA -> NaN, while not allowing float(NA) to mean NaN. So at this moment, I'd probably vote for IntegerArray.astype(float) / np.asarray(integer_array, dtype=float) raising when there are NA values present, but I'm not sure.

[Dr-Irv]

Great question @jorisvandenbossche . This all relates to the semantics of pd.NA vs. np.nan, and the legacy issue of pandas using np.nan to mean "missing value" independent of type.

I think there are a few cases to consider:

1. Currently (pandas 0.25.x) pd.Series.values returns a numpy array, so that has to change, right? That then has two suboptions:
   a. It still returns a numpy array, in which case I don't see how you solve the conversion issue except by assuming pd.NA values goes to NaN, or by raising an exception
   b. It returns a different iterative object (a pandas array type), in which case we don't have to worry about it.
2. How to convert any array to a numpy array. I like the array.to_numpy(dtype=None, na_value=None) option.
3. For astype, we are using pandas types, so I think that pd.NA should sustain across type conversions. So IntegerArray.astype(float) would preserve pd.NA from the integers over to the floats, since the result is still a Series.
4. For np.asarray, raise an exception if any pd.NA values are present, which makes users aware that pd.NA doesn't convert to a representation that numpy supports, and the workaround is to use the to_numpy method. I don't know if that's possible to do, in which case we return the pd.NA type and numpy will make the dtype of the result an object.

In general, I'm in favor of raising exceptions whenever we try to convert pd.NA to something else, which forces users to understand that pd.NA is a different thing than None or np.nan, and then we provide new methods (or arguments, as appropriate) to force users to make a choice of how the conversion happens.

[TomAugspurger]

Series.values can return an ndarray or an extension array (e.g. Categorical). I don't think we'll want to change the behavior of that anywhere. We have .array and .to_numpy() as alternatives, and .to_numpy() will grow an option to control the NA value.

So IntegerArray.astype(float) would preserve pd.NA from the integers over to the floats, since the result is still a Series.

At the moment, aliases like float, or 'float' fall back to np.dtype(thing) when there's no extension type for them. So IntegerArray.astype(float) will return a NumPy array. And Series[Int64].astype(float) will return a Series[numpy.float64] (a series backed by a float64-dtype NumPy array).

[Dr-Irv]

Series.values can return an ndarray or an extension array (e.g. Categorical). I don't think we'll want to change the behavior of that anywhere. We have .array and .to_numpy() as alternatives, and .to_numpy() will grow an option to control the NA value.

So, IMHO, if someone uses Series.values and the result is a numpy array (ndarray), and the series has pd.NA in it, I think we should raise an exception, because we shouldn't make any assumptions about how pd.NA is stored in a numpy array.

At the moment, aliases like float, or 'float' fall back to np.dtype(thing) when there's no extension type for them. So IntegerArray.astype(float) will return a NumPy array. And Series[Int64].astype(float) will return a Series[numpy.float64] (a series backed by a float64-dtype NumPy array).

So if I understand you right, if you have [WhateverType]Array.astype(float), we return a numpy array, then I would suggest doing the same behavior as above. Raise an exception if the array has pd.NA in it.

But for Series.astype(newtype), the result is always a Series, so in this case we can preserve the pd.NA values independent of newtype. Right?

[jorisvandenbossche]

So, IMHO, if someone uses Series.values and the result is a numpy array (ndarray)

All the cases we are talking about are new ExtensionArrays, and for those Series.values will actually always be the ExtensionArray itself. For conversion to numpy you need something as np.(as)array(..) or astype. So I don't think we need to discuss .values.

But for Series.astype(newtype), the result is always a Series, so in this case we can preserve the pd.NA values independent of newtype. Right?

Yes, that is correct. But, as Tom said above, we don't have a "float newtype", so all discussions about converting to float is at this moment per definition conversion to numpy (until we add a float ExtensionArray).

In general, I'm in favor of raising exceptions whenever we try to convert pd.NA to something else, which forces users to understand that pd.NA is a different thing than None or np.nan, and then we provide new methods (or arguments, as appropriate) to force users to make a choice of how the conversion happens.

I think I am also in favor of this.

[jorisvandenbossche]

Rereading this again, it seems the conclusion is somewhat that we want to raise for this (so no automatic conversion of pd.NA to np.nan).

I am wondering how that impacts projects like eg scikit-learn. They will need to use to_numpy then explicitly when they want to handle eg nullable integer (I should check if it actually works now, as they work on the dataframe level, you probably get a object array, that they then might try to convert to float).