Missing values proposal: concrete steps for 1.0

[jorisvandenbossche]

Updated with to do list:

• Implement pd.NA scalar -> ENH: add NA scalar for missing value indicator, use in StringArray. #29597
• Basic BooleanArray -> ENH: add BooleanArray extension array #29555
• Use pd.NA in BooleanArray -> Use new NA scalar in BooleanArray #29961
  • Implement kleene-logic in logical ops on BooleanArray -> ENH: Implement Kleene logic for BooleanArray #29842
  • Update the behaviour of any/all reductions with skipna=False (API: any/all in context of boolean dtype with missing values #29686) -> API: BooleanArray any/all with NA logic #30062
• Use BooleanArray in comparison ops of StringArray -> StringArray comparisions return BooleanArray #30231
• Use pd.NA in IntegerArray -> API: Uses pd.NA in IntegerArray #29964
• Use BooleanArray as the return value for logical ops in IntegerArray -> API: Uses pd.NA in IntegerArray #29964
• Enable boolean indexing with BooleanArray ( DISCUSS: boolean dtype with missing value support #28778) -> DOC/TST: Indexing with NA raises #30308
• Use BooleanArray as the return value for boolean .str methods. -> API: Return BoolArray for string ops when backed by StringArray #30239
• Implement NA.__array_ufunc__ -> Implement NA.__array_ufunc__ #30245
• Base class for IntegerArray & BooleanArray -> REF: Implement BaseMaskedArray class for integer/boolean ExtensionArrays #30789
• Ensure everything is properly documented

---

Original issue:

Issue to discuss the implementation strategy for #28095.
Opening a new issue, as the other one already has a lot of discussion in several discussion, and would propose to keep this one focused on the practical aspects of how to implement this (regardless of certain aspects of the NA proposal such as single NA vs dtype-specific NAs -> for that will post a summary of the discussion on #28095 tomorrow).

I would like to propose the following way forward:

On the short term (ideally for 1.0):

• Already implement and provide the pd.NA scalar, and recognize it in the appropriate places as missing value (e.g. pd.isna). This way, it can already be used in external ExtentionArrays
• Implement a BooleanArray with support for missing values and appropriate NA behaviour. To start, we can just use a numpy masked array approach (similar to the existing IntegerArray), not involving any pyarrow memory optimizations.
• Start using this BooleanArray as the boolean result of comparison operations for IntegerArray/StringArray (breaking change for nullable integers)
  • Other arrays will keep using the numpy bool, this means we have two "boolean" dtypes side by side with different behaviour, and which one you get depends on the original data type (potentially confusing for users)
• Start using pd.NA as the missing value indicator for Integer/String/BooleanArray (breaking change for nullable integers)

On the intermediate term (after 1.0)

• Investigate if it can be implemented optionally for other data types and "activated" to have users opt-in for existing dtypes (to be further thought out).

---

I think the main discussion point is if we are OK with such a breaking change for IntegerArray.
I would personally do this: IntegerArray was only introduced recently, still regarded as experimental, and the perfect use case for those changes. But, it's certainly a clear backwards incompatible, breaking change.

cc @pandas-dev/pandas-core

[TomAugspurger]

My initial reaction is +1 on having StringArray and IntegerArray start using this new pd.NA value. Of all the "experimental" things in StringArray, the NA scalar is the one I know we'll want to break, since using np.nan really doesn't make sense.

If we're able to get BooleanArray done for 1.0, then that'd be great. I think time-wise, it's OK to settle for a simple ndarray + a private mask attribute. We optimize to use pyarrow or a bitmask later. (edit: just saw #29555 (review) Nice!)

So to be explicit, +1 on the proposed breaking changes to IntegerArray.

[jreback]

well we would need a hard dependency on pyarrow to allow BoolenArray

since we haven’t had this discussion then -0 on it

pd.NA is fine but i see this as too aggressive to use in IntegerArray for 1.0

this should be in master for a while and get usage

[TomAugspurger]

It seems to me like we could do integer array using a Boolean ndarray as a mask, and use pyarrow / a bitmask as an optimization later. It’s 2x the memory for now, but that’s maybe Ok. And I think we’ll eventually want IntegerArray ops to return nullable booleans, so better to do before 1.0?

…

On Nov 11, 2019, at 18:20, Jeff Reback ***@***.***> wrote:  well we would need a hard dependency on pyarrow to allow BoolenArray since we haven’t had this discussion then -0 on it pd.NA is fine but i see this as too aggressive to use in IntegerArray for 1.0 this should be in master for a while and get usage — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[jreback]

It seems to me like we could do integer array using a Boolean ndarray as a mask, and use pyarrow / a bitmask as an optimization later. It’s 2x the memory for now, but that’s maybe Ok. And I think we’ll eventually want IntegerArray ops to return nullable booleans, so better to do before 1.0?
…

it’s not 2x it’s like 10x compared to a native impl

[WillAyd]

I can't find one issue for it but I still think we should just take pyarrow on as a hard dependency if it simplifies our development. If import time is the big blocker IMO that's not really that big of a deal and would make it easier to manage / communicate BoolArray and similar developments

+0 on pd.NA. Not sure I see understand the entire scope of it from what is described but maybe something that just needs to be developed out

[jorisvandenbossche]

Sorry, I should have been clearer about this: all what I said above is meant to be pure numpy-based masked arrays, without any dependency on pyarrow, for now. This at least enables us to experiment with the API design without needing a heavy dependency such as pyarrow for the storage. Although I am clearly biased towards pyarrow, I don't think any of this needs pyarrow on the short term, we can perfectly implement this using numpy (and doing so doesn't complicate it, on the contrary, there is still a lot more functionality available for numpy masks compared to pyarrow arrays).

I indeed started a PR for the BooleanArray part of this: #29555. The numpy mask-based approach will not be the most efficient (both in memory and speed), but IMO it is now more important to first experiment with the design (the same can be said about the already existing IntegerArray).

it’s not 2x it’s like 10x compared to a native impl

For IntegerArray it is 2x (which is what Tom was saying I think), for BooleanArray the optimization could indeed be a lot bigger.

[TomAugspurger]

IMO it is now more important to first experiment with the design (the same can be said about the already existing IntegerArray).

Agreed (can add StringArray here too). I want to have 1.0 be as object-dtype free as possible :)

[TomAugspurger]

For IntegerArray it is 2x (which is what Tom was saying I think), for BooleanArray the optimization could indeed be a lot bigger.

For BooleanArrays with no missing values, isn't it exactly 2x more memory (assuming we store the mask when there are no missing values)? Then it's just the boolean ndarry of values + a boolean mask, so 2x. And we can, like pyarrow, skip allocating the mask if there are no NAs (though that's an optimization, not a requirement).

For one with missing values, isn't it less memory than our current implementation?

In [13]: s = pd.Series([None] + [True] * 1000)

In [14]: s.memory_usage(deep=True, index=False)
Out[14]: 36024

versus

In [15]: mask = np.array([True] + [False] * 1000)

In [16]: 2 * mask.nbytes
Out[16]: 2002

Even if we're using NaN for a boolean with missing values, we still have an improvement, since we use float64 by default

In [19]: s.astype(float).nbytes
Out[19]: 8008

[jorisvandenbossche]

It's indeed already an optimization compared to object dtype array of True/False/None. But I think Jeff meant the potential optimization of using pyarrow's boolean array storage, which can give a lot of additional memory improvement (8x compared to the double numpy array):

In [3]: mask = np.array([True] + [False] * 1000) 

In [4]: 2 * mask.nbytes 
Out[4]: 2002

In [5]: import pyarrow

In [6]: pyarrow.array(mask, mask=mask).nbytes
Out[6]: 252

But anyway, as I argued above, let's leave this for the future, and focus now on the API using numpy arrays under the hood (that use some more memory).

[TomAugspurger]

From the call today, there were no objections to changing IntegerArray to start using NA instead of np.nan.

@jorisvandenbossche do you intend to work on that? Otherwise, I can push up a WIP tomorrow (which will depend on #29597).

[jorisvandenbossche]

Feel free to already do a WIP on that, there is plenty to do ;)
I would first like to get those 2 open PRs merged