ENH: investigate using a bitarray as the mask in the nullable/masked ExtensionArrays

[jorisvandenbossche]

Currently, our nullable / masked extension arrays (boolean, integer, for now) are using a numpy boolean array as their _mask to keep track of missing values. A potential route for improving memory and performance would be using a bitarray instead of a boolean numpy array (which is a byte per value).

This should require some exploration: what are options how to implement this? (existing libraries, custom implementation) What is the performance impact? (some things like masking will also be slower, since we still rely on numpy for that, which needs boolean arrays) Is this worth it to do a custom implementation rather than using pyarrow for this? etc

[jbrockmendel]

xref #21839.

I asked @seberg if numpy had thought about implementing this directly and got the impression it had been discussed but never as a priority. I'm sure we could implement this here, but it definitely seems like a better fit for numpy.

Does pyarrow have an implementation? (or will it in the not-too-distant future?)

[seberg]

The issue with it within NumPy is simple. NumPy has a strided memory layout, which implies byte sized strides to reach elements. To do this we would have to ensure that everything inside NumPy can somehow deal with these sub-byte strides (when the dtype or a flag requests it).

It could be interesting to explore, but it seems like it likely is just a can of worms, especially since strides are somewhat public API. Making a bitarray class behaves mostly like an ndarray may be better/easier, although not sure how complete that has to be to be good enough for this use case.

[jorisvandenbossche]

Also, for pandas we "just" need a 1D bitarray, which I assume will make it a lot simpler than the full multidimensional requirements for numpy.

---

In #22238, @WillAyd used https://github.com/ilanschnell/bitarray

[jreback]

and i made it work here: #25415