EA: revisit interface

[jbrockmendel]

This is as good a time as any to revisit the "experimental" EA interface.

My read of the Issues and recollection of threads suggests there are three main groups of topics:

Clarification of the Interface
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

1. _values_for_argsort and values_for_factorize
   • Do we need both? The docs both say they should be order-preserving.
   • Is it safe to return a view? (Categorical.values_for_argsort makes a copy for no obvious reason)
   • What else can they be used for internally? e.g. in CLN: use _values_for_argsort for join_non_unique, join_monotonic #32467 _values_for_argsort is used for ExtensionIndex join_non_unique and join_monotonic
2. What characteristics should _ndarray_values have? Is it needed? (DEPR: _ndarray_values vs to_numpy vs __array__ #32412) _ndarray_values has been removed
3. What should _from_sequence accept?
   • Should it only be sequences that are unambiguously this dtype?
   • In particular, should DTA/TDA/PA not accept i8 values?
4. What should fillna accept? (Should ExtensionBlock.fillna unbox Series / Index? #22954, EA: fillna should accept same type #32414)
   4.5) Require that __iter__ return native types? API: ExtensionArrays and conversion to "native" types (eg in tolist, to_dict, iteration, ..) #29738

Ndarray Compat
^^^^^^^^^^^^^^^^^
5) Headaches have been caused by trivial ndarray methods not being on EA
- #31199 size
- #32342 "T" (just the most recent; this has come up a lot)
- #24583 ravel
6) For arithmetic we're going to need something like either tile or broadcast_to

Methods Needed/Wanted For Index/Series/DataFrame/Block
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
7) Suggested Methods (partial list)
- #27264 duplicated
- [x] #23437 _empty
- #28955 apply
- #23179 map
- #22680 hasnas
- [x] #27081 equals
- [x] #24144 _where
- [x] _putmask would be helpful for ExtensionIndex

I suggest we discuss these in order. Before jumping in, is there anything vital missing from this list? (this is only a small subset of the issues on the tracker)

cc @pandas-dev/pandas-core @xhochy

[TomAugspurger]

_values_for_argsort and values_for_factorize

Do we need both?

I went back through #20361, but it isn't very illuminating. My early version used _values_for_argsort, but 30941cb changed it to _values_for_factorize. I think the main difference (aside from the return type including the NA value) is that the values in _values_for_factorize need to be hashable.

What characteristics should _ndarray_values have? Is it needed? (#32412)

This has always been nebulous, at least to me. I think it's primarily used in indexing engines, but I may be out of date.

What should _from_sequence accept?

Joris has argued for a while now that it should only accept instances of ExtensionDtype.type or NA values. We had a recent issue (perhaps around groupby?) that was caused by _from_sequence being too permissive.

What should fillna accept? (#22954, #32414)

I think my preference is to restrict this to instances of ExtensionDtype.type. It's somewhat strange for .fillna() to result in an object dtype. The obvious exception is something like IntegerArray.fillna(1.5), which may reasonable be cast to float.

Will comment more on the "additional methods" later, but my opening thought is that we should come up with a guiding principle (e.g. "strive for minimalism" or "drop-in replacement for 1D ndarray" and apply that to cases, rather than deciding case by case).

[jbrockmendel]

What characteristics should _ndarray_values have? Is it needed? (#32412)

This has always been nebulous, at least to me. I think it's primarily used in indexing engines, but I may be out of date.

By my count we have 6 places left where we use _ndarray_values. #32646 removes two, #32538 removes one, #32476 removes one, and #32641 deprecates one. The last one I'll be making a PR for shortly. Unless there is an objection, I'd like to get rid of _ndarray_values once we no longer use it.

I think the main difference (aside from the return type including the NA value) is that the values in _values_for_factorize need to be hashable.

That is referenced indirectly in the _values_for_factorize docstring, should probably be made explicit.

The ExtensionIndex code is making an implicit assumption about values_for_argsort that should probably be made explicit: the ordering should be preserved not just within left._values_for_argsort() but also with right._values_for_argsort() (when right.dtype matches left.dtype).

Unless we can find a compelling reason to keep both _values_for_argsort and _values_for_factorize, I'd advocate dropping one of them.

What should _from_sequence accept?

Joris has argued for a while now that it should only accept instances of ExtensionDtype.type or NA values

I'm leaning more and more in that direction, just found another problem caused by being too lenient, xref #32668. I expect DTA/TDA/PA are the worst offenders, will experiment with tightening those up and see if it breaks the world.

As a heuristic, maybe something like "cls._from_sequence(values) works if and only if pd.array(values) would return cls". The heuristic breaks down for non-internal EAs until infer_dtype can handle them.

Will comment more on the "additional methods" later, but my opening thought is that we should come up with a guiding principle (e.g. "strive for minimalism" or "drop-in replacement for 1D ndarray" and apply that to cases, rather than deciding case by case).

+1 on "drop-in replacement for 1D ndarray"; my understanding is that the E in EA is about "extending np.ndarray"

A couple other potential guiding principles (not mutually exclusive):

• The minimal set of methods so that Block/ExtensionIndex methods don't have to roundtrip EA->ndarray->EA
• The minimal set of methods so that we're not special-casing Categorical all over the place.

[jorisvandenbossche]

@jbrockmendel thanks for gathering all those

I think the main difference (aside from the return type including the NA value) is that the values in _values_for_factorize need to be hashable.

Some additional, assorted thoughts about this aspect:

• _values_for_factorize works with NA sentinel (and I don't think the NA sentinel needs to sort correctly), while _values_for_argsort does not.
  Do we assume that missing values in argsort are smallest? At least in sorting.py::nargsort, we also still compute a mask, so there we don't make any assumption about it.
• For _values_for_factorize, I think we should also consider making it an option to return (values, mask) in addition to (values, na_sentinel) in case this is easier/cheaper to provide (which is especially the case for the masked arrays; this will need to support of the factorize algos for masks though)
• In GeoPandas, we implement _values_for_factorize, because geometries can be factorized, but we do not implement _values_for_argsort, because we consider geometries as "not sortable" (the hashed values from _values_for_factorize of course have a certain order, but eg Series.sort_values raises an error currently). Having those two methods makes this distinction possible.
  That said: we could probably indicate "not-orderable" in another way as well, if needed. And I am also not fully sure we did the best thing in GeoPandas, as being able to sort still has value for non-orderable values (similar to how an unordered categories has no min/max, but can still be sorted).

It's certainly good to think about if we can combine the two.

Joris has argued for a while now that it should only accept instances of ExtensionDtype.type or NA values

I'm leaning more and more in that direction, just found another problem caused by being too lenient, xref #32668. I expect DTA/TDA/PA are the worst offenders, will experiment with tightening those up and see if it breaks the world.

Yes, I think that in many places where we now use _from_sequence, we should be more strict and only allow actual scalars (a related issue here is #31108).
However, it's probably still useful to have the less strict version as well. Eg in pd.array(..) with a specified dtype, you can be more liberal in the conversion.

So one option might be to keep _from_sequence as is to for example be used in pd.array, and introduce a new _from_scalars that has the more strict behaviour (only accepting actual scalars). And by default that could use _from_sequence for backwards compatibility.

Another option might be to let this more general conversion be handled by the new astype machinery (#22384), since you could see a general conversion from list-like as in pd.array as an astype from object to the specific dtype. And then we can keep _from_sequence for the strict conversion.
(but the first option is probably easier short term, since refactoring astype is a bigger topic)

Aside, we should probably have defined methods like _from_sequence/_from_scalars on the dtype instead of the array.

What should fillna accept? (#22954, #32414)

I think my preference is to restrict this to instances of ExtensionDtype.type. It's somewhat strange for .fillna() to result in an object dtype. The obvious exception is something like IntegerArray.fillna(1.5), which may reasonable be cast to float.

Personally, I think we should adopt the rule of fillna being strictly filling values, and thus never to change the dtype.
If that is the case, then I think it is up to the array to accept anything it wants as fill value, as long as that can be converted to something that fits in the array (or being strict and only accept scalars, if the EA implementor prefers that).

It might be we need something similar like _can_hold_element for other places to know what values to accept.

By my count we have 6 places left where we use _ndarray_values ... Unless there is an objection, I'd like to get rid of _ndarray_values once we no longer use it.

I am not that optimistic we can actually fully get rid of it without some replacement (there are cases were we need an ndarray). Although in some cases _values_for_factorize could maybe be used instead, eg for the Index engine once we support general EAs in Index.

---

For the "bigger" topics discussed here (like the factorize/argsort values), it might be worth to start separate the discussion?

[jbrockmendel]

It looks like we have a special Categorical._values_for_rank used in factorize, do we need this more generally?

[jorisvandenbossche]

It looks like we have a special Categorical._values_for_rank used in factorize, do we need this more generally?

Or putting this the other way around: could rank rather use _values_for_argsort instead? (I suppose this _values_for_rank from Categorical predates the EA interface work)

[xhochy]

_from_sequence(values)

As a heuristic, maybe something like "cls._from_sequence(values) works if and only if pd.array(values) would return cls". The heuristic breaks down for non-internal EAs until infer_dtype can handle them.

This would break fletcher quite a bit as nearly all scalar types we support are also types pandas supports natively, so pd.array would always return the pandas.array type, not the fletcher one.

Also there was a mention here that _from_sequence should only work for ExtensionDtype.type, this is also problematic. For example in the datetime case, one would like to support datetime.datetime, np.datetime and pd.Timestamp. It would be good to just leave the check for valid scalars in _from_sequence inside the ExtensionDtype.

Methods defined on pd.Series/DataFrame

I think I would be able to supply a long list here would I have made further progress with fletcher. The important thing here is, that it should be optional to override these in the ExtensionArray. For a lot of possible implementation of ExtensionArray, the defaults are probably better what someone would write but in the Arrow case, it would quite often make sense to overwrite them as there someone already went the extra mile and did a more efficient implementation that was tailored to Arrow's bitmask setup.

One more thing that troubles me currently is that the the dt and str accessors only work on the pandas-internal types. It would also be nice to optionally implement some methods from them in the ExtensionArray.

Test Setup

I really like this. This was an amazing thing for being able to implement the interface.

(also: thanks for the mention; I sadly don't have the bandwidth to watch the pandas repo but am always interested in giving input on EAs)

[xhochy]

cc @crepererum who works on https://github.com/JDASoftwareGroup/rle-array

[TomAugspurger]

Also there was a mention here that _from_sequence should only work for ExtensionDtype.type, this is also problematic.

Is it possible for you to define .type as a metaclass that registers each scalar type, as in https://github.com/ContinuumIO/cyberpandas/blob/dbce13f94a75145a59d7a7981a8a07571a2e5eb6/cyberpandas/ip_array.py#L22-L29?

[jorisvandenbossche]

Also there was a mention here that _from_sequence should only work for ExtensionDtype.type, this is also problematic. For example in the datetime case, one would like to support datetime.datetime, np.datetime and pd.Timestamp. It would be good to just leave the check for valid scalars in _from_sequence inside the ExtensionDtype.

Can you clarify how that would be problematic?
And to be clear, if we do this change, we would only call _from_sequence in places where we know the data came from the array. Not in general construction functions like pd.array(.., dtype=). For that, something less strict is still needed (see my comment above about either introducing a separate _from_scalars, or either letting this general conversion be handled by astype).

[xhochy]

Also there was a mention here that _from_sequence should only work for ExtensionDtype.type, this is also problematic.

Is it possible for you to define .type as a metaclass that registers each scalar type, as in https://github.com/ContinuumIO/cyberpandas/blob/dbce13f94a75145a59d7a7981a8a07571a2e5eb6/cyberpandas/ip_array.py#L22-L29?

Yes and no, we could though specify it currently as the union of dict, list, int, float, bytes, str, datetime, np.datetime, pd.Timestamp and some more. But I'm not sure whether this would bring any valuable information into the interface as it a very broad range of types.

fletcher is here the extreme case where have FletcherContinuousArray._from_sequence which is not yet actually yet bound to a specific dtype. Depending on the scalars you pass in, you get a different ExtensionDtype instance.

Can you clarify how that would be problematic?
And to be clear, if we do this change, we would only call _from_sequence in places where we know the data came from the array.

Currently for all results of an operation on a FletcherArray, we would also return a FletcherArray again. In some cases it though has a different type. For example we have __equals__(fletcher[int], fletcher[int]) -> fletcher[bool] or __div__(fletcher[int], fletcher[int]) -> fletcher[float], always to carry on the valid mask into the results.

[jorisvandenbossche]

But I'm not sure whether this would bring any valuable information into the interface as it a very broad range of types.

And I also don't think this should be the goal / point of ExtensionDtype.type

fletcher is here the extreme case where have FletcherContinuousArray._from_sequence which is not yet actually yet bound to a specific dtype. Depending on the scalars you pass in, you get a different ExtensionDtype instance.

_from_sequence gets the dtype object passed, so even if the implementation is not bound to a dtype-specific class, it can use the dtype information to return the correct object.

Currently for all results of an operation on a FletcherArray, we would also return a FletcherArray again.

That's not where _from_sequence would be used. There are occasions that you do not return a FletcherArray, such as getitem for a scalar indexer or a reduction function, right?
It is only for those cases where _from_sequence would be used (again, the strict version would only be used for that, for general construction we still need something else).