dtype for tz-naive DatetimeArray and TimedeltaArray

[TomAugspurger]

Right now, Datetime.Array.dtype can be either np.dtype('M8[ns]) or a DatetimeTZDtype, depending on whether the values are tz-naive or tz-aware. This means that while DatetimeArray[tz-naive] is an instance of ExtensionArray, it doesn't satisfy the minimum ExtensionArray API, which requires that array.dtype be an ExtensionDtype.

In [4]: pd.date_range('2000', periods=4)._data.dtype
Out[4]: dtype('<M8[ns]')

In [5]: pd.date_range('2000', periods=4, tz="CET")._data.dtype
Out[5]: datetime64[ns, CET]

The causes some type-unsoundness for places that are supposed to return an ExtensionArray. The two most prominent being pd.array and Series.array. As an example, following isn't necessarily safe code

def f(ser: Series) -> Callable: 
    return ser.array.dtype.construct_array_type

that will fail for tz-naive datetime data, because its .dtype is a NumPy dtype.

---

Proposal:

1. Make a DatetimeDtype (or allow DatetimeTZDtype to have tz=None). Make a TimedeltaArray.dtype
2. Ensure that DatetimeArray.dtype and TimedeltaArray.dtype is always an ExtensionDtype
3. Wrap Series.dtype and DatetimeIndex.dtype to continue returning the NumPy dtype

The last step is to avoid breaking code relying on Series[tz-naive].dtype being a NumPy dtype.

[TomAugspurger]

@jorisvandenbossche would you expect the following to be placed in an ExtensionBlock (backed by a TimedeltaArra), or a TimedeltaBlock (backend by an ndarray)?

pd.Series(pd.array(['1H', '2H'], dtype=pd.core.dtypes.dtypes.TimedeltaDtype()))

For now, I think this should continue to be a TimedeltaBlock.

[jorisvandenbossche]

Yes, that is what I expect as well (so that for those arrays a special case is made to construct a different kind of block). Because we should internally have only one way to store timedelta data.

[TomAugspurger]

I abandoned the PR #24674. There are a few issues around DatetimeArray.astype #24674 (comment)

[jbrockmendel]

leaning towards "dont bother"

[jbrockmendel]

On further consideration, I think this might be useful for code de-duplication. In particular a bunch of methods that are currently implemented on both Timestamp and DatetimeArray could instead be implemented on DatetimeDtype with a signature that takes int64_t*, which think could then be shared by Timestamp/DTA. Similar with Timedelta/TDA.

[jorisvandenbossche]

What methods are you thinking about?

[jbrockmendel]

What methods are you thinking about?

The clearest example is .normalize. Pretty much anything that goes through tslibs.fields.

[jorisvandenbossche]

We're speaking about a dtype, right?
Since both Timestamp or DatetimeArray don't inherit from a dtype (for DatetimeArray it's an attribute), how does implementing methods on the dtype help sharing?