Review/Overhaul __array__ and view methods

[jbrockmendel]

#23524 fixes a problem with DatetimeIndex.__array__; we should do a thorough check of the __array__ methods on other classes to make sure (and test) that they make sense

[jorisvandenbossche]

So question is what do we want to support in __array__ (for the datetime-like EAs):

• default np.array(EA) will return an array of M8/m8 dtype, compatible with what it does for Series/Index (on master this currenlty gives object dtype, due to using the default iteration), and object array for PeriodArray
• np.array(EA, dtype=object) converts to object array of pandas Timestamp/Timedelta scalars.

But do we also want to support np.array(EA, dtype=int) to convert to i8/ordinals ?

[jorisvandenbossche]

But do we also want to support np.array(EA, dtype=int) to convert to i8/ordinals ?

For datetime/timedelta, this will happen automatically if we do nothing. So the only other option is to raise an error, but that does not seem necessary I think.
The question is still whether to explicitly enable it for PeriodArray.

[TomAugspurger]

Sorry I missed this earlier. I'm concerned about the default behavior on DatetimeIndex.__array__ and Series[datetime64[ns, tz]] with tz-aware data. Both of these convert to UTC, drop the tz, and return an M8[ns]-dtype array

In [7]: idx = pd.date_range('2000', periods=4, tz='US/Central')

In [8]: ser = pd.Series(idx)

In [9]: np.asarray(ser)
Out[9]:
array(['2000-01-01T06:00:00.000000000', '2000-01-02T06:00:00.000000000',
       '2000-01-03T06:00:00.000000000', '2000-01-04T06:00:00.000000000'],
      dtype='datetime64[ns]')

In [10]: np.asarray(idx)
Out[10]:
array(['2000-01-01T06:00:00.000000000', '2000-01-02T06:00:00.000000000',
       '2000-01-03T06:00:00.000000000', '2000-01-04T06:00:00.000000000'],
      dtype='datetime64[ns]')

This unfortunately conflicts with __iter__.

In [11]: list(ser)
Out[11]:
[Timestamp('2000-01-01 00:00:00-0600', tz='US/Central', freq='D'),
 Timestamp('2000-01-02 00:00:00-0600', tz='US/Central', freq='D'),
 Timestamp('2000-01-03 00:00:00-0600', tz='US/Central', freq='D'),
 Timestamp('2000-01-04 00:00:00-0600', tz='US/Central', freq='D')]

In [12]: list(idx)
Out[12]:
[Timestamp('2000-01-01 00:00:00-0600', tz='US/Central', freq='D'),
 Timestamp('2000-01-02 00:00:00-0600', tz='US/Central', freq='D'),
 Timestamp('2000-01-03 00:00:00-0600', tz='US/Central', freq='D'),
 Timestamp('2000-01-04 00:00:00-0600', tz='US/Central', freq='D')]

Do we care about that (I do)? Should we deprecate the current behavior?

diff --git a/pandas/core/arrays/datetimes.py b/pandas/core/arrays/datetimes.py
index e731dd33f..0f0577b1a 100644
--- a/pandas/core/arrays/datetimes.py
+++ b/pandas/core/arrays/datetimes.py
@@ -404,6 +404,9 @@ class DatetimeArrayMixin(dtl.DatetimeLikeArrayMixin,
     # Array-Like / EA-Interface Methods
 
     def __array__(self, dtype=None):
+        if dtype is None:
+            warnings.warn("bad", FutureWarning)
+            dtype = object
         if is_object_dtype(dtype):
             return np.array(list(self), dtype=object)
         elif is_int64_dtype(dtype):

usage:

In [6]: np.asarray(idx)
/Users/taugspurger/sandbox/pandas/pandas/core/arrays/datetimes.py:408: FutureWarning: bad
  warnings.warn("bad", FutureWarning)
Out[6]:
array([Timestamp('2000-01-01 00:00:00-0600', tz='US/Central', freq='D'),
       Timestamp('2000-01-02 00:00:00-0600', tz='US/Central', freq='D'),
       Timestamp('2000-01-03 00:00:00-0600', tz='US/Central', freq='D'),
       Timestamp('2000-01-04 00:00:00-0600', tz='US/Central', freq='D')],
      dtype=object)

In [7]: np.asarray(ser)
/Users/taugspurger/sandbox/pandas/pandas/core/arrays/datetimes.py:408: FutureWarning: bad
  warnings.warn("bad", FutureWarning)
Out[7]:
array([Timestamp('2000-01-01 00:00:00-0600', tz='US/Central', freq='D'),
       Timestamp('2000-01-02 00:00:00-0600', tz='US/Central', freq='D'),
       Timestamp('2000-01-03 00:00:00-0600', tz='US/Central', freq='D'),
       Timestamp('2000-01-04 00:00:00-0600', tz='US/Central', freq='D')],
      dtype=object)

[TomAugspurger]

As an example of the kind of strange knock-on effects this has, see https://github.com/pandas-dev/pandas/pull/23601/files#diff-425b4da47d01dc33d86c5c697e196b70R1128.

Ideally we'd just write array = np.asarray(values) there, but we had to special case categorical there, since it dispatches to .categories.__array__, which did the "wrong" thing for tz-aware datetime data.

[jreback]

I agree here. np.asarray() should preserve dtype if possible. so ok with making this consistent.