BUG: Index.astype does not localize to UTC first like Series.astype with datetime64[ns, tz] dtypes

[mroeschke]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

In [4]: pd.__version__
Out[4]: '1.1.0.dev0+1202.g1de2cf1ac'

In [5]: pd.Series([pd.Timestamp('2020-01-01 15:00')]).astype('datetime64[ns, EST]')
Out[5]:
0   2020-01-01 10:00:00-05:00
dtype: datetime64[ns, EST]

In [6]: pd.Index([pd.Timestamp('2020-01-01 15:00')]).astype('datetime64[ns, EST]')
Out[6]: DatetimeIndex(['2020-01-01 15:00:00-05:00'], dtype='datetime64[ns, EST]', freq=None)

Problem description

Out[5] is the expected output as documented in https://pandas.pydata.org/docs/user_guide/timeseries.html#time-zone-series-operations. Out[6] should align therefore with Out[5]

Expected Output

In [6]: pd.Index([pd.Timestamp('2020-01-01 15:00')]).astype('datetime64[ns, EST]')
Out[6]: DatetimeIndex(['2020-01-01 10:00:00-05:00'], dtype='datetime64[ns, EST]', freq=None)

[jbrockmendel]

cc @jorisvandenbossche @TomAugspurger fixing this inconsistency would move us a step closer to not-special-casing DatetimeTZBlock

[TomAugspurger]

Cool. Agreed with the expected output.

[jorisvandenbossche]

I am fully on board with making this consistent; but I am not fully sure I find the output of Series more expected than the output of DatetimeIndex' astype (it's rather subjective though).

The docs say that astype can "localize and convert" tz-naive timestamps, but doesn't explicitly says "localize to UTC and convert".

Basically the two options (expressed with an explicit localize):

In [12]: s_naive = pd.Series(pd.date_range('20130101', periods=3))  

In [13]: s_naive.astype('datetime64[ns, EST]')
Out[13]: 
0   2012-12-31 19:00:00-05:00
1   2013-01-01 19:00:00-05:00
2   2013-01-02 19:00:00-05:00
dtype: datetime64[ns, EST]

In [15]: s_naive.dt.tz_localize("UTC").astype('datetime64[ns, EST]') 
Out[15]: 
0   2012-12-31 19:00:00-05:00
1   2013-01-01 19:00:00-05:00
2   2013-01-02 19:00:00-05:00
dtype: datetime64[ns, EST]

In [16]: s_naive.dt.tz_localize("EST").astype('datetime64[ns, EST]')
Out[16]: 
0   2013-01-01 00:00:00-05:00
1   2013-01-02 00:00:00-05:00
2   2013-01-03 00:00:00-05:00
dtype: datetime64[ns, EST]

In the astype code, we also explicitly do a tz_localize with the new timezone, indicating it is likely done on purpose and not an accident (so might be worth looking at the history of this change).

Now, regardless of my comment about the expected result: is this something we want to simply change? (which is a breaking change) Or can we deprecate it first? (I think users can then avoid the warning by using tz_localize explicitly)
Given that this is about DatetimeIndex, the index version might also be used quite a lot.

[jorisvandenbossche]

On second thought, the "localize and convert" in the docs probably mean "localize to UTC and convert", indeed as @mroeschke suggested. Because otherwise if would just be a "localize" of course (there is nothing to convert anymore when localizing to the target timezone ..)

[jorisvandenbossche]

So do we actually have arguments for either behaviours? (apart from "because Series is doing it that way")

[jbrockmendel]

I think the Technically Correct answer may be to raise and require users to use tz_localize

The semantics of astype usually mean "represent the same information in another dtype". When going from tznaive<->tzaware, no matter which convention we use, the result does not represent "the same information"

[jreback]

I recall the initial implementation was pure convenience and in retrospect was a mistake. think we should actually deprecate this entirely. Being explicit about using .tz_localize or .tz_convert is a huge win here.