ERR: raise in tz incompat with DatetimeIndex ops

[jreback]

xref #6552

idx = pd.date_range(..., tz='UTC')
idx2 = idx.reindex(pd.date_range(..., tz='US/Pacific')

• subtraction of datetime like
• intersection / union / difference
• reindex

[jreback]

cc @immerrr

[immerrr]

I think that the point in raising in such situations is to prevent guessing in the face of ambiguity.

There's an ambiguity when you do pretty much any binary operation between a naive and an aware timestamps as the former can correspond to about 40 different wallclock timestamps and it's unclear which one exactly, so this should be reported right away just as with vanilla python datetime.

I have doubts about "subtraction of datetime like" though as there's no ambiguity in subtracting two aware timestamps from different timezones. There also may be very annoying issues because AFAIR dst transition involves changing timezones and subtracting a properly localized summer timestamp from the winter one will raise then.

"intersection / union / difference" is ambiguous w.r.t what should be the tz of the result, seems ok to raise.

As for "reindex", when I'm doing output, reindexer = index.reindex(new_index), I expect output to contain the same elements as in new_index which in case of tz-aware dt-index includes the timezone, so I don't see anything ambiguous here and the operation feels fine.

[jreback]

The reason to raise on subtraction is that its almost always an unintended behavior. The user should have them in the same timezone. Yes you can auto-convert it, but I just think that's asking for trouble. Reindexing is the same thing, yes it can work (and maybe even is non-ambiguous), forget about transition times for a second. The problem is that it is almost never what you actually want and is likely an error.

cc @rockg
cc @sinhrks
cc @ischwabacher

[rockg]

I agree that always forcing the user to convert themselves for subtraction makes the most sense. Trying to guess what the user intends is always a frustrating endeavor for both the programmer and user.

For reindex if the indices are the same size then I think it's fine as it is essentially a replacement. But if you are using a method to fill in missing values, then it gets hairy. Maybe we need to look at a couple of examples to get on the same page.

[ischwabacher]

My opinion made a complete about-face while I was writing this comment. I now think the reindex question is morally equivalent to this:

In [1]: import pandas as pd

In [2]: pd.Series(1, index=[1, 2, 3])
Out[2]: 
1    1
2    1
3    1
dtype: int64

In [3]: _.index
Out[3]: Int64Index([1, 2, 3], dtype='int64')

In [4]: _2.reindex([1., 2., 3.])
Out[4]: 
1    1
2    1
3    1
dtype: int64

In [5]: _.index
Out[5]: Float64Index([1.0, 2.0, 3.0], dtype='float64')

It's like any other coercion of one value to an equal value of a different type. Naive Timestamps are, of course, unequal to aware Timestamps, and the only contact between them should be via tz_localize (or tz_forget, if we ever decide to create such a thing).

[rockg]

tz_forget was just implemented tz_localize(None). I agree the coercion is fine but if you have an daily series in US/Eastern and you reindex with a UTC hourly index with method='pad' what is that supposed to do? Do you see changes in the new series at hour 4/5 (DST/not DST) or hour 0?

[jreback]

I am thinking we should simply raise if its not a completely non-ambiguous conversion, e.g.
if both/either are UTC/naive then its ok, otherwise raise?

or are you ok with reindex on an Index returns a tuple of new_index, indexer

[9] 'works' but is a bit odd (this is on master, I guess the suggestion is to make [9] work like [10]), but have tz='US/Eastern' in the result

In [9]: date_range('20130101',periods=5,tz='US/Eastern').reindex(date_range('20130101',periods=3))
Out[9]: 
(<class 'pandas.tseries.index.DatetimeIndex'>
 [2013-01-01, ..., 2013-01-03]
 Length: 3, Freq: D, Timezone: None, array([-1, -1, -1]))

In [10]: date_range('20130101',periods=5).reindex(date_range('20130101',periods=3))
Out[10]: 
(<class 'pandas.tseries.index.DatetimeIndex'>
 [2013-01-01, ..., 2013-01-03]
 Length: 3, Freq: D, Timezone: None, array([0, 1, 2]))

[immerrr]

Speaking of metadata and dst transitions, what happens now if there's a date range that starts in summer time and ends in winter time?

[ischwabacher]

Right, I was even paying attention to the tz_convert(None) issue. Herp derp.

I think since a DatetimeIndex is conceptually an array of Timestamps, each of which represents a precise moment in time, this conversion is pretty straightforward. I approve of this current behavior:

In [1]: import pandas as pd

In [2]: idx = pd.date_range('20131101', tz='America/Chicago', periods=7)

In [3]: newidx = pd.date_range('20131103', tz='UTC', periods=36, freq='H')

In [4]: pd.Series(range(7), index=idx).reindex(newidx, method='ffill')
Out[4]: 
2013-11-03 00:00:00+00:00    1
2013-11-03 01:00:00+00:00    1
2013-11-03 02:00:00+00:00    1
2013-11-03 03:00:00+00:00    1
2013-11-03 04:00:00+00:00    1
2013-11-03 05:00:00+00:00    2  # midnight before DST ends, offset = -0500
2013-11-03 06:00:00+00:00    2
2013-11-03 07:00:00+00:00    2
2013-11-03 08:00:00+00:00    2
2013-11-03 09:00:00+00:00    2
2013-11-03 10:00:00+00:00    2
2013-11-03 11:00:00+00:00    2
2013-11-03 12:00:00+00:00    2
2013-11-03 13:00:00+00:00    2
2013-11-03 14:00:00+00:00    2
2013-11-03 15:00:00+00:00    2
2013-11-03 16:00:00+00:00    2
2013-11-03 17:00:00+00:00    2
2013-11-03 18:00:00+00:00    2
2013-11-03 19:00:00+00:00    2
2013-11-03 20:00:00+00:00    2
2013-11-03 21:00:00+00:00    2
2013-11-03 22:00:00+00:00    2
2013-11-03 23:00:00+00:00    2
2013-11-04 00:00:00+00:00    2
2013-11-04 01:00:00+00:00    2
2013-11-04 02:00:00+00:00    2
2013-11-04 03:00:00+00:00    2
2013-11-04 04:00:00+00:00    2
2013-11-04 05:00:00+00:00    2
2013-11-04 06:00:00+00:00    3  # midnight after DST ends, offset = -0600
2013-11-04 07:00:00+00:00    3
2013-11-04 08:00:00+00:00    3
2013-11-04 09:00:00+00:00    3
2013-11-04 10:00:00+00:00    3
2013-11-04 11:00:00+00:00    3
Freq: H, dtype: int64

and disapprove of this current behavior:

In [24]: pd.Series(range(7), index=idx).reindex(newidx.tz_convert(None), method='ffill')
Out[24]: # Should raise
2013-11-03 00:00:00    1
2013-11-03 01:00:00    1
2013-11-03 02:00:00    1
2013-11-03 03:00:00    1
2013-11-03 04:00:00    1
2013-11-03 05:00:00    2
2013-11-03 06:00:00    2
2013-11-03 07:00:00    2
2013-11-03 08:00:00    2
2013-11-03 09:00:00    2
2013-11-03 10:00:00    2
2013-11-03 11:00:00    2
2013-11-03 12:00:00    2
2013-11-03 13:00:00    2
2013-11-03 14:00:00    2
2013-11-03 15:00:00    2
2013-11-03 16:00:00    2
2013-11-03 17:00:00    2
2013-11-03 18:00:00    2
2013-11-03 19:00:00    2
2013-11-03 20:00:00    2
2013-11-03 21:00:00    2
2013-11-03 22:00:00    2
2013-11-03 23:00:00    2
2013-11-04 00:00:00    2
2013-11-04 01:00:00    2
2013-11-04 02:00:00    2
2013-11-04 03:00:00    2
2013-11-04 04:00:00    2
2013-11-04 05:00:00    2
2013-11-04 06:00:00    3
2013-11-04 07:00:00    3
2013-11-04 08:00:00    3
2013-11-04 09:00:00    3
2013-11-04 10:00:00    3
2013-11-04 11:00:00    3
Freq: H, dtype: int64

Also, if we were to have the values change at midnight UTC, there might be the undesirable possibility of repeated conversions going "around the world" and ending up off by a day.

[jreback]

@ischwabacher interesting, so you are ok with reindexing a tz-aware with a UTC (explicity) series, but not a naive series. Maybe like tz_convert/localize should disallow a tz-aware with naive. (may go even further and require them to be the SAME tz). That way the rule is simple:

• if both have same tz or are both naive -> ok
• else raise