Wrong hour of DST end for Europe TZ

[izderadicka]

xref #11708

In Europe DST ends by 2 am (this year on Sunday 2015-10-25), when clock is moved one hour back. However in Timestamp this transition is one hour earlier ( 1 am - which is hour, when transition happens in US I think). pytz works as expected.

See code ( tested in pandas 0.17.0 and 0.16.2):

In [1]: import pandas as pd
In [2]: import pytz
In [3]: from pandas import Timestamp

In [7]: pd.__version__
Out[7]: u'0.17.0'

In [8]: cz_tz=pytz.timezone('Europe/Prague')

In [9]: Timestamp('2015-10-25 00:00', tz=cz_tz)
Out[9]: Timestamp('2015-10-25 00:00:00+0200', tz='Europe/Prague')

In [10]: Timestamp('2015-10-25 01:00', tz=cz_tz)
Out[10]: Timestamp('2015-10-25 02:00:00+0200', tz='Europe/Prague')

In [11]: Timestamp('2015-10-25 02:00', tz=cz_tz)
Out[11]: Timestamp('2015-10-25 02:00:00+0100', tz='Europe/Prague')

In [13]: from pandas.tseries.offsets import Hour

In [14]: Timestamp('2015-10-25 01:00', tz=cz_tz) + Hour(1)
Out[14]: Timestamp('2015-10-25 02:00:00+0100', tz='Europe/Prague')

In [16]: from datetime import datetime

In [17]: cz_tz.dst(datetime(2015,10,25, 1, 0))
Out[17]: datetime.timedelta(0, 3600)

In [18]: cz_tz.dst(datetime(2015,10,25, 2, 0))
---------------------------------------------------------------------------
AmbiguousTimeError                        Traceback (most recent call last)
<ipython-input-18-4b6d0ae06c09> in <module>()
----> 1 cz_tz.dst(datetime(2015,10,25, 2, 0))

/home/ivan/tmp/pandas-test/local/lib/python2.7/site-packages/pytz/tzinfo.pyc in dst(self, dt, is_dst)
    445             return None
    446         elif dt.tzinfo is not self:
--> 447             dt = self.localize(dt, is_dst)
    448             return dt.tzinfo._dst
    449         else:

/home/ivan/tmp/pandas-test/local/lib/python2.7/site-packages/pytz/tzinfo.pyc in localize(self, dt, is_dst)
    347         # ambiguous case
    348         if is_dst is None:
--> 349             raise AmbiguousTimeError(dt)
    350 
    351         # Filter out the possiblilities that don't match the requested

AmbiguousTimeError: 2015-10-25 02:00:00

In [19]: cz_tz.dst(datetime(2015,10,25, 2, 0), is_dst=True)
Out[19]: datetime.timedelta(0, 3600)

In [20]: cz_tz.dst(datetime(2015,10,25, 2, 0), is_dst=False)
Out[20]: datetime.timedelta(0)

In [21]: cz_tz.dst(datetime(2015,10,25, 3, 0))
Out[21]: datetime.timedelta(0)

[jorisvandenbossche]

I think it is an issue in the datetime string parsing, instead of the underlying datetime type itself:

In [18]: tz=pytz.timezone('Europe/Brussels')

In [19]: pd.Timestamp('2015-10-25 00:00', tz=tz)
Out[19]: Timestamp('2015-10-25 00:00:00+0200', tz='Europe/Brussels')

In [20]: pd.Timestamp('2015-10-25 01:00', tz=tz)   # <-- this should still be unambiguous and returns just a plain wrong timestamp
Out[20]: Timestamp('2015-10-25 02:00:00+0200', tz='Europe/Brussels')

In [21]: pd.Timestamp('2015-10-25 03:00', tz=tz)
Out[21]: Timestamp('2015-10-25 03:00:00+0100', tz='Europe/Brussels')

In [22]: pd.Timestamp('2015-10-25 00:00', tz=tz) + pd.Timedelta('1 hour')  # <-- manually adding one hour however gives the correct value
Out[22]: Timestamp('2015-10-25 01:00:00+0200', tz='Europe/Brussels')

In [23]: pd.Timestamp('2015-10-25 00:00', tz=tz) + pd.Timedelta('2 hour')
Out[23]: Timestamp('2015-10-25 02:00:00+0200', tz='Europe/Brussels')

In [24]: pd.Timestamp('2015-10-25 00:00', tz=tz) + pd.Timedelta('3 hour')
Out[24]: Timestamp('2015-10-25 02:00:00+0100', tz='Europe/Brussels')

In [25]: pd.Timestamp('2015-10-25 00:00', tz=tz) + pd.Timedelta('4 hour')
Out[25]: Timestamp('2015-10-25 03:00:00+0100', tz='Europe/Brussels')

[jreback]

not a bug a all. you are just using this in an incorrect manner. you need to localize a time, simply constructing is exactly that, it just takes what you give it.

In [16]: Timestamp('2015-10-25').tz_localize(cz_tz)
Out[16]: Timestamp('2015-10-25 00:00:00+0200', tz='Europe/Prague')

In [17]: Timestamp('2015-10-25 02:00:00').tz_localize(cz_tz)
AmbiguousTimeError: Cannot infer dst time from Timestamp('2015-10-25 02:00:00'), try using the 'ambiguous' argument

In [18]: Timestamp('2015-10-25 01:00:00').tz_localize(cz_tz)
Out[18]: Timestamp('2015-10-25 01:00:00+0200', tz='Europe/Prague')

In [19]: Timestamp('2015-10-25 03:00:00').tz_localize(cz_tz)
Out[19]: Timestamp('2015-10-25 03:00:00+0100', tz='Europe/Prague')

In [21]: Timestamp('2015-10-25 02:00:00').tz_localize(cz_tz, ambiguous=False)
Out[21]: Timestamp('2015-10-25 02:00:00+0100', tz='Europe/Prague')

In [22]: Timestamp('2015-10-25 02:00:00').tz_localize(cz_tz, ambiguous=True)
Out[22]: Timestamp('2015-10-25 02:00:00+0200', tz='Europe/Prague')

[jorisvandenbossche]

@jreback I agree that localizing is the better option, but I don't really see how the below can be correct:

In [3]: pd.Timestamp('2015-10-25 01:00', tz='Europe/Brussels')
Out[3]: Timestamp('2015-10-25 02:00:00+0200', tz='Europe/Brussels')

1 o'clock should not become 2 o'clock as it is both in the same DST part?

[jreback]

you misunderstand what is happening.

you are taking a naive time, and simply SETTING it to a tz. you HAVE to localize. If you don't you get the CURRENT time zone.

[jorisvandenbossche]

In any case, I agree this is not good documented. It is not really clear how the tz parameter is handled in Timestamp (no docs on that). I would expect it actually does a tz_localize afterwards?

BTW, DatetimeIndex handles this fine:

`In [15]: pd.DatetimeIndex(['2015-10-25 01:00'], tz='Europe/Brussels')
Out[15]: DatetimeIndex(['2015-10-25 01:00:00+02:00'], dtype='datetime64[ns]', freq=None, tz='Europe/Brussels')

and pytz itself also gives another result:

In [26]: tz = pytz.timezone('Europe/Brussels')

In [27]: print tz.localize(datetime.datetime(2015,10,25,1))
2015-10-25 01:00:00+02:00

In [28]: print pd.Timestamp('2015-10-25 01:00', tz=tz)
2015-10-25 02:00:00+02:00

(there is actually nothing ambiguous to pick in this case, 1am can only be 1am)

DatetimeIndex also raised an AmbiguousTimeError for '2015-10-25 02:00' (while Timestamp picks the second occurence), which seems the more correct behaviour.

@jreback It is true I don't really understand what is happening under the hood, but IMO this is not 'incorrect code' from a user perspective. This is perfectly allowed in our API (why is there otherwise a tz argument in a public class?), and is handled correctly in a related class (DatetimeIndex).

[jreback]

@jorisvandenbossche

I think this is using a different path for the localization when it is passing thru the Timestamp constructor. It should first create it as naive, THEN localize.

In [1]: pd.DatetimeIndex(['2015-10-25 01:00']).tz_localize(tz='Europe/Brussels')
Out[1]: DatetimeIndex(['2015-10-25 01:00:00+02:00'], dtype='datetime64[ns, Europe/Brussels]', freq=None)

In [2]: pd.DatetimeIndex(['2015-10-25 01:00'], tz='Europe/Brussels')
Out[2]: DatetimeIndex(['2015-10-25 01:00:00+02:00'], dtype='datetime64[ns, Europe/Brussels]', freq=None)

In [3]: Timestamp('2015-10-25 01:00').tz_localize('Europe/Brussels')
Out[3]: Timestamp('2015-10-25 01:00:00+0200', tz='Europe/Brussels')

In [4]: Timestamp('2015-10-25 01:00',tz='Europe/Brussels')
Out[4]: Timestamp('2015-10-25 02:00:00+0200', tz='Europe/Brussels')

I suspect the path for a dateutil tz actually works, but this does look incorrect for pytz (e.g. [3] == [4] should be True)

[jreback]

our DST experts
cc @rockg
cc @ischwabacher
cc @adamgreenhall
cc @sinhrks

[izderadicka]

Thanks for all comments - just from user perspective - it would be quite weird if Timestamp(x, tz=tz) behave differently from Timestamp(x).tz_localize(tz) - the result e in both cases is timestamp in same tz, right? And it should behave correctly - e.g. DST end is is 2 am in this time zone.