API/BUG: hashing of datetimes is based on UTC values

[jreback]

These should are 3 different 'views' of the same time. We DO disambiguate these in mains. So we should do so when hashing as well.

xref #16346

In [1]: from pandas.util import hash_pandas_object

In [8]: hash_pandas_object(pd.date_range('20130101', periods=3, tz='UTC').tz_convert('US/Eastern'))
Out[8]: 
2012-12-31 19:00:00-05:00     4326795898974544501
2013-01-01 19:00:00-05:00     2833560015380952180
2013-01-02 19:00:00-05:00    14913883737423839247
Freq: D, dtype: uint64

In [9]: hash_pandas_object(pd.date_range('20130101', periods=3, tz='UTC'))
Out[9]: 
2013-01-01 00:00:00+00:00     4326795898974544501
2013-01-02 00:00:00+00:00     2833560015380952180
2013-01-03 00:00:00+00:00    14913883737423839247
Freq: D, dtype: uint64

In [10]: hash_pandas_object(pd.date_range('20130101', periods=3))
Out[10]: 
2013-01-01     4326795898974544501
2013-01-02     2833560015380952180
2013-01-03    14913883737423839247
Freq: D, dtype: uint64

[jreback]

cc @mrocklin

@TomAugspurger @jorisvandenbossche

from a practical perspective I don't think this makes a whole lot of difference, but should fix to be correct.

[jorisvandenbossche]

What alternative do you think off to hash upon? Hash the timezone separately and combine the hashes?

[jreback]

I think you could do something like this.

In [1]: df = pd.DataFrame({'tz': pd.date_range('20130101', periods=3, tz='UTC').tz_convert('US/Eastern'),
   ...: 'utc': pd.date_range('20130101', periods=3, tz='UTC'),
   ...: 'naive': pd.date_range('20130101', periods=3)})

In [2]: df
Out[2]: 
       naive                        tz                       utc
0 2013-01-01 2012-12-31 19:00:00-05:00 2013-01-01 00:00:00+00:00
1 2013-01-02 2013-01-01 19:00:00-05:00 2013-01-02 00:00:00+00:00
2 2013-01-03 2013-01-02 19:00:00-05:00 2013-01-03 00:00:00+00:00

In [3]: from pandas.util import hash_pandas_object

In [6]: hash_pandas_object(pd.DataFrame({'tz':df['tz'],'zone':df['tz'].dt.tz}), index=False)
Out[6]: 
0    11960632900184590671
1    17909201100930397932
2      244240496600445005
dtype: uint64

In [7]: hash_pandas_object(pd.DataFrame({'utc':df['tz'],'zone':df['utc'].dt.tz}), index=False)
Out[7]: 
0     557885042773898185
1    1996380570925580138
2    5435501107539799243
dtype: uint64

In [8]: hash_pandas_object(pd.DataFrame({'naivec':df['naive']}), index=False)
Out[8]: 
0    14376405836841727586
1     1052390041072582175
2    12596642793234779168
dtype: uint64

IOW, hash the tz as an additional column and combine (which is what we do with a DataFrame with index=False).

This would break backward compat for tz-aware, but (and maybe should document this more), that this is version-to-version hashing, it is not (necessarily) designed to be backward compat.