Bugs related to DatetimeIndex, date_range. Question: Is is possible to create a index of datetime, without being auto converted to a DatetimeIndex?

[xdong]

Is is possible to create a index of datetime.datetime, without being auto converted to a DatetimeIndex containing Timestamp's? Even though datetime.datetime is not efficient, I thought the user should have a choice of using it for backward compatibility.

I wrote a lot of code using Index containing timezone-aware datetime.datetime. This caused me a lot of headache when I tried to port my codes from 07.3 to 10.1.

I also encountered several weird bugs (in 10.1).

(I) When DatetimeIndex is assigned as a column of a DataFrame, a silent error occurs:

In [1]: from pandas import *

In [2]: a = date_range('20130101', periods=3, tz='US/Eastern')

In [3]: b = date_range('20130215', periods=3, tz='US/Eastern')

In [4]: Series(a, b) #works as expected
Out[4]:
2013-02-15 00:00:00-05:00 2013-01-01 05:00:00
2013-02-16 00:00:00-05:00 2013-01-02 05:00:00
2013-02-17 00:00:00-05:00 2013-01-03 05:00:00
Freq: D

In [5]: DataFrame({'a': a}, b) #???
Out[5]:
a
2013-02-15 00:00:00-05:00 1970-01-18 05:00:00
2013-02-16 00:00:00-05:00 1970-01-19 05:00:00
2013-02-17 00:00:00-05:00 1970-01-20 05:00:00

(II) concat two Series will result in an incorrect DatetimeIndex:

In [18]: dates_a = [Timestamp('20130219 13:00', tz='US/Eastern'), Timestamp('20130220 13:00', tz='US/Eastern')]

In [19]: dates_b = [Timestamp('20130222 13:00', tz='US/Eastern'), Timestamp('20130223 13:00', tz='US/Eastern')]

In [20]: a = Series(range(2), dates_a)

In [21]: b = Series(range(2), dates_b)

In [22]: concat([a, b])
Out[22]:
2013-02-19 18:00:00-05:00 0
2013-02-20 18:00:00-05:00 1
2013-02-22 18:00:00-05:00 0
2013-02-23 18:00:00-05:00 1

Note that the index of Out[22] is wrong (time shifted by 5 hours).

(III) issue #2906 (already fixed in #2935)

[stephenwlin]

Well, i'll let someone else comment on backward compatibility and whether or not there's any way around implicit conversions to datetime64..I don't believe there is unfortunately but I could be wrong as the whole switch was before my time with pandas.

Anyway, unless I'm doing something wrong, the first issue seems to be resolved in master already (I think @jreback fixed it while overhauling other issues with Series/DataFrame constructor inconsistencies)...I don't think its a tz-aware-specific issue as much as a general numpy datetime64[ns]->object bug that we're unfortunately having to spend a lot of effort masking on the pandas side with special-case logic...as for the second issue, I've figured out what the issue is and will submit a PR for once I test it some more.

Sorry for the inconvenience, I've been on the tz-awareness bug thread for a little while now (eg #2788, #2935) so feel free to ping me with @stephenwlin if you come across more...

[jreback]

I think this will work in prior versions
just specify 'object' as your dtype

In [5]: s = pd.Series([ datetime.datetime(2001,1,1), datetime.datetime(2001,1,2) ], dtype='object')

In [6]: print s
0    2001-01-01 00:00:00
1    2001-01-02 00:00:00
Dtype: object

In [7]: print s[0]
2001-01-01 00:00:00

In [8]: type(s[0])
Out[8]: datetime.datetime

In [9]: print pd.__version__
0.11.0.dev-5ad19f1

[stephenwlin]

ok i integrated a fix into #2935

[stephenwlin]

the good news is that these bugs are pretty shallow actually, since all the logic to handle tz-awareness is already there: mostly boils down to 1-2 line fixes to correct where the tz gets double-counted or not counted in a conversion (it's basically a parity issue)

[xdong]

Thanks @stephenwlin for the fixes.

Thanks @jreback for the pointer. However it doesn't solve my problem. Yes, I can construct a stand alone Index with datetime.datetime entries. But the Index is auto converted to DatetimeIndex when I use it as the index of a Series.

In [12]: i = Index([datetime.datetime(2013, 1, 1), datetime.datetime(2013, 2, 27)], dtype='object')

In [13]: i # OK, not converted to DatetimeIndex
Out[13]: Index([2013-01-01 00:00:00, 2013-02-27 00:00:00], dtype=object)

In [14]: s = Series(range(2), i)

In [15]: s.index # now it's converted
Out[15]:
<class 'pandas.tseries.index.DatetimeIndex'>
[2013-01-01 00:00:00, 2013-02-27 00:00:00]
Length: 2, Freq: None, Timezone: None

[jreback]

I think the issue is that since the indicies for datetimes are actually int64 under the hood somewhere the conversion is happening....

an alternative is to instread of a series, use a frame with your datetimes in an object column and say an integer index (or whatever).

pseudo codish:

df = DataFrame({ 'A' : Series([datetimes...],dtype='object'), 'B' : dataseries })

df[df.A > df.A[10]]

but this will be REALLY slow. if you go this route I would then keep an index map (a dict) so you can just index directly

index_map = dict([ (i,d) for i, d in enumerate(datestimes_list) ])
df[index_map[0]:index_map[100]] or something like that

[jreback]

close this?

[stephenwlin]

yep