BUG: coercion of non-M8[ns] in datetime ops

[jreback]

import datetime
s = pd.Series(pd.date_range('20130101',periods=3))
s-pd.Timestamp('20130101')
s-datetime.datetime(2013,1')

This fails as the datetime64 is not converted properly (because numpy datetime ops suck)

s-np.datetime64('20130101')

e.g. np.datetime64('20130101').astype('M8[ns]') is a bug, no?

[jorisvandenbossche]

I am not fully following here. Isn't this just a limitation of numpy's string parsing? np.datetime64('2013-01-01').astype('M8[ns]') looks OK to me.

Numpy only parses ISO timestrings, so np.datetime64('20130101') will never work? Doing np.datetime64('2013-01-01').astype('M8[ns]') does not look wrong. Or is that not what you mean?

But in any case s-np.datetime64('2013-01-01') should still ideally work. Can pandas work around this? (by always doing a .astype('M8[ns]') in __rsub__?)

[jreback]

no I think it's just broken in numpy

np.datetime64('2013-01-01') make this dtype of M8[D] but I don't think it allows astype to M8ns

[jreback]

it still IS possible though I think you just have to get the value and figure it out based in the dtype (which is where pandas can handle it - if the astype worked then it would be easy)

[jorisvandenbossche]

In [94]: np.datetime64('2013-01-01')
Out[94]: numpy.datetime64('2013-01-01')

In [95]: np.datetime64('2013-01-01').astype('M8[ms]')
Out[95]: numpy.datetime64('2013-01-01T01:00:00.000+0100')

In [96]: np.datetime64('2013-01-01').astype('M8[ns]')
Out[96]: numpy.datetime64('2013-01-01T01:00:00.000000000+0100')

This looks like astype to M8[ns] works?

[jreback]

hmm maybe doesn't work in 1.7 I think

[jorisvandenbossche]

that was in 1.7.1 (and 1.8.1)

[jreback]

try exactly how I have it (the format)

[jorisvandenbossche]

In [98]: np.datetime64('20130101').astype('M8[ns]')
Out[98]: numpy.datetime64('2151-06-04T08:32:39.009206272+0200')

That is indeed bullshit output, but that is because numpy does not support non-iso string parsing, not because of the astype not working.
Without the astype, it is also not really working (see that the datetime is not parse, but it is strange it does not give an error here):

In [12]: np.datetime64('20130101')
Out[12]: numpy.datetime64('20130101')

[jorisvandenbossche]

That seems a bug in numpy, that is does not raise:

In [99]: np.datetime64('20130101')
Out[99]: numpy.datetime64('20130101')

In [100]: np.datetime64('20130101 10:00')
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-100-6968cb1137d9> in <module>()
----> 1 np.datetime64('20130101 10:00')

ValueError: Error parsing datetime string "20130101 10:00" at position 8

In [101]: np.datetime64('2013-01-01 10:00')
Out[101]: numpy.datetime64('2013-01-01T10:00+0100')

it does raise when there is also an hour and not only date. Or is there a reason a date-only would allow more flexible string parsing?

[jreback]

ahh ok

but we still don't handle this input correctly (a non ns numpy datetime input)

[jorisvandenbossche]

yes, so it a legitimate issue :-) (only the comment about astype was not correct)