Timestamp() is wrong if np.datetime64 < 1678-01-01

[timcera]

The following should work, but since it doesn't should raise an error...

In [116]: pd.Timestamp(np.datetime64('1677-01-01'),'D')
Out[116]: Timestamp('2261-07-22 23:34:33.709551616', tz=None)

In [117]: pd.Timestamp(np.datetime64('1678-01-01'),'D')
Out[117]: Timestamp('1678-01-01 00:00:00', tz=None)

In [121]: pd.version
Out[121]: '0.11.1.dev-06cd915'

In [125]: sys.version
Out[125]: '3.3.2 (default, May 21 2013, 15:40:45) \n[GCC 4.8.0 20130502 (prerelease)]'

[SleepingPills]

I'll try to tackle this since I already mucked around with Timestamp.

[jreback]

great!

[danbirken]

The heart of the bug was not doing a _check_dts_bounds() in _get_datetime64_nanos when converting from a datetime64 that wasn't in [ns] units.

But fixing this lead to another issue: the way _check_dts_bounds() worked previously, it did a broad check on the validity of a pandas_datetimestruct.year, and then assuming that triggered, it did a more fine grained check using the integer value of the given datetime64. However, if you have a datetime64 in another unit, the integer value isn't useful for comparison (because by definition it will map into a valid datetime64 somewhere in the [ns] range), so you don't really know if it is valid or not.

So if you have a datetime64 that might be any unit, the only way to verify if it is in bounds is by just looking at the pandas_datetimestruct and just seeing if the values are in bounds or not. So I changed _check_dts_bounds() to do bounds checking this way.

[jreback]

can you add some tests with to_datetime (which ultimtately call s array_to_datetime) with a mix of np.datetime64 that have some valid, some invalid dtypes? (like you are doing with Timestamp, but with an array of them); the code paths are different, but should yield the same results

e.g.

result1 = np.array([ Timestamp(v) for v in test_values ])
result2 = pd.to_datetime(test_values,box=False)

assume several cases, test_values are valid (with various types of dtypes), and another case of an invalid value

thxs..

this is great stuff getting all of these bugs out of the way!

[danbirken]

Yeah this is a good point, I need to do a little more work here.

When you have a situation like:

In [10]: pd.to_datetime(['1000-01-01', '1000-01-02'])
Out[10]: array(['1000-01-01', '1000-01-02'], dtype=object)

You give it strings, it gives back strings because it can't convert them to valid datetime64[ns]-es because they are out of bounds.

Currently, if you give it datetime64s it can't deal with, it just maps them randomly into the datetime64[ns] range:

In [8]: pd.to_datetime([np.datetime64('1000-01-01')], box=False)
Out[8]: array(['2169-02-08T15:09:07.419103232-0800'], dtype='datetime64[ns]')

This is bad, but at least it has 2 nice properties: it doesn't throw errors and it always returns datetime64[ns]s.

So I think there are a few potential solutions. First of all, in all cases if coerce=True, we just convert the out-of-bounds dt64s into NaTs, so we'll ignore that case:

a) pd.to_datetime([np.datetime64('1000-01-01')], box=False) throws an error, pandas can't deal with it

b) pd.to_datetime([np.datetime64('1000-01-01')], box=False) returns back an array of datetime64[whatever]s. This will probably lead to weird situations elsewhere because I think a lot of code assumes it is dealing with datetime64[ns]es. I just did a quick test and if pd.to_datetime() returns the datetime64[whatevers]s right back, some other aspect of the code maps them to datetime64[ns]es. I didn't investigate too deeply because I assume it would be a big deal, if not pandas would just support arbitrary units and it doesn't.

c) pd.to_datetime([np.datetime64('1000-01-01')], box=False) returns back an array of strings ['1000-01-01']. IE, if we get an out-of-bounds datetime64, just convert it to a string and return that.

I like solution (c). Even though it is kind of surprising, it generally does the right thing and will make sure developers don't have to worry about non datetime64[ns]-es cropped up all over the place in these edge cases. Then in the future if pandas ever supports datetime64s of various units, it will be easy delete the code which converts the dt64s to strings and just return them.

Though I'm amenable to any of the above solutions or a different solution, no hugely strong feelings.

Also if this is confusing or I am missing something obvious, please let me know. I'm pretty new to this section of the code.

[jreback]

@danbirken

c) is the current/right way, but with one additional check

In [6]: x = np.datetime64('1000-01-01')

In [7]: x
Out[7]: numpy.datetime64('1000-01-01')

In [8]: x.dtype
Out[8]: dtype('<M8[D]')

There is a bug lurking that allows a np.datetime64 < ns range in on conversion (either the bounds are not checked or the conversion is wrong)

This should raise be NaT if coerce=True, or raise the bounds error (which ALWAYS raises)

better to let the user know that this is not an acceptable value

side issue is whether their are raises if coerce=True and the date is out of bounds (I don't know what it does currently, my thinking is that if you pass coerce=Truethen just make it NaTeven if out-of-bounds)

[jreback]

@danbirken another related issue: #3944

[jreback]

related is #4066 as well

[danbirken]

Alright, I'm a little confused as to the exact interface you are looking for and while digging through the code and all of these related issues i've found a few more potential fixes, so here are some clarification statements. Let me know if any of these are wrong:

1. pd.to_datetime(value) == pd.to_datetime([value])[0] should be true in all cases (there are some cases right now where this isn't true)

2. pd.to_datetime('1000-01-01', errors='ignore', box=False) should return '1000-01-01'

3. pd.to_datetime('1000-01-01', errors='raise', box=False) should throw ValueError

4. pd.to_datetime(np.datetime64('1000-01-01'), errors='ignore', box=False) should return '1000-01-01' --- ie, datetime64 converted to a string because it can't be represented as a datetime64[ns]

5. pd.to_datetime(np.datetime64('1000-01-01'), errors='raise', box=False) should throw ValueError

6. pd.to_datetime('1000-01-01', coerce=True) should return a NaT

7. pd.to_datetime(np.datetime64('1000-01-01'), coerce=True) should return a NaT

[jreback]

can you explain 4 again?

when is 1) not true?

all others look good