Fastpath for to_datetime when providing ISO format as keyword?

[jorisvandenbossche]

Say you have to parse some nicely ISO formatted date strings, you can just parse this with todatetime very fast. But if you were 'overcautious' and provided the format="%Y-%m-%d %H:%M:%S" for safety, this seems to be around 20 times slower.
Would it be possible to provide a fastpath for certain provided format strings (as already exists for %Y%m%d I think).

In [129]: s = pd.Series(pd.date_range('2000-01-01', periods=1000, freq='H'))

In [130]: s_as_dt_strings = s.apply(lambda x: x.strftime("%Y-%m-%dT%H:%M:%S.%f"))

In [131]: %timeit pd.to_datetime(s_as_dt_strings)
1000 loops, best of 3: 406 µs per loop

In [132]: %timeit pd.to_datetime(s_as_dt_strings, format="%Y-%m-%dT%H:%M:%S.%f")
100 loops, best of 3: 9.73 ms per loop

In [133]: s_as_dt_strings = s.apply(lambda x: x.strftime("%Y-%m-%d %H:%M:%S"))

In [134]: %timeit pd.to_datetime(s_as_dt_strings)
1000 loops, best of 3: 361 µs per loop

In [135]: %timeit pd.to_datetime(s_as_dt_strings, format="%Y-%m-%d %H:%M:%S")
100 loops, best of 3: 8.36 ms per loop

For non-standard formats, providing format does give a big improvement:

In [136]: s_as_dt_strings = s.apply(lambda x: x.strftime("%Y/%m/%d %H:%M:%S"))

In [137]: %timeit pd.to_datetime(s_as_dt_strings)
10 loops, best of 3: 92.2 ms per loop

In [138]: %timeit pd.to_datetime(s_as_dt_strings, format="%Y/%m/%d %H:%M:%S")
100 loops, best of 3: 9.08 ms per loop

[jreback]

if u provide a format I think it bypasses the infer_datetime_format option which is much faster if it knows the format

so need to short circuit on certain formats

[jorisvandenbossche]

But the infer_datetime_format is False by default no?

In [134]: %timeit pd.to_datetime(s_as_dt_strings)
1000 loops, best of 3: 382 µs per loop

In [135]: %timeit pd.to_datetime(s_as_dt_strings, infer_datetime_format=True)
1000 loops, best of 3: 963 µs per loop

In [136]: %timeit pd.to_datetime(s_as_dt_strings, format="%Y-%m-%dT%H:%M:%S.%f")
100 loops, best of 3: 9.36 ms per loop

But indeed, the fact that it is faster with infer_datetime_format (which guesses the same format as I provide with format=, I would think) seems to indicate that with providing format= it could also be faster by using some same short circuiting

[jreback]

exactly

I think needs a simple check if format is provided if we can fastpath it (eg for recognized formats), if so then just follow the infer_datetime_format=True path and ignore the format); I think this is done for YYYYMMDD (only) atm

[jreback]

or in the basic format whee we don't need to infer at all just pass it thru (eg your first timeit with the 3rd format)

[jorisvandenbossche]

With infer_datetime_format there is indeed a bypass for iso formatted strings: https://github.com/pydata/pandas/blob/master/pandas/tseries/tools.py#L251, so this should maybe also be done with provided formats

[chris-b1]

@jorisvandenbossche, fyi this is also closed by PR #10615