pd to_datetime with format specified returns doesn't match format specified instead of OutOfBoundsDatetime

[asishm]

Code Sample, a copy-pastable example if possible

In [70]: pd.to_datetime('2417-10-27 00:00:00', format='%Y-%m-%d %H:%M:%S')
ValueError: time data 2417-10-27 00:00:00 doesn't match format specified

--------------------------------------------------------------------------- TypeError Traceback (most recent call last) ~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\tools\datetimes.py in _convert_listlike(arg, box, format, name, tz) 376 try: --> 377 values, tz = conversion.datetime_to_datetime64(arg) 378 return DatetimeIndex._simple_new(values, name=name, tz=tz)

pandas_libs\tslibs\conversion.pyx in pandas._libs.tslibs.conversion.datetime_to_datetime64()

TypeError: Unrecognized value type: <class 'str'>

During handling of the above exception, another exception occurred:

ValueError Traceback (most recent call last)
in ()
----> 1 pd.to_datetime('2417-10-27 00:00:00', format='%Y-%m-%d %H:%M:%S')

~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\tools\datetimes.py in to_datetime(arg, errors, dayfirst, yearfirst, utc, box, format, exact, unit, infer_datetime_format, origin, cache)
467 result = _convert_listlike(arg, box, format)
468 else:
--> 469 result = _convert_listlike(np.array([arg]), box, format)[0]
470
471 return result

~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\tools\datetimes.py in _convert_listlike(arg, box, format, name, tz)
378 return DatetimeIndex._simple_new(values, name=name, tz=tz)
379 except (ValueError, TypeError):
--> 380 raise e
381
382 if arg is None:

~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\tools\datetimes.py in _convert_listlike(arg, box, format, name, tz)
366 dayfirst=dayfirst,
367 yearfirst=yearfirst,
--> 368 require_iso8601=require_iso8601
369 )
370

pandas_libs\tslib.pyx in pandas._libs.tslib.array_to_datetime()

pandas_libs\tslib.pyx in pandas._libs.tslib.array_to_datetime()

ValueError: time data 2417-10-27 00:00:00 doesn't match format specified

In [71]: pd.to_datetime('2417-10-27 00:00:00')
OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 2417-10-27 00:00:00

--------------------------------------------------------------------------- TypeError Traceback (most recent call last) ~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\tools\datetimes.py in _convert_listlike(arg, box, format, name, tz) 376 try: --> 377 values, tz = conversion.datetime_to_datetime64(arg) 378 return DatetimeIndex._simple_new(values, name=name, tz=tz)

pandas_libs\tslibs\conversion.pyx in pandas._libs.tslibs.conversion.datetime_to_datetime64()

TypeError: Unrecognized value type: <class 'str'>

During handling of the above exception, another exception occurred:

OutOfBoundsDatetime Traceback (most recent call last)
in ()
----> 1 pd.to_datetime('2417-10-27 00:00:00')

~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\tools\datetimes.py in to_datetime(arg, errors, dayfirst, yearfirst, utc, box, format, exact, unit, infer_datetime_format, origin, cache)
467 result = _convert_listlike(arg, box, format)
468 else:
--> 469 result = _convert_listlike(np.array([arg]), box, format)[0]
470
471 return result

~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\tools\datetimes.py in _convert_listlike(arg, box, format, name, tz)
378 return DatetimeIndex._simple_new(values, name=name, tz=tz)
379 except (ValueError, TypeError):
--> 380 raise e
381
382 if arg is None:

~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\tools\datetimes.py in _convert_listlike(arg, box, format, name, tz)
366 dayfirst=dayfirst,
367 yearfirst=yearfirst,
--> 368 require_iso8601=require_iso8601
369 )
370

pandas_libs\tslib.pyx in pandas._libs.tslib.array_to_datetime()

pandas_libs\tslib.pyx in pandas._libs.tslib.array_to_datetime()

pandas_libs\tslib.pyx in pandas._libs.tslib.array_to_datetime()

pandas_libs\tslib.pyx in pandas._libs.tslib.array_to_datetime()

pandas_libs\tslibs\np_datetime.pyx in pandas._libs.tslibs.np_datetime.check_dts_bounds()

OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 2417-10-27 00:00:00

Problem description

The first case should also raise the OutOfBoundsDatetime error. The doesn't match format specified error is misleading.

Expected Output

The first example should also raise an OutOfBoundsDatetime exception.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.5.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 61 Stepping 4, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.23.0
pytest: 3.5.1
pip: 10.0.1
setuptools: 39.1.0
Cython: 0.28.2
numpy: 1.14.3
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 6.4.0
sphinx: 1.7.4
patsy: 0.5.0
dateutil: 2.7.3
pytz: 2018.4
blosc: None
bottleneck: 1.2.1
tables: 3.4.3
numexpr: 2.6.5
feather: None
matplotlib: 2.2.2
openpyxl: 2.5.3
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.4
lxml: 4.2.1
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.7
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[gfyoung]

That makes sense. A PR would be most welcome here.

[affanv14]

Can I take this up?

[gfyoung]

@affanv14 : Go for it! FYI, unless someone has a PR open or has explicitly stated that they want to tackle it, the issue is fair game for anyone.

[affanv14]

pandas/pandas/_libs/tslib.pyx

Lines 684 to 697 in e5c90e5

	except OutOfBoundsDatetime:
	# GH#19382 for just-barely-OutOfBounds falling back to
	# dateutil parser will return incorrect result because
	# it will ignore nanoseconds
	if is_coerce:
	iresult[i] = NPY_NAT
	continue
	elif require_iso8601:
	if is_raise:
	raise ValueError("time data {val} doesn't "
	"match format specified"
	.format(val=val))
	return values, tz_out
	raise

So this is happening because a ValueError is raised in the 'elif require_iso6801' statement, I'm a little confused as to why this was there. would it be appropriate to just remove the these lines raising the ValueError?

[varadgunjal]

@gfyoung I'm having some trouble with a PR I made for this. May I request you for some help?