Skip to content

BUG: Fix parsing of ISO8601 durations #37159

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Oct 17, 2020
Merged

BUG: Fix parsing of ISO8601 durations #37159

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
9292728
Fix parsing of ISO8601 strings with empty period
mgmarino Oct 16, 2020
d8e8ad9
Add 'W' as a valid ISO 8601 designator
mgmarino Oct 16, 2020
fbf93da
Parse all components of ISO 8601 durations, even if S is not included
mgmarino Oct 16, 2020
949fa5e
Use chained comparison
mgmarino Oct 16, 2020
0511e01
Add what's new entry
mgmarino Oct 16, 2020
3786f8b
Remove limitation that seconds has two or fewer digits
mgmarino Oct 16, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions doc/source/whatsnew/v1.2.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -348,8 +348,7 @@ Datetimelike
Timedelta
^^^^^^^^^
- Bug in :class:`TimedeltaIndex`, :class:`Series`, and :class:`DataFrame` floor-division with ``timedelta64`` dtypes and ``NaT`` in the denominator (:issue:`35529`)
-
-
- Bug in parsing of ISO 8601 durations in :class:`Timedelta`, :meth:`pd.to_datetime` (:issue:`37159`, fixes :issue:`29773` and :issue:`36204`)

Timezones
^^^^^^^^^
Expand Down
28 changes: 13 additions & 15 deletions pandas/_libs/tslibs/timedeltas.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -604,7 +604,7 @@ cdef inline int64_t parse_iso_format_string(str ts) except? -1:

for c in ts:
# number (ascii codes)
if ord(c) >= 48 and ord(c) <= 57:
if 48 <= ord(c) <= 57:

have_value = 1
if have_dot:
Expand All @@ -620,27 +620,28 @@ cdef inline int64_t parse_iso_format_string(str ts) except? -1:
if not len(unit):
number.append(c)
else:
# if in days, pop trailing T
if unit[-1] == 'T':
unit.pop()
elif 'H' in unit or 'M' in unit:
if len(number) > 2:
raise ValueError(err_msg)
r = timedelta_from_spec(number, '0', unit)
result += timedelta_as_neg(r, neg)

neg = 0
unit, number = [], [c]
else:
if c == 'P':
pass # ignore leading character
if c == 'P' or c == 'T':
pass # ignore marking characters P and T
elif c == '-':
if neg or have_value:
raise ValueError(err_msg)
else:
neg = 1
elif c in ['D', 'T', 'H', 'M']:
elif c in ['W', 'D', 'H', 'M']:
unit.append(c)
if c in ['H', 'M'] and len(number) > 2:
raise ValueError(err_msg)
r = timedelta_from_spec(number, '0', unit)
result += timedelta_as_neg(r, neg)

neg = 0
unit, number = [], []
elif c == '.':
# append any seconds
if len(number):
Expand All @@ -661,11 +662,8 @@ cdef inline int64_t parse_iso_format_string(str ts) except? -1:
r = timedelta_from_spec(number, '0', dec_unit)
result += timedelta_as_neg(r, neg)
else: # seconds
if len(number) <= 2:
r = timedelta_from_spec(number, '0', 'S')
result += timedelta_as_neg(r, neg)
else:
raise ValueError(err_msg)
r = timedelta_from_spec(number, '0', 'S')
result += timedelta_as_neg(r, neg)
else:
raise ValueError(err_msg)

Expand Down
9 changes: 8 additions & 1 deletion pandas/tests/scalar/timedelta/test_constructors.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,14 @@ def test_overflow_on_construction():
("P0DT0H0M0.001S", Timedelta(milliseconds=1)),
("P0DT0H1M0S", Timedelta(minutes=1)),
("P1DT25H61M61S", Timedelta(days=1, hours=25, minutes=61, seconds=61)),
("PT1S", Timedelta(seconds=1)),
("PT0S", Timedelta(seconds=0)),
("P1WT0S", Timedelta(days=7, seconds=0)),
("P1D", Timedelta(days=1)),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we handle negatives of these? (IIRC that's on one of the issues)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't change the behavior of negativity, and the question is how to handle it. One of the only references I can find regarding the expected behavior is this comment, which also seems to indicate it's not very well defined in the ISO 8601 standard. How would you feel about pulling that out into a separate issue?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes pls create a separate issue. only thing for now is do we raise if a negative is passed in? (i think we should unless / until we decide on how to handle it), though isn't the result just negative of the Timedelta itself which is well defined.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, will do.

No, nothing is raised when it is negative, and the behavior is, admittedly, quite odd, i.e.

"P-6DT0H50M3.010010012S" parses as Timedelta( days=-6, minutes=50, seconds=3, milliseconds=10, microseconds=10, nanoseconds=12, )
, and the negative is only allowed right after the P descriptor. A negative in any other position will raise an error. As far as I can tell, there are two possibilities to go here:

  • only support an overall "-" e.g. "-P6DT1H" = Timedelta('-7 days +23:00:00') and/or
  • support positive/negative on each, e.g. "P7DT-1H3M" = Timedelta('6 days 23:03:00')

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue now here: #37172. I would prefer to change/update the negativity behavior in a separate PR if that's ok.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep that's great. well scoped PRs are good. one thing at a time. and thanks for working on this.

("P1DT1H", Timedelta(days=1, hours=1)),
("P1W", Timedelta(days=7)),
("PT300S", Timedelta(seconds=300)),
("P1DT0H0M00000000000S", Timedelta(days=1)),
],
)
def test_iso_constructor(fmt, exp):
Expand All @@ -241,7 +249,6 @@ def test_iso_constructor(fmt, exp):
"PDTHMS",
"P0DT999H999M999S",
"P1DT0H0M0.0000000000000S",
"P1DT0H0M00000000000S",
"P1DT0H0M0.S",
],
)
Expand Down