BUG: Timedelta drops decimals if precision is greater than nanoseconds #36771

phofl · 2020-10-01T12:09:11Z

closes BUG: to_timedelta drops decimals from input if precision is greater than nanoseconds #36738
tests added / passed
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

I am not quite sure, if we should round to nanoseconds or cut if off after the 9th decimal. I implemented a simple cut off, but would be happy to change if rounding is better.

I would also add a note to the user guide if the future behavior is clear

phofl · 2020-10-01T13:35:49Z

/azp run

azure-pipelines · 2020-10-01T13:35:53Z

Commenter does not have sufficient privileges for PR 36771 in repo pandas-dev/pandas

phofl · 2020-10-01T13:36:25Z

Failure seems weird, maybe rerunning the pipeline?

mroeschke · 2020-10-11T21:08:11Z

I think truncation is reasonable. I think it would be good to document this behavior in Timedelta and to_timedelta.

Could you also add a test that goes through to_timedelta?

� Conflicts: � doc/source/whatsnew/v1.2.0.rst

phofl · 2020-10-11T21:41:26Z

The test ran through to_timedelta but not through Timedelta. Added this one. Also added a note to the docstrings

pandas/tests/tools/test_to_timedelta.py

phofl · 2020-10-11T23:44:15Z

Failure unrelated

� Conflicts: � doc/source/whatsnew/v1.2.0.rst

github-actions · 2020-11-11T00:08:45Z

This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.

mroeschke · 2020-11-11T00:52:20Z

Mind merge master @phofl? This LGTM after that

phofl · 2020-11-11T11:09:46Z

@mroeschke done

phofl · 2020-11-11T11:42:05Z

ci failure is from master and will be fixed with #37756

jorisvandenbossche · 2020-11-11T11:50:23Z

@phofl I merged that PR, so you can update here as well now

phofl · 2020-11-11T11:58:43Z

Thanks, done

jbrockmendel · 2020-11-12T04:09:11Z

pandas/core/tools/timedeltas.py

@@ -66,6 +66,12 @@ def to_timedelta(arg, unit=None, errors="raise"):
    to_datetime : Convert argument to datetime.
    convert_dtypes : Convert dtypes.

+    Notes
+    -----
+


no extra line here

Thx, changed

jbrockmendel · 2020-11-12T04:10:01Z

pandas/tests/tools/test_to_timedelta.py

+            ("8:53:08.7180000001", "8:53:08.7180000001"),
+        ],
+    )
+    @pytest.mark.parametrize("func", ["Timedelta", "to_timedelta"])


use pd.Timedelta and pd.to_datetime instead of passing these as strings and using getattr below

pandas/tests/tools/test_to_timedelta.py

jbrockmendel · 2020-11-14T03:30:34Z

doc/source/whatsnew/v1.2.0.rst

@@ -404,6 +404,7 @@ Datetimelike
 - Bug in :meth:`DatetimeArray.shift` incorrectly allowing ``fill_value`` with a mismatched timezone (:issue:`37299`)
 - Bug in adding a :class:`BusinessDay` with nonzero ``offset`` to a non-scalar other (:issue:`37457`)
 - Bug in :func:`to_datetime` with a read-only array incorrectly raising (:issue:`34857`)
+- Bug in :class:`Timedelta` incorrectly deleted all decimals when input had a higher precision than nanoseconds (:issue:`36738`)


"incorrectly truncating to sub-second portion of a string input when it has precision higher than nanoseconds"

jbrockmendel · 2020-11-14T03:32:59Z

pandas/_libs/tslibs/timedeltas.pyx

@@ -1143,6 +1145,9 @@ class Timedelta(_Timedelta):
    Notes
    -----
    The ``.value`` attribute is always in ns.
+
+    If the precision is higher than nanoseconds, the precision of the duration is
+    truncated to nanoseconds.


clarify this is for a string input

do additional digits round or just drop (and if just drop, are we sure thats the desired behavior?)

Dropd additional digits. Was not sure either so I started with the easier implementation
#36771 (comment)

jbrockmendel · 2020-11-14T03:33:57Z

pandas/tests/tools/test_to_timedelta.py

+    @pytest.mark.parametrize(
+        ("input", "expected"),
+        [
+            ("8:53:08.71800000001", "8:53:08.718"),


can you add a case with the trailing digit being 9 to clarify the truncating/rounding distinction

jreback · 2020-11-14T16:46:43Z

pandas/tests/tools/test_to_timedelta.py

+        # GH: 36738
+        expected = pd.Timedelta(expected)
+        result = func(input)
+        print(result)


remove the print

Sorry forgot that, is removed

jreback · 2020-11-14T16:47:00Z

doc/source/whatsnew/v1.2.0.rst

@@ -404,6 +404,7 @@ Datetimelike
 - Bug in :meth:`DatetimeArray.shift` incorrectly allowing ``fill_value`` with a mismatched timezone (:issue:`37299`)
 - Bug in adding a :class:`BusinessDay` with nonzero ``offset`` to a non-scalar other (:issue:`37457`)
 - Bug in :func:`to_datetime` with a read-only array incorrectly raising (:issue:`34857`)
+- Bug in :class:`Timedelta` incorrectly truncating to sub-second portion of a string input when it has precision higher than nanoseconds (:issue:`36738`)


can you move to section below.

jreback · 2020-11-15T17:31:51Z

thanks @phofl

phofl added 2 commits October 1, 2020 14:04

Fix timedelta problem when input is more precise than nanoseconds

e1e4296

Modify whatsnew

78a9938

phofl added the Timedelta Timedelta data type label Oct 1, 2020

phofl added 2 commits October 11, 2020 23:20

Merge branch 'master' of https://github.com/pandas-dev/pandas into 36738

52463cf

� Conflicts: � doc/source/whatsnew/v1.2.0.rst

Add docstring and test

4d6c9dc

mroeschke approved these changes Oct 11, 2020

View reviewed changes

Fix docstring

f4ff3e0

jreback requested changes Oct 11, 2020

View reviewed changes

pandas/tests/tools/test_to_timedelta.py Outdated Show resolved Hide resolved

Add additional tests

f79106d

Merge branch 'master' of https://github.com/pandas-dev/pandas into 36738

e3c133d

� Conflicts: � doc/source/whatsnew/v1.2.0.rst

github-actions bot added the Stale label Nov 11, 2020

Merge branch 'master' of https://github.com/pandas-dev/pandas into 36738

193921e

mroeschke approved these changes Nov 11, 2020

View reviewed changes

mroeschke removed the Stale label Nov 11, 2020

mroeschke added this to the 1.2 milestone Nov 11, 2020

jbrockmendel reviewed Nov 12, 2020

View reviewed changes

pandas/tests/tools/test_to_timedelta.py Show resolved Hide resolved

phofl added 2 commits November 12, 2020 13:01

Adress review comments

83111d9

Merge branch 'master' of https://github.com/pandas-dev/pandas into 36738

d038085

jbrockmendel reviewed Nov 14, 2020

View reviewed changes

Add comments and testcases

4ce27c8

jreback requested changes Nov 14, 2020

View reviewed changes

Move whatsnew

8b00890

jreback approved these changes Nov 14, 2020

View reviewed changes

jreback merged commit d85e9a2 into pandas-dev:master Nov 15, 2020

phofl deleted the 36738 branch November 15, 2020 17:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Timedelta drops decimals if precision is greater than nanoseconds #36771

BUG: Timedelta drops decimals if precision is greater than nanoseconds #36771

phofl commented Oct 1, 2020

phofl commented Oct 1, 2020

azure-pipelines bot commented Oct 1, 2020

phofl commented Oct 1, 2020

mroeschke commented Oct 11, 2020

phofl commented Oct 11, 2020

phofl commented Oct 11, 2020

github-actions bot commented Nov 11, 2020

mroeschke commented Nov 11, 2020

phofl commented Nov 11, 2020

phofl commented Nov 11, 2020

jorisvandenbossche commented Nov 11, 2020

phofl commented Nov 11, 2020

jbrockmendel Nov 12, 2020

phofl Nov 12, 2020

jbrockmendel Nov 12, 2020

phofl Nov 12, 2020

jbrockmendel Nov 14, 2020

phofl Nov 14, 2020

jbrockmendel Nov 14, 2020

phofl Nov 14, 2020

jbrockmendel Nov 14, 2020

phofl Nov 14, 2020

jreback Nov 14, 2020

phofl Nov 14, 2020

jreback Nov 14, 2020

phofl Nov 14, 2020

jreback commented Nov 15, 2020

BUG: Timedelta drops decimals if precision is greater than nanoseconds #36771

BUG: Timedelta drops decimals if precision is greater than nanoseconds #36771

Conversation

phofl commented Oct 1, 2020

phofl commented Oct 1, 2020

azure-pipelines bot commented Oct 1, 2020

phofl commented Oct 1, 2020

mroeschke commented Oct 11, 2020

phofl commented Oct 11, 2020

phofl commented Oct 11, 2020

github-actions bot commented Nov 11, 2020

mroeschke commented Nov 11, 2020

phofl commented Nov 11, 2020

phofl commented Nov 11, 2020

jorisvandenbossche commented Nov 11, 2020

phofl commented Nov 11, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Nov 15, 2020