Regression in 0.24: to_timedelta handling of float values

[Sup3rGeo]

Code Sample, a copy-pastable example if possible

Creating an TimedeltaIndex with 1 microsecond steps:

import pandas as pd
import scipy as sp
t = sp.arange(0,1,1e-6)
index = pd.to_timedelta(t, unit='s')

Problem description

Output in 0.24:

We have repeated index values (e.g. 99992 in the image) and missing ones (e.g. 99994 in the image).

This looks like some sort of rounding issue.

Expected Output

Output in 0.23.4:

In this version everything is okay.

Output of pd.show_versions()

pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.7.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 58 Stepping 9, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.24.0
pytest: 4.1.0
pip: 18.1
setuptools: 39.0.1
Cython: 0.29.2
numpy: 1.15.4
scipy: 1.1.0
pyarrow: 0.11.1
xarray: 0.11.2
IPython: 7.2.0
sphinx: 1.8.3
patsy: None
dateutil: 2.7.5
pytz: 2018.7
blosc: 1.7.0
bottleneck: 1.2.1
tables: 3.4.4
numexpr: 2.6.9
feather: None
matplotlib: 3.0.2
openpyxl: 2.5.12
xlrd: 1.2.0
xlwt: 1.3.0
xlsxwriter: 1.1.2
lxml.etree: 4.3.0
bs4: 4.7.1
html5lib: 1.0.1
sqlalchemy: 1.2.15
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: 0.2.0
fastparquet: 0.2.1
pandas_gbq: None
pandas_datareader: None
gcsfs: None

[TomAugspurger]

Can you fill out the issue?

…

On Feb 1, 2019, at 11:23, Victor Maryama ***@***.***> wrote: Code Sample, a copy-pastable example if possible # Your code here Problem description [this should explain why the current behaviour is a problem and why the expected output is a better solution.] Note: We receive a lot of issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates! Note: Many problems can be resolved by simply upgrading pandas to the latest version. Before submitting, please check if that solution works for you. If possible, you may want to check if master addresses this issue, but that is not necessary. For documentation-related issues, you can check the latest versions of the docs on master here: https://pandas-docs.github.io/pandas-docs-travis/ If the issue has not been resolved there, go ahead and file it in the issue tracker. Expected Output Output of pd.show_versions() — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[Sup3rGeo]

Sorry, was in the process of doing it!

[TomAugspurger]

Thanks. Likely some rounding issue. Are floating point values allowed in to_timedelta?

[gfyoung]

cc @jreback @mroeschke

Feel free to change milestone depending on what your thoughts on this are.

[jorisvandenbossche]

Are floating point values allowed in to_timedelta?

It's not really documented very clearly (under unit there is a mention of "float number"), but in any case it works in practice and is also tested behaviour:

pandas/pandas/tests/indexes/timedeltas/test_tools.py

Lines 104 to 108 in 5278cc6

	# Tests with fractional seconds as input:
	expected = np.array(
	[0, 500000000, 800000000, 1200000000], dtype='timedelta64[ns]')
	result = to_timedelta([0., 0.5, 0.8, 1.2], unit='s', box=False)
	tm.assert_numpy_array_equal(expected, result)

So clearly a regression I would say.

[jreback]

this is just a precision issue.

In [7]: index.asi8
Out[7]: 
array([        0,      1000,      2000, ..., 999996999, 999998000,
       999999000])

In [8]: np.diff(index.asi8)
Out[8]: array([1000, 1000, 1000, ...,  999, 1001, 1000])

try this to see full precision printing

In [17]: [t._repr_base(format='all') for t in index[0:10]]
Out[17]: 
['0 days 00:00:00.000000000',
 '0 days 00:00:00.000001000',
 '0 days 00:00:00.000002000',
 '0 days 00:00:00.000003000',
 '0 days 00:00:00.000004000',
 '0 days 00:00:00.000005000',
 '0 days 00:00:00.000006000',
 '0 days 00:00:00.000007000',
 '0 days 00:00:00.000008000',
 '0 days 00:00:00.000009000']

not a bug / regression at all; I suppose docs could be enhanced