Sum of TimedeltaIndex raises TypeError: len() of unsized object

[H-SG]

Example code

import pandas as pd
import numpy as np

dti = pd.date_range('2018-01-01', periods=100, freq='T')
dtd = dti[1:] - dti[:-1]

print(np.sum(dtd))

Problem description

Summing a TimedeltaIndex when using pandas 0.24.0 or newer with any version of numpy, or pandas 0.23.4 with any version of numpy newer than 0.15.4 fails, np.sum(dtd) gives the following output:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-d64fe48d0fec> in <module>
      6 
      7 print(np.mean(dtd)) # works
----> 8 print(np.sum(dtd)) # breaks

~/.local/lib/python3.6/site-packages/numpy/core/fromnumeric.py in sum(a, axis, dtype, out, keepdims, initial)
   2074 
   2075     return _wrapreduction(a, np.add, 'sum', axis, dtype, out, keepdims=keepdims,
-> 2076                           initial=initial)
   2077 
   2078 

~/.local/lib/python3.6/site-packages/numpy/core/fromnumeric.py in _wrapreduction(obj, ufunc, method, axis, dtype, out, **kwargs)
     84                 return reduction(axis=axis, out=out, **passkwargs)
     85 
---> 86     return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
     87 
     88 

~/.local/lib/python3.6/site-packages/pandas/core/indexes/base.py in __array_wrap__(self, result, context)
    658         attrs = self._get_attributes_dict()
    659         attrs = self._maybe_update_attributes(attrs)
--> 660         return Index(result, **attrs)
    661 
    662     @cache_readonly

~/.local/lib/python3.6/site-packages/pandas/core/indexes/base.py in __new__(cls, data, dtype, copy, name, fastpath, tupleize_cols, **kwargs)
    301                   (dtype is not None and is_timedelta64_dtype(dtype))):
    302                 from pandas.core.indexes.timedeltas import TimedeltaIndex
--> 303                 result = TimedeltaIndex(data, copy=copy, name=name, **kwargs)
    304                 if dtype is not None and _o_dtype == dtype:
    305                     return Index(result.to_pytimedelta(), dtype=_o_dtype)

~/.local/lib/python3.6/site-packages/pandas/core/indexes/timedeltas.py in __new__(cls, data, unit, freq, start, end, periods, closed, dtype, copy, name, verify_integrity)
    250 
    251         # check that we are matching freqs
--> 252         if verify_integrity and len(data) > 0:
    253             if freq is not None and not freq_infer:
    254                 index = cls._simple_new(data, name=name)

TypeError: len() of unsized object

Other methods such as np.mean() and np.median() continue to function normally

Expected Output

5940000000000 nanoseconds

Output of pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.7.final.0
python-bits: 64
OS: Linux
OS-release: 4.15.0-45-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.1
pytest: None
pip: 19.0.2
setuptools: 40.8.0
Cython: None
numpy: 1.16.1
scipy: 1.2.1
pyarrow: None
xarray: None
IPython: 7.2.0
sphinx: None
patsy: 0.5.1
dateutil: 2.8.0
pytz: 2018.9
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 3.0.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: 4.3.1
bs4: 4.7.1
html5lib: 1.0.1
sqlalchemy: 1.2.17
pymysql: None
psycopg2: 2.7.7 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

[WillAyd]

Can you post versions from both environments where you had the issue? Seems kind of strange that the version of numpy only matters for one of the two pandas versions you've tested

[H-SG]

It's the same environment, I was updating my numpy and pandas libraries when I ran into the issue, all other packages remained the same. Playing around with installing specific versions of each led to the strange combination of working versions.

[jorisvandenbossche]

The above code indeed shows a regression in my 0.23.4 vs master environments. I have numpy 1.13 in both of them, so I don't think this is related to numpy.

It's the repr that fails, and it seems to be related to the fact that it now returns an "invalid" TimedeltaIndex, instead of a numpy.timedelta64 scalar:

In [8]: pd.__version__
Out[8]: '0.23.4'

In [9]: res = np.sum(dtd)

In [10]: type(res)
Out[10]: numpy.timedelta64

vs

In [15]: pd.__version__                                                                                                                                                         
Out[15]: '0.25.0.dev0+113.gb8306f19d.dirty'

In [16]: res = np.sum(dtd)                                                                                                                                                      

In [17]: type(res)                                                                                                                                                              
Out[17]: pandas.core.indexes.timedeltas.TimedeltaIndex

In [18]: res.values                                                                                                                                                             
Out[18]: array(5940000000000, dtype='timedelta64[ns]')

In [19]: res.values.ndim                                                                                                                                                        
Out[19]: 0

[jorisvandenbossche]

Might be related to the fact that the TimedeltaIndex/TimedeltaArray constructor no longer fails on scalars, but produces an invalid index/array:

In [12]: pd.__version__
Out[12]: '0.23.4'

In [13]: idx = pd.TimedeltaIndex(np.array(1, dtype='timedelta64[ns]'))
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-13-7b4cf68e5224> in <module>()
----> 1 idx = pd.TimedeltaIndex(np.array(1, dtype='timedelta64[ns]'))

~/miniconda3/lib/python3.5/site-packages/pandas/core/indexes/timedeltas.py in __new__(cls, data, unit, freq, start, end, periods, closed, dtype, copy, name, verify_integrity)
    250 
    251         # check that we are matching freqs
--> 252         if verify_integrity and len(data) > 0:
    253             if freq is not None and not freq_infer:
    254                 index = cls._simple_new(data, name=name)

TypeError: len() of unsized object

vs

In [29]: pd.__version__  
Out[29]: '0.25.0.dev0+113.gb8306f19d.dirty'

In [30]: idx = pd.TimedeltaIndex(np.array(1, dtype='timedelta64[ns]'))

In [31]: idx.array._data  
Out[31]: array(1, dtype='timedelta64[ns]')

cc @TomAugspurger @jbrockmendel seems EA refactor related

[jbrockmendel]

@jorisvandenbossche I think you're right that the TimedeltaArray constructor is behaving oddly. Would __array_ufunc__ be the place to look to figure out why np.sum(tdi) behaves differently from np.sum(tdi._data)?

[jorisvandenbossche]

Would __array_ufunc__ be the place to look to figure out why np.sum(tdi) behaves differently from np.sum(tdi._data)?

Long term, yes. But short term simply fixing the constructor should also fix the sum bug (due to the way how numpy deal with it), as you already did in the PR in the meantime.