Skip to content

BUG: Series.ffill() of Series with mixed dtypes containing tz-aware datetimes, fails. #14956

Closed
@RobertasA

Description

@RobertasA

Problem description

Doing a ffill/bfill on a mixed dtype series fails if there are tz-aware datetime.datetime instances.
It does work with timestamps, but as it's mixed type column anyway, it probably should just work with non-native types, leaving them as it is. Furthermore, it used to work in 0.18.1.

Code Sample, a copy-pastable example if possible

>>> s = pd.Series([datetime.datetime(2015, 1, 1, tzinfo=pytz.utc), 1])
>>> s.ffill()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/core/generic.py", line 3302, in ffill
    limit=limit, downcast=downcast)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/core/series.py", line 2368, in fillna
    **kwargs)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/core/generic.py", line 3250, in fillna
    downcast=downcast)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/core/internals.py", line 3177, in interpolate
    return self.apply('interpolate', **kwargs)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/core/internals.py", line 3056, in apply
    applied = getattr(b, f)(**kwargs)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/core/internals.py", line 918, in interpolate
    downcast=downcast, mgr=mgr)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/core/internals.py", line 960, in _interpolate_with_fill
    return self._maybe_downcast(blocks, downcast)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/core/internals.py", line 1917, in _maybe_downcast
    for b in blocks])
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/core/internals.py", line 1876, in convert
    self.values.ravel(), **fn_kwargs).reshape(self.values.shape)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/types/cast.py", line 641, in _soft_convert_objects
    values = lib.maybe_convert_objects(values, convert_datetime=datetime)
  File "pandas/src/inference.pyx", line 842, in pandas.lib.maybe_convert_objects (pandas/lib.c:56711)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/util/decorators.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/tseries/index.py", line 319, in __new__
    yearfirst=yearfirst)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/util/decorators.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/tseries/tools.py", line 428, in to_datetime
    return _convert_listlike(arg, box, format)
  File "/Users/charon/.virtualenvs/pandas/lib/python2.7/site-packages/pandas/tseries/tools.py", line 413, in _convert_listlike
    raise e
ValueError: Tz-aware datetime.datetime cannot be converted to datetime64 unless utc=True

Expected Output

(Output returned in 0.18.1)

>>> s = pd.Series([datetime.datetime(2015, 1, 1, tzinfo=pytz.utc), 1])
>>> s.ffill()
0    2015-01-01 00:00:00+00:00
1                            1
dtype: object

Output of pd.show_versions()

Paste the output here pd.show_versions() here

INSTALLED VERSIONS ------------------ commit: None python: 2.7.10.final.0 python-bits: 64 OS: Darwin OS-release: 15.6.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_GB.UTF-8 LOCALE: None.None
pandas: 0.19.1
nose: 1.3.7
pip: 1.5.6
setuptools: 25.1.1
Cython: 0.23.2
numpy: 1.11.1
scipy: 0.14.0
statsmodels: 0.6.1
xarray: None
IPython: 2.3.1
sphinx: None
patsy: 0.4.1
dateutil: 2.5.3
pytz: 2016.6.1
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.4.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.999
httplib2: 0.9
apiclient: 1.1
sqlalchemy: None
pymysql: None
psycopg2: 2.6.1 (dt dec pq3 ext lo64)
jinja2: 2.7.3
boto: 2.26.0
pandas_datareader: None

Metadata

Metadata

Assignees

No one assigned

    Labels

    Dtype ConversionsUnexpected or buggy dtype conversionsMissing-datanp.nan, pd.NaT, pd.NA, dropna, isnull, interpolateTimezonesTimezone data dtype

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions