BUG: strange timeseries plot behavior

[rosnfeld]

After some discussion below, here's a simple repro case:

s1 = pd.Series([1, 2, 3], index=[datetime.datetime(1995, 12, 31), datetime.datetime(2000, 12, 31), datetime.datetime(2005, 12, 31)])
s2 = pd.Series([1, 2, 3], index=[datetime.datetime(1997, 12, 31), datetime.datetime(2003, 12, 31), datetime.datetime(2008, 12, 31)])

# plot first series, then add the second series to those axes, then try adding the first series again
ax = s1.plot()
s2.plot(ax=ax)
s1.plot(ax=ax)

causes

Traceback (most recent call last):
  File "simple_repro.py", line 10, in <module>
    s1.plot(ax=ax)
  File "/home/andrew/git/pandas-rosnfeld/pandas/tools/plotting.py", line 2116, in plot_series
    plot_obj.generate()
  File "/home/andrew/git/pandas-rosnfeld/pandas/tools/plotting.py", line 920, in generate
    self._make_plot()
  File "/home/andrew/git/pandas-rosnfeld/pandas/tools/plotting.py", line 1482, in _make_plot
    self._make_ts_plot(data)
  File "/home/andrew/git/pandas-rosnfeld/pandas/tools/plotting.py", line 1577, in _make_ts_plot
    _plot(data, 0, ax, label, self.style, **kwds)
  File "/home/andrew/git/pandas-rosnfeld/pandas/tools/plotting.py", line 1553, in _plot
    style=style, **kwds)
  File "/home/andrew/git/pandas-rosnfeld/pandas/tseries/plotting.py", line 82, in tsplot
    left, right = _get_xlim(ax.get_lines())
  File "/home/andrew/git/pandas-rosnfeld/pandas/tseries/plotting.py", line 226, in _get_xlim
    left = min(x[0].ordinal, left)
AttributeError: 'datetime.datetime' object has no attribute 'ordinal'

-- ORIGINAL MESSAGE --

Here's a small dataset:

date,region,value
1996-12-31,BRA,4.5
2003-12-31,BRA,3.7
2007-12-31,BRA,2.2
1995-12-31,COL,6.3
2000-12-31,COL,4.9
2005-12-31,COL,5.1
2010-12-31,COL,3.4
1997-12-31,PAN,6.3
2003-12-31,PAN,5.1
2008-12-31,PAN,3.9
1990-12-31,VEN,6.7
1991-12-31,VEN,5.4
1992-12-31,VEN,4.5
1993-12-31,VEN,4
1994-12-31,VEN,3.9
1995-12-31,VEN,4.1
1996-12-31,VEN,4.4
1997-12-31,VEN,4.5
1998-12-31,VEN,4.6
1999-12-31,VEN,4.1
2000-12-31,VEN,3.9
2007-12-31,VEN,3.7

If I read this in using

data = pd.read_csv('./data.csv', parse_dates='date', index_col='date')

and then try and plot it using

data.groupby('region').value.plot(legend=True)

I get more or less what I expect (perhaps the xlim doesn't go up to 2010-12-31, but otherwise fine).

If I delete out the BRA rows and try this again, I get:

Traceback (most recent call last):
  File "repro.py", line 6, in <module>
    data.groupby('region').value.plot()
  File "/home/andrew/git/pandas-rosnfeld/pandas/core/groupby.py", line 342, in wrapper
    return self.apply(curried)
  File "/home/andrew/git/pandas-rosnfeld/pandas/core/groupby.py", line 428, in apply
    return self._python_apply_general(f)
  File "/home/andrew/git/pandas-rosnfeld/pandas/core/groupby.py", line 432, in _python_apply_general
    self.axis)
  File "/home/andrew/git/pandas-rosnfeld/pandas/core/groupby.py", line 958, in apply
    res = f(group)
  File "/home/andrew/git/pandas-rosnfeld/pandas/core/groupby.py", line 426, in f
    return func(g, *args, **kwargs)
  File "/home/andrew/git/pandas-rosnfeld/pandas/core/groupby.py", line 333, in curried
    return f(x, *args, **kwargs)
  File "/home/andrew/git/pandas-rosnfeld/pandas/tools/plotting.py", line 1921, in plot_series
    plot_obj.generate()
  File "/home/andrew/git/pandas-rosnfeld/pandas/tools/plotting.py", line 912, in generate
    self._make_plot()
  File "/home/andrew/git/pandas-rosnfeld/pandas/tools/plotting.py", line 1379, in _make_plot
    self._make_ts_plot(data, **self.kwds)
  File "/home/andrew/git/pandas-rosnfeld/pandas/tools/plotting.py", line 1450, in _make_ts_plot
    _plot(data, 0, ax, label, self.style, **kwds)
  File "/home/andrew/git/pandas-rosnfeld/pandas/tools/plotting.py", line 1434, in _plot
    style=style, **kwds)
  File "/home/andrew/git/pandas-rosnfeld/pandas/tseries/plotting.py", line 82, in tsplot
    left, right = _get_xlim(ax.get_lines())
  File "/home/andrew/git/pandas-rosnfeld/pandas/tseries/plotting.py", line 226, in _get_xlim
    left = min(x[0].ordinal, left)
AttributeError: 'datetime.datetime' object has no attribute 'ordinal'

If I delete out both BRA and VEN rows, then there is no exception raised but I only see one series plotted and the x-axis is not formatted as a date.

One could also approach this whole exercise via something like

data = pd.read_csv('./data.csv', parse_dates='date')
data.pivot('date', 'region', 'value').plot()

but this works even worse, I just get a truncated VEN series and nothing else.

This is with current master pandas (but also happens in 0.13.1) and matplotlib 1.3.1.

Are there known issues with plotting sparse-yet-overlapping timeseries?

[rosnfeld]

I guess if I do

data.pivot('date', 'region', 'value').interpolate().plot()

or

pivoted = data.pivot('date', 'region', 'value')
pivoted.index = pd.to_datetime(pivoted.index)
pivoted.interpolate(method='time').plot()

I get something like what I wanted as it cleans up the missing values. Interpolate's a new feature I hadn't seen before. (cool!)

Maybe this is all user-error but I had been doing groupby plotting like this for a while, and had been getting what looked like correct results. I feel there may actually be a bug here someplace, that groupby behavior seems so bizarre.

[rosnfeld]

Actually, interpolation is not really what I want, as it makes it seem as if there is more data than is actually present. I basically just want to see the various region series all plotted as they would be if they were plotted individually, except all together on the same axes.

(and a loop like

figure = plt.figure()
ax = figure.gca()
data = pd.read_csv('./data.csv', parse_dates='date')
for region in data.region.unique():
    subset = data[data.region == region]
    subset = subset.set_index('date')
    subset.value.plot(ax=ax, label=region)

seems to just over-write the axes)

[jorisvandenbossche]

I think the groupby/plot issue seems certainly like a bug. I can't fully lay my hand on it, but I think it has something to do with combining regular/irregular timeseries.

The issue with the xlim not respecting the data is because it is updated by the last group (while this is a smaller group), and this is a seperate issue I think (you can open another issue for that).

The reason you get almost no points on the plot with pivot you seem to already figured out, this is indeed because of all the NaN values. You can also deal with this by plotting points instead of lines.

[rosnfeld]

Thanks @jorisvandenbossche, it looks like the "separate issue" is already filed as #2960 . So this one is just the groupby/plot weirdness.

Did you mean to add to your comment? (it ends in what looks like the start to some code)

[jorisvandenbossche]

@rosnfeld ah yes, I first wanted to add a code snippet how to do your last example easier, but this also had the same bug, but forgot to remove it. Removed it now

[jreback]

@rosnfeld @jorisvandenbossche

so this is the exception that's in the top section?

what is causing this?

[rosnfeld]

Yeah, the exception is the most alarming thing, though changing the data slightly causes some other incorrect behavior (missing/incorrect plotting, which is harder to spot/diagnose).

I don't know what's causing it without further investigation, but I can try to investigate and hopefully submit a fix. (it will be my first time digging into the plotting code, not sure how involved it is)

[TomAugspurger]

The error is occurring in

def _get_xlim(lines):
    left, right = np.inf, -np.inf
    for l in lines:
        x = l.get_xdata()
        left = min(x[0].ordinal, left)
        right = max(x[-1].ordinal, right)
    return left, right

The line left = min(to_ordinal(x[0]), left) apparently expects a PeriodIndex.
For whatever reason, when you select sub = data[data.region != 'BRA'] and plot that, you get an array of datetime.datetime objects at that point, instead of a PeriodIndex.

I'm not too familiar with our Datetime code, but does anyone know why these aren't the same?

In [40]: import datetime
In [41]: b = datetime.datetime(1997, 12, 31, 0, 0)

In [42]: y = pd.Period(year=1997, month=12, day=31, hour=0, minute=0, freq='S')

In [43]: b.toordinal()
Out[43]: 729389

In [44]: y.ordinal
Out[44]: 883526400

[jorisvandenbossche]

Some time ago I looked into this (for another issue), and then one of the fundamental problems was the design of pandas plotting for timeseries splitted in two ways: with datetimes when having irregular serieses, and with ordinals when having a regular timeseries (which was then converted to periodindex), and that those two types are incompatible with each other (so when you combine both types in a certain way it gives problems).

But I have to dig it up again to fully remember (I have some overview of the problem somewhere, but never finished it). I don't know i I find some time in the short term, but will try.

[jorisvandenbossche]

And the datetime.datetime.toordinal is in days, the pandas.Period.ordinal int he frequency you specified (in this case seconds).
Plus datetime.datetime.toordinal is since 01/01/0001, pandas base is 1970

[rosnfeld]

Yes, I looked at this a little last night, and agree with what you're saying - when you whittle the dataset down to just COL/PAN/VEN rows and then plot, COL and VEN get converted to have PeriodIndexes, but PAN stays with a DatetimeIndex for some reason, and then plotting them all on the same axes (via groupby) blows up somehow.