DataFrame.interpolate() extrapolates over trailing missing data

[grahamjeffries]

See also the discussion at StackOverflow.

Linear interpolation on a series with missing data at the end of the array will overwrite trailing missing values with the last non-missing value. In effect, the function extrapolates rather than strictly interpolating.

Example:

import pandas as pd
import numpy as np

a = pd.Series([np.nan, 1, np.nan, 3, np.nan])
a.interpolate()

Yields (note the extrapolated 4):

0   NaN
1     1
2     2
3     3
4     4
5     4
dtype: float64

not

0   NaN
1     1
2     2
3     3
4     4
5     NaN
dtype: float64

I believe the fix is something along the lines of changing lines 1545:1546 in core/common.py from

result[firstIndex:][invalid] = np.interp(inds[invalid], inds[valid], yvalues[firstIndex:][valid])

to

result[firstIndex:][invalid] = np.interp(inds[invalid], inds[valid], yvalues[firstIndex:][valid], np.nan, np.nan)

[jreback]

pls show a complete reproducible example (e.g. copy-pastable code).

Then if you would like to do a pull-request would be great. These examples serve as the basis for a test, which should fail w/o a fix and pass after.

[TomAugspurger]

Traveling back today. I can take a look this weekend.

I'd like to see what the behavior was before I refactored this stuff.

[jreback]

@TomAugspurger can you circle back on this?

[TomAugspurger]

OK, so this is the same behavior as back in 0.11 before I refactored all the interpolate stuff.

>>> pd.__version__
>>> s = pd.Series([np.nan, 1, np.nan, 3, np.nan])
>>> s
0   NaN
1     1
2   NaN
3     3
4   NaN
dtype: float64
>>> s.interpolate()
0   NaN
1     1
2     2
3     3
4     3
dtype: float64

I'll look into adding an argument to handle the NaNs before and after. The default will have to stay the same for now, I think. Possibly switch to the "correct' default of not extrapolating later on.