API: Interpolate at new values

[rubennj]

First time I used the .interpolate() method I thought that it receives a new index and then interpolates on it, similar to scipy.interpolate.interp1d
From scipy web:

from scipy import interpolate
x = np.arange(0, 10)
y = np.exp(-x/3.0)
f = interpolate.interp1d(x, y)
xnew = np.arange(0,9, 0.1)
ynew = f(xnew)   # use interpolation function returned by `interp1d`

Later I saw the .reindex() method, so I understood that this role is done by .reindex(). However .reindex() is not really doing a powerful interpolation, just extending the current values using the method keyword.

The current way to achieve it (joining previous and new index and then using .reindex()), in version 0.15.0,

index_joined = df.index.join(new_index, how='outer')
df.reindex(index=index_joined).interpolate().reindex(new_index)

A simpler syntax could be accepting the methods from .interpolate() into the 'method' keyword of .reindex():

df.reindex(index=new_index, method='linear')

[TomAugspurger]

See the (long) discussion at #4915

Basically, we wanted to keep the API of interpolate simple, but it's probably too clever since you have to be familiar with reindexing first. I'd actually favor changing the API of interploate to have a new parameter at, which is an array you evaluate the interpolation function at (xnew in your first example).

Also it could take some kind of parameter for whether to return just the interpolated values, or all the values.

[rubennj]

Probably I didn't address well. Actually, I see that to reindex is what I really meant, since Pandas works with objects that already have an index and the real intention of this proposal is to change the index (and consequently to interpolate the corresponding values).

I would say that .interpolate() is well defined as already is (to act at missing datapoints), since .reindex() is present.

Anyhow, to have a simple syntax (on .reindex() or .interpolate()) to get this action is very helpful.

[rubennj]

Any news? @TomAugspurger

[shoyer]

@rubennj I agree, we should have some sort of interpolation method that works like reindex. See also my recent PR to add a 'nearest' method to reindex: #9258

It is a bit awkward from an internals perspective to put this on reindex, because reindex currently does not do any interpolation, but rather only takes existing (possibly repeated) values. This is sometimes advantageous: you can always reindex, even if the data values are non-numeric.

[shoyer]

What about creating two interpolate methods: .interpolate_na() and .interpolate_at()? The former would be an alias for the current interpolate (eventually to be deprecated); the later would work for this new functionality.

I worry that hiding this functionality in reindex means it's unlikely to be easily found -- "interpolate" is a much more obvious name.

[rubennj]

OK, I see that .reindex() shouldn't be touched.

I think that one method, .interpolate(), modulated by parameters looks more compact, in the way @TomAugspurger suggested, but I wonder how compatible would be with the current behaviour.

[shoyer]

We could add a new optional parameter at to interpolate, which if not None will trigger this alternate interpolate API.

My hesitation with combining the functionality into one method is that there are at least two steps required to transform from one mode to the other, e.g., s.interpolate_na() <=> s.dropna().interpolate_at(s.index) (the equivalent to s.interpolate_at(target) is even worse, as shown in the first post). The two methods do pretty fundamentally different things, though both involve interpolation.

Another option would be move the interpolate_na functionality to fillna, and reserve interpolate for the interpolate_at functionality (at least pending deprecation cycles, etc.). That might be a slightly more awkward transition, though.

[jreback]

I think a nice soln here is to create a cookbook recipe for this pattern and a link from the docs

[shoyer]

I think a nice soln here is to create a cookbook recipe for this pattern and a link from the docs

At least in my experience and anecdotally from my colleagues, the current API of interpolate (filling missing values) is unexpected and not what we were looking for. For example, it's pretty different from what scipy and numpy's interpolate functions do. So I think some sort of API change to make this more intuitive is warranted :).

Interpolation at new values is also a very common pattern in my experience. The cookbook recipe will need to be even more complex than @rubennj's example if it is to handle propagating NA correctly, e.g., to ensure that the result of pd.Series([1, 2, np.nan, 4]).interpolate_at([2.5]) is NaN, not 2.5. It's also non-trivial to wrap SciPy directly, because SciPy uses a different meaning for NaN (see scipy/scipy#4086 and the linked issues). So I believe pretty strongly that incorporating some sort of interpolate_at functionality in pandas itself would be a good idea (regardless of what it's called).

[rubennj]

I also thought that .interpolate() was doing that at first.
Interpolating with a new index looks fundamental when you want to compare datasets with different indexes but close related, e.g. several spectra or time-series measured wih different instruments.

I don't understand why overloading interpolate() is a problem, adding the new parameter at. It's not the most elegant solution, but the "two functions" solution doesn't look so ideally clear and it will be a pity to lose the interpolate() method.

To transfer the current function to fill_na() and to use interpolate() for interpolation with a new index is ideal and quite clear to understand in my opinion. However I don't know how bad the transition can be in this case.