DEPR: combine_first (replace with update(..., join='outer'); for both Series/DF)

[h-vetinari]

I always found the mechanics of combine_first very unintuitive, and constantly need to look into the docs to see what's happening. I haven't checked the git history, but it seems that the method was a direct response from wesm to a SO question (https://stackoverflow.com/a/9794891). In particular, I think this would be much more intuitive to do with df.update, which is a subset of what #21855 proposes -- it introduces join='outer' for DataFrame.update (currently, only 'left' is supported, but even the source code notes # TODO: Support other joins).

With that new option, df1.combine_first(df2) would be the same as df1.update(df2, join='outer', overwrite=False), only that combine_first has much fewer options and controls (i.e. filter_func and raise_conflict). The only difference is that df.update currently returns None, see #21858.

Since it's quite a well-established function, the deprecation cycle would maybe have to be longer than usual, but I think the update variant is much cleaner, as well as more versatile, than this single-purpose function.

[gfyoung]

cc @jreback

[shoyer]

One downside of DataFrame.update() vs combine_first() is that it modifies the dataframe inplace rather than having a return value.

Just as a point of reference, in xarray we implement combine_first() in terms of an outer join + fillna:

    def combine_first(self, other):
        """Combine two DataArray objects, with union of coordinates.

        This operation follows the normal broadcasting and alignment rules of
        ``join='outer'``.  Default to non-null values of array calling the
        method.  Use np.nan to fill in vacant cells after alignment.

        Parameters
        ----------
        other : DataArray
            Used to fill all matching missing values in this array.

        Returns
        -------
        DataArray
        """
        return ops.fillna(self, other, join="outer")

[h-vetinari]

@shoyer

One downside of DataFrame.update() vs combine_first() is that it modifies the dataframe inplace rather than having a return value.

I agree, which is why I opened #21858 and linked it with the issue here... :)

[jorisvandenbossche]

While looking through #22812, I was also thinking about how combine_first is basically fillna, but without the outer alignment. So basically what @shoyer said :)

Adding a join or align keyword to fillna might also be an option.

[jorisvandenbossche]

I haven't checked the git history, but it seems that the method was a direct response from wesm to a SO question (https://stackoverflow.com/a/9794891)

And for historical fun, combineFirst was already present in the first commit of the git history: c6b236d, so at least a couple of years before the SO question :-)