API: sum of Series of all NaN should return 0 or NaN ?

[shoyer]

Summary

The question is what the sum of a Series of all NaNs should return (which is equivalent to an empty Series after skipping the NaNs): NaN or 0?

In [1]: s = Series([np.nan])                 

In [2]: s.sum(skipna=True)  # skipping NaNs is the default
Out[2]: nan or 0     <---- DISCUSSION POINT

In [3]: s.sum(skipna=False)
Out[3]: nan

The reason this is a discussion point has the following cause: the internal nansum implementation of pandas returns NaN. But, when bottleneck is installed, pandas will use bottlenecks implementation of nansum, which returns 0 (for the versions >= 1.0).
Bottleneck changed the behaviour from returning NaN to returning 0 to model it after numpy's nansum function.

This has the very annoying consequence that depending on whether bottleneck is installed or not (which is only an optional dependency), you get a different behaviour.

So the decision we need to make, is to either:

• adapt pandas internal implementation to return 0, so in all cases 0 is returned for all NaN/empty series.
• workaround bottlenecks behaviour or not use it for nansum, in order to consistently return NaN instead of 0
• choose one of both above as the default, but have an option to switch behaviour

---

Original title: nansum in bottleneck 1.0 will return 0 for all NaN arrays instead of NaN

xref pydata/bottleneck#96
xref #9421

• Tests are turned off for bottleneck >1.0 (xref TST: test_nanops turns off bottneck for all tests after #10986)

This matches a change from numpy 1.8 -> 1.9.

We should address this for pandas 0.16.

Should we work around the new behavior (probably the simplest choice) or change nansum in pandas?

[jreback]

I guess the np.nan* methods 'ignore' nans by default and thus all-nans is really an empty array. But this seems wrong to me. You can never get a NaN result back.

However pandas wants to propogate NaN results, so this would break existing behavior (which looking at it seems slightly wrong).

In [13]: s = Series([np.nan])                 

In [15]: s.sum(skipna=True)
Out[15]: nan

In [16]: s.sum(skipna=False)
Out[16]: nan

so I think that you are proposing that [15] should be 0.0 (this is the default, FYI).

[shoyer]

Yes, we might want to change your line [15] to 0.0. There is certainly something to be said for staying in sync with numpy/bottleneck.

I think sum and prod are the only functions that would need to change to match the numpy behavior (nanprod will be new in numpy 1.10: numpy/numpy#5418). All the other NA-skipping aggregation functions (mean, std, var, median, etc) are undefined for empty arrays, so it still makes sense for them to return NaN in the all NaN case.

This change also has implications for windowed and grouped sum/prod. These might also need a corresponding change to match.