BUG: min on non-numeric data with nans

[hayd]

related #4279
related #5967

Min doesn't seem to work as expected with NaNs and non-numeric data:

In [1]: s = pd.Series(['alpha', np.nan, 'charlie', 'delta'])

In [2]: s
Out[2]:
0      alpha
1        NaN
2    charlie
3      delta
dtype: object

In [3]: s.min()  # by default docstring suggests this should skipnan
Out[3]: inf

The hack/workaround is to exclude them, perhaps we should special case this in the code:

In [4]: s[s.notnull()].min()
Out[4]: 'alpha'

From discussion on this pandas SO question. and I also posted a corresponding numpy issue:
numpy/numpy#3508.

[hayd]

@jreback Think we should fix this on the pandas side, can you think of a cheaper/efficient way to do this.

Sticking this at the top of min/max/more?

if s.dtype == object and np.nan in s.values:
   s = s[s.notnull()]

[jreback]

from core/nanops.py
easy change

@bottleneck_switch()
def nanmin(values, axis=None, skipna=True):
    values, mask, dtype = _get_values(values, skipna, fill_value_typ = '+inf')

    # numpy 1.6.1 workaround in Python 3.x
    if values.dtype == np.object)
_
            if sys.version_info[0] >= 3):  # pragma: no cover
                import __builtin__
                if values.ndim > 1:
                   apply_ax = axis if axis is not None else 0
                   result = np.apply_along_axis(__builtin__.min, apply_ax, values)
                else:
                   result = __builtin__.min(values)

             ##### add something like this #####
             else:
                 result = values[mask].min()
    else:
        if ((axis is not None and values.shape[axis] == 0)
                or values.size == 0):
            result = com.ensure_float(values.sum(axis))
            result.fill(np.nan)
        else:
            result = values.min(axis)

    result = _wrap_results(result,dtype)
    return _maybe_null_out(result, axis, mask)

[jreback]

also can prob also do this if regardless of skipna (I don't think that makes sense for object type anyhow)

[hayd]

I think there might just be a typo in nanops._get_values (a isnull rather than a notnull).

[hayd]

maybe not

[hayd]

@jreback fancy throwing two cents to the numpy issue/discussion?

Will have another go making the "easy change" this evening :). Bit confusing as _get_values seems to return the opposite mask then you'd expect (i.e. the null values are True).

[jreback]

yeh that was how it was as far as i can remember

prob should change it as its only internal to this module

[miketkelly]

Related to #4006.

[jreback]

why do u think these are related?

[miketkelly]

In that a solution to this is likely to solve the other. They only differ in how null is represented in the series.

[jreback]

your solution to the other issue looks ok
but this is different, and a numpy bug that needs working around

[hayd]

Will have another go at this at the weekend. (Maybe I don't understand your "easy change" @jreback )

[jreback]

hah!

look at core/nanops