BUG: None is not equal to None

[mapio]

If you have None in a series, it will not be considered equal to None (even in the same series).

A = pd.Series([None]*3)
A == A

gives

0    False
1    False
2    False
dtype: bool

but, of course

A.values == A.values

gives

array([ True,  True,  True])

I can't find any reference to this behaviour in the documentation and looks quite unnatural.

Output of pd.show_versions()

Here the output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.6.4.final.0 python-bits: 64 OS: Darwin OS-release: 16.7.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: en_US.UTF-8 LANG: en_US.UTF-8 LOCALE: en_US.UTF-8

pandas: 0.22.0
pytest: None
pip: 9.0.1
setuptools: 38.5.2
Cython: None
numpy: 1.14.2
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: None
patsy: None
dateutil: 2.7.0
pytz: 2018.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.2.0
openpyxl: None
xlrd: 1.1.0
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[jschendel]

Also a bit strange that == is inconsistent with the eq method in this case:

In [2]: pd.__version__
Out[2]: '0.23.0.dev0+658.g17c1fad'

In [3]: s = pd.Series([None]*3)

In [4]: s
Out[4]:
0    None
1    None
2    None
dtype: object

In [5]: s == s
Out[5]:
0    False
1    False
2    False
dtype: bool

In [6]: s.eq(s)
Out[6]:
0    True
1    True
2    True
dtype: bool

[mroeschke]

I believe this occurs because this comparison hits this block:

pandas/pandas/_libs/ops.pyx

Lines 162 to 163 in 17c1fad

	if checknull(x) or checknull(y):
	result[i] = False

And the checknull function resolves down this path which considers None as null:

pandas/pandas/_libs/src/util.pxd

Lines 156 to 158 in 17c1fad

	cdef inline bint _checknull(object val):
	try:
	return val is None or (cpython.PyFloat_Check(val) and val != val)

[jreback]

see the warnings box: http://pandas.pydata.org/pandas-docs/stable/missing_data.html

This was done quite a while ago to make the behavior of nulls consistent, in that they don't compare equal. This puts None and np.nan on an equal (though not-consistent with python, BUT consistent with numpy) footing.

So this is not a bug, rather a consequence of stradling 2 conventions.

I suppose the documentation could be slightly enhanced.

[jreback]

however this should be consistent with this:In [7]: s = pd.Series([np.nan]*3)

In [8]: s == s
Out[8]: 
0    False
1    False
2    False
dtype: bool

In [9]: s.eq(s)
Out[9]: 
0    False
1    False
2    False
dtype: bool

so [6] form @jschendel should match [9] here. The incorrect case is actually .eq

[jreback]

here's the dichotomy in scalars

In [11]: None == None
Out[11]: True

In [12]: np.nan == np.nan
Out[12]: False

[mapio]

I do not agree with @jreback and the current status of equality for None in pandas.

I've seen the warning box and I completely agree that, according to IEEE floating-point standard, "a NaN is never equal to any other number (including another NaN)" (cited from What Every Computer Scientist Should Know About Floating-Point Arithmetic).

I've no problems, if a data type of a series is numeric, to accept that NaNs are not equal and even that None get converted to NaNs.

But it a series contains objects (let them be string, list, or whatever), the equality should be consistent with Python equality for such objects.

It's a basic application of the principle of least astonishment. I don't see any benefit in mangling None if the data type is non numeric.

[jreback]

@mapio you are missing the point. None is NOT the default missing indicator for anything. np.nan IS. None is accepted for compatibility.

[mapio]

Where in the documentation is stated that None is the "missing indicator"? And, if this is the case, I find incredibly confusing that

>>> pd.Series([1,None,2])
0    1.0
1    NaN
2    2.0
dtype: float64

but

>>> pd.Series(['a', None, 'b'])
0       a
1    None
2       b
dtype: object

If you pick na.nan as a missing data indicator, than you must convert every None in a series to it, not only in numeric series, but also in series of objects. To be clear, I expect the second example to look

>>> pd.Series(['a', None, 'b'])
0       a
1    NaN
2       b
dtype: object

Given such a convention I would have probably still asked why you decide to translate a None to a NaN, but I would have not been surprised at all to have a False comparing the second entry of the two series.

Probably I'm missing the point, but the current situation is, at best, inconsistent (and as such, is very confusing).

[TomAugspurger]

http://pandas-docs.github.io/pandas-docs-travis/missing_data.html#values-considered-missing

@mapio how familiar are you with NumPy / pandas type system? When you do Series([1, None, 2]), we infer it as floats. If you want python objects, specify that.

In [5]: s = pd.Series([1, None, 2], dtype=object)

In [6]: s
Out[6]:
0       1
1    None
2       2
dtype: object

But as you've seen, many places in pandas, especially in ops like ==, treat None as missing.