BUG: `pd.testing.assert_frame_equal` fails if nested `DataFrame` contains `np.nan`

[ChiQiao]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

inner = pd.DataFrame({"a": [1, np.nan]})
foo = pd.DataFrame({"df": [inner]})
bar = pd.DataFrame({"df": [inner]})
pd.testing.assert_frame_equal(foo, bar)

Problem description

Get KeyError: 0
The error goes away if inner does not contain missing values.

Expected Output

None

Output of pd.show_versions()

INSTALLED VERSIONS

commit : c7f7443
python : 3.7.11.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19041
machine : AMD64
processor : Intel64 Family 6 Model 142 Stepping 10, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : None.None

pandas : 1.3.1
numpy : 1.20.3
pytz : 2021.1
dateutil : 2.8.2
pip : 21.2.2
setuptools : 52.0.0.post20210125
Cython : 0.29.24
pytest : 6.1.2
hypothesis : 6.14.1
sphinx : 4.0.2
blosc : None
feather : None
xlsxwriter : 1.4.4
lxml.etree : 4.6.3
html5lib : 1.1
pymysql : 1.0.2
psycopg2 : None
jinja2 : 3.0.1
IPython : 7.22.0
pandas_datareader: None
bs4 : 4.9.3
bottleneck : 1.3.2
fsspec : 2021.07.0
fastparquet : None
gcsfs : None
matplotlib : 3.4.2
numexpr : 2.7.3
odfpy : None
openpyxl : 3.0.7
pandas_gbq : None
pyarrow : 3.0.0
pyxlsb : None
s3fs : 0.4.2
scipy : 1.6.2
sqlalchemy : 1.4.22
tables : 3.6.1
tabulate : 0.8.9
xarray : 0.19.0
xlrd : 2.0.1
xlwt : 1.3.0
numba : 0.53.0

[pratyushsharan]

take

[pratyushsharan]

I've checked and this bug does exist on the master branch. The issue is with pandas.core.dtypes.missing._array_equivalent_object() where the code checkes whether the inner dataframe inside foo, bar are equal:
np.any(np.asarray(left_value != right_value)), where left_value is foo.iloc[:,0]._values[0]
This returns false because of presence of np.nan in inner and then the code in pandas._libs.testing.pyx tries to index the foo.iloc[:,0]._values[0] object, which throws the KeyError.

A simple solution would be to check whether left_value and right_value are of DataFrame type, and then check if the DataFrame objects are equal: left_value.equals(right_value). I've tried this solution on master and it works as expected.

@jbrockmendel - does this solution suffice? And I think it would be good if we include some tests too for this?

[jbrockmendel]

A simple solution would be to check whether left_value and right_value are of DataFrame type,

could we check for is_list_like more generally? a check for DataFrame would entail a pretty big performance hit

[pratyushsharan]

Indeed that's a better solution, adding this solves the issue:

if lib.is_list_like(left_value) and lib.is_list_like(right_value):
    return _array_equivalent_object(left_value, right_value, strict_nan)

Should this require any tests?

[jbrockmendel]

Should this require any tests?

yes. you can make a test out of the example in the OP

[MichaelTiemannOSC]

AN FYI: when I build a Series with uncertainties, I also trip over the fact that the nans in that world cannot be understood by stock pandas. Example:

import pandas as pd
import numpy as np
from uncertainties import ufloat

aa = pd.Series([1, np.nan])
bb = pd.Series([1, np.nan])

# These compare the same, no differences
aa.compare(bb)

xx = pd.Series([ufloat(1, 0), ufloat(np.nan, 0)])
yy = pd.Series([ufloat(1, 0), ufloat(np.nan, 0)])

# These compare different because of the nan+/-0
xx.compare(yy)