API: how strict should the equals() method be?

[jorisvandenbossche]

While adding an equals() method to ExtensionArray (#27081, #30652), some questions come up about how strict the method should be:

1. Other array-likes are not equivalent, even if they are all equal?
2. But subclasses are equivalent, when they are all equal?
3. Objects with different dtype are not equivalent (eg int8 vs int16), even if all values are equal?

And it seems that right now, we are somewhat inconsistent with this in pandas regarding being strict on the data type.

Series is strict about the dtype, while Index is not:

>>> pd.Series([1, 2, 3], dtype="int64").equals(pd.Series([1, 2, 3], dtype="int32"))
False

>>> pd.Index([1, 2, 3], dtype="int64").equals(pd.Index([1, 2, 3], dtype="int32"))
True

For Index, this not only gives True for different integer dtypes as above, but also for float/int, object/int (both examples give False for Series):

>>> pd.Index([1, 2, 3], dtype="int64").equals(pd.Index([1, 2, 3], dtype="object"))
True

>>> pd.Index([1, 2, 3], dtype="int64").equals(pd.Index([1, 2, 3], dtype="float64"))
True

Index and Series are consistent when it comes to not being equal with other array-likes:

# all those cases return False
pd.Series([1, 2, 3]).equals(np.array([1, 2, 3])) 
pd.Index([1, 2, 3]).equals(np.array([1, 2, 3])) 
pd.Series([1, 2, 3]).equals(pd.Index([1, 2, 3])) 
pd.Index([1, 2, 3]).equals(pd.Series([1, 2, 3]))
pd.Series([1, 2, 3]).equals([1, 2, 3])
pd.Index([1, 2, 3]).equals([1, 2, 3])

Both Index and Series also seem to allow subclasses:

class MySeries(pd.Series): 
    pass 

>>> pd.Series([1, 2, 3]).equals(MySeries([1, 2, 3]))
True

So in the end, I think the main discussion point is: should the dtype be exactly the same, or should only the values be equal?

For DataFrame, it shares the implementation with Series so follows that behaviour (except that for DataFrame there are some additional rules about how column names need to compare equal).

[dsaxton]

It feels to me like when calling equals on some container you'd generally be interested in equality of the entire container not just the values, otherwise you'd use == (maybe modulo some differences around missing values). You might argue that even this behavior is questionable:

[ins] In [11]: pd.Series([1, 2, 3], name="abc").equals(pd.Series([1, 2, 3], name="xyz"))
Out[11]: True

[jorisvandenbossche]

If you want equality of the entire container, which I assume you typically do in testing (?), you also have the assert_.. functions to use.

On the other hand, in non-testing code, you might be more be interested in equal values (this is just guessing, I don't really know how people would like to use this).
For example, if you want to check if two columns contain the same elements (df['a'].equals(df['b'])), it might be annoying to have to rename first before being able to call equals.

otherwise you'd use == (maybe modulo some differences around missing values).

You could say that equals mainly exists to handle this "modulo" clause .. ? (but again, hard to say how it is exactly used)

[jorisvandenbossche]

Regarding use cases, it might already be interesting to check how this method is used internally (as the reason I started looking at this is because I actually needed EA.equals to implement Index.equals when storing EAs in the Index).

[attack68]

Just to add to this from #36065 the testing mod equals also behave inconsistently:

import pandas as pd
import pandas._testing as tm

i1 = pd.date_range("2008-01-01", periods=1000, freq="12H")
i2 = pd.date_range("2008-01-01", periods=1000, freq="12H")
i2.freq = None

i1.equals(i2)  # True
tm.assert_index_equal(i1, i2)  # True

df = pd.DataFrame({'dates': i1})
df2 = pd.DataFrame({'dates': i2})

df.equals(df2)  # True
tm.assert_frame_equal(df, df2)  # True

df.index = i1
df2.index = i2

df.equals(df2)  #True
tm.assert_frame_equal(df, df2)  # FAILS lidx.freq != ridx.freq

The reason being that for assert_series_equals after checking the index it checks the frequency of the index. A very easy change would be to add the freq check to the assert_index_equals with argument and remove it from the assert_series_equals.

[YarShev]

@attack68 , it's a good point!

[jbrockmendel]

I think this is at the heart of #33531