DOC: Add example of drop_duplicates dropping a first-level

[smarie]

Feature Type

• Adding new functionality to pandas

• Changing existing functionality in pandas

• Removing existing functionality in pandas

Problem Description

There is currently no top-level API to easily drop records with duplicate indices in a Series or a Dataframe. Series.drop_duplicates does not work on indices.

Many workarounds have been discussed on stackoverflow, together with perf comparisons:

• https://stackoverflow.com/questions/13035764/remove-pandas-rows-with-duplicate-indices
• https://stackoverflow.com/questions/52105181/consider-duplicate-index-in-drop-duplicates-method-of-a-pandas-dataframe
• https://stackoverflow.com/questions/70429614/pandas-dataframes-remove-duplicate-index-keep-largest-value-first-depending-on

None of them feels "right", in a sense that it is not very readable, while this is a fairly standard operation that would be quite consistent with the drop_duplicates method (especially for Series!)

Feature Description

It would be convenient in my opinion to

• add a new method drop_duplicate_indices(keep='first') that would basically perform df[~df.index.duplicated(keep=keep)] (from this answer)
• document this method so as to explain that users wishing to be "more intelligent" about the records (for example keeping the record with max, min, etc.) to keep should consider using groupby or other kind of aggregations. OR, enhance both drop_duplicates and drop_duplicate_indices with something as suggested in ENH: keep='random' option for .duplicated and .drop_duplicates #25838 (maybe a separate improvement)

Alternative Solutions

See above in the stackoverflow posts

Additional Context

No response

[jreback]

-1 on new api

documentation is ok

eg put the example right in drop_duplicates is good

[smarie]

Ok. The drawback of "documentation only" is still the lack of readability of the solution (even if documented).

An alternate solution that I did not mention would be to modify drop_duplicates to add an argument.

Current:

• DataFrame.drop_duplicates(subset=None, keep='first', inplace=False, ignore_index=False)
• Series.drop_duplicates(keep='first', inplace=False)`¶

Possible API mod (without new method):

• Series.drop_duplicates: add a parameter index=True/False (default False) and a parameter values=True/False (default True). If both are False a ValueError is raised
• DataFrame.drop_duplicates: add a parameter index=True/False (default False).

That way dropping duplicate indices could be done with the following code :

# Drop records with duplicate index - For series
s = s.drop_duplicates(index=True, values=False)

# Drop records with duplicate index - For dataframe
df = df.drop_duplicates(index=True, subset=())

this is more readable than df[~df.index.duplicated(keep=keep)] don't you think ? Of course doc needs to be updated accordingly