API: Add NDFrame property to disallow duplicates

[TomAugspurger]

edit: see #27108 (comment)

---

I'd like to be able to have an index, and ensure that no operation introduces duplicates.

idx = pd.Index(..., allow_duplicates=False)
s = pd.Series(..., index=idx)

From here, any pandas operation that introduces duplicates (e.g. s.loc[['a', 'a']]) would raise, rather than return an Index with two values.

[jorisvandenbossche]

Discussion we had about this: should this "property" live on the Index, or on the DataFrame/Series object?

[TomAugspurger]

Yeah, my example kinda exposes that difficulty, as the new index in s.loc[['a', 'a']] isn't the original index. So maybe it would be better on the dataframe?

[TomAugspurger]

I played with this a bit today.

For a user-API, I think it makes the most since to have an allow_duplicate_labels keyword that can be passed to Series / DataFrame. IOW,

>>> pd.Series(data, index, allow_duplicate_labels=False)

rather than

>>> pd.Series(data, pd.Index(..., allow_duplicate_labels=False))

While the duplicate detection is done in the Index, it seems more ergonomic to have it on NDFrame.

A potential downside is that you can't disallow duplicates in the columns while allowing duplicates in the index. If we really wanted that, we can support setting allow_duplicate_labels=True/False/rows/columns. But let's leave that aside for now.

---

For an implementation, it seems like _metadata offers a non-invasive solution. Without having to touch rename, the following is possible

In [1]: import pandas as pd

In [2]: df = pd.DataFrame(index=['a', 'A'], allow_duplicate_labels=False)

In [3]: df
Out[3]:
Empty DataFrame
Columns: []
Index: [a, A]

In [4]: df.rename(str.upper)
---------------------------------------------------------------------------
DuplicateLabelError                       Traceback (most recent call last)
<ipython-input-4-17c8fb0b7c7f> in <module>
----> 1 df.rename(str.upper)

~/sandbox/pandas/pandas/util/_decorators.py in wrapper(*args, **kwargs)
    233         @wraps(func)
    234         def wrapper(*args, **kwargs) -> Callable[..., Any]:
--> 235             return func(*args, **kwargs)
    236
    237         kind = inspect.Parameter.POSITIONAL_OR_KEYWORD

...

~/sandbox/pandas/pandas/core/indexes/base.py in _maybe_check_unique(self)
    562             # TODO: position, value, not too large.
    563             msg = "Index has duplicates."
--> 564             raise DuplicateLabelError(msg)
    565
    566     # --------------------------------------------------------------------

DuplicateLabelError: Index has duplicates.

Likewise for concat, __getitem__, .loc, etc. These all call __finalize__. Since I think we want propagation by default, it does exactly what we want.

The changes are at https://github.com/pandas-dev/pandas/compare/master...TomAugspurger:unique-index?expand=1, but all I've needed so far is

1. Adding allow_duplicate_labels to the NDFrame constructors
2. Adding a getter / setter NDFrame.allows_duplicate_labels
3. Adding allows_duplicate_labels to the NDFrame._metadata

[TomAugspurger]

This was closed by #28394.