DOC: pd.Series.nlargest(keep='all') does not ignore duplicates when counting the largest values

[asishm]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the master branch of pandas.

Reproducible Example

In [1]: import pandas as pd

In [3]: s = pd.Series([1,2,3,3,4])
   ...: s.nlargest(2, keep='all')
Out[3]:
4    4
2    3
3    3
dtype: int64

In [4]: s = pd.Series([1,2,3,3,4])
   ...: s.nlargest(3, keep='all')
Out[4]:
4    4
2    3
3    3
dtype: int64

In [5]: pd.__version__
Out[5]: '1.4.0.dev0+1453.ga3c0e7bcfb

Issue Description

nlargest = 2 and nlargest = 3 gives the same output here with keep='all'

Expected Behavior

I'd expect s.nlargest(3, keep='all') to return

4    4
2    3
3    3
1    2
dtype: int64

Installed Versions

INSTALLED VERSIONS ------------------ commit : a3c0e7b python : 3.8.12.final.0 python-bits : 64 OS : Linux OS-release : 4.19.128-microsoft-standard Version : #1 SMP Tue Jun 23 12:58:10 UTC 2020 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : C.UTF-8 LOCALE : en_US.UTF-8

pandas : 1.4.0.dev0+1453.ga3c0e7bcfb
numpy : 1.21.2
pytz : 2021.3
dateutil : 2.8.2
pip : 21.3
setuptools : 58.2.0
Cython : 0.29.24
pytest : 6.2.5
hypothesis : 6.23.2
sphinx : 4.2.0
blosc : None
feather : None
xlsxwriter : 3.0.1
lxml.etree : 4.6.3
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.0.2
IPython : 7.28.0
pandas_datareader: None
bs4 : 4.10.0
bottleneck : 1.3.2
fsspec : 2021.10.0
fastparquet : 0.7.1
gcsfs : 2021.10.0
matplotlib : 3.4.3
numexpr : 2.7.3
odfpy : None
openpyxl : 3.0.9
pandas_gbq : None
pyarrow : 5.0.0
pyxlsb : None
s3fs : 2021.10.0
scipy : 1.7.1
sqlalchemy : 1.4.25
tables : 3.6.1
tabulate : 0.8.9
xarray : 0.18.2
xlrd : 2.0.1
xlwt : 1.3.0
numba : 0.53.1

[phofl]

The docs clearly state retun the First n rows, so this looks as expected to me

[vovavili]

The docs clearly state retun the First n rows, so this looks as expected to me

Not true, documentation states that duplicates should be retained in case of keep='all', and yet in the OP's examples they are not retained:

I ran into this issue independently on my machine as well.

[vovavili]

@phofl In my case, I ran into this issue while trying to solve a Leetcode problem:

import pandas as pd

data = [
    [1, "Joe", 85000, 1],
    [2, "Henry", 80000, 2],
    [3, "Sam", 60000, 2],
    [4, "Max", 90000, 1],
    [5, "Janet", 69000, 1],
    [6, "Randy", 85000, 1],
    [7, "Will", 70000, 1],
]
employee = pd.DataFrame(data, columns=["id", "name", "salary", "departmentId"]).astype(
    {"id": "Int64", "name": "object", "salary": "Int64", "departmentId": "Int64"}
)
data = [[1, "IT"], [2, "Sales"]]
department = pd.DataFrame(data, columns=["id", "name"]).astype(
    {"id": "Int64", "name": "object"}
)

(
    employee.merge(
        department.rename(columns={"name": "Department", "id": "departmentId"}),
        how="left",
    )
    .groupby("Department")["salary"]
    .nlargest(3, keep="all")
    .pipe(print)
)

Output, where I expected more than three rows under the "IT" index:

Department   
IT          3    90000
            0    85000
            5    85000
Sales       1    80000
            2    60000
Name: salary, dtype: Int64

So, either the documentation must be updated to avoid confusion, or the method must be reworked to handle duplicates like these. Because judging from the documentation, the kind of behavior I expected was the following:

Input:

import pandas as pd

def nlargest_with_duplicates(s):
    largest_vals = s.drop_duplicates().nlargest(3)
    return s[s.isin(largest_vals)].sort_values(ascending=False)

(
    employee.merge(
    department.rename(columns={"name": "Department", "id": "departmentId"}), how="left"
    )
    .groupby("Department")["salary"]
    .apply(nlargest_with_duplicates)
    .pipe(print)
)

Output:

Department   
IT          3    90000
            0    85000
            5    85000
            6    70000
Sales       1    80000
            2    60000
Name: salary, dtype: Int64

[paulreece]

By request of @vovavili I have reopened this issue. I can confirm it still occurs on the main development branch:

>>> s = pd.Series([1,2,3,3,4])
>>> s.nlargest(2, keep='all')
4    4
2    3
3    3
dtype: int64
>>> s = pd.Series([1,2,3,3,4])
>>> s.nlargest(3, keep='all')
4    4
2    3
3    3
dtype: int64

[rhshadrach]

The documentation on keep states:

When there are duplicate values that cannot all fit in a Series of n elements:

The argument keep is only used in this scenario. keep does not mean "ignore duplicates when counting".

The behavior appears correct to me, and I think the documentation is correct as stated, but perhaps this subtle difference can be better emphasized. If there is a desire to change the behavior of this method, then a new issue should be opened as an enhancement rather than a bug report.

[paulreece]

To further elaborate what Richard said here's the Series from the example:

s = pd.Series(countries_population)
s
Italy       59000000
France      65000000
Malta         434000
Maldives      434000
Brunei        434000
Iceland       337000
Nauru          11300
Tuvalu         11300
Anguilla       11300
Montserrat      5200

So the example you cited from the Docs earlier @vovavili is only returning 5 elements because one of the duplicates happens to be on of the top n(3) largest elements.

That's why we get the same return values for n=2 and n=3 in the OP's post.