Skip to content

BUG: drop_duplicates on categorical not preserving NA #44405

Closed
@phofl

Description

@phofl

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the master branch of pandas.

Reproducible Example

ser = Series(
    Categorical(
        [True, False, True, False, NA], categories=[True, False], ordered=True
    )
)
ser.drop_duplicates()

Issue Description

NA gets converted to NaN

0     True
1    False
4      NaN
dtype: category
Categories (2, object): [True < False]

Expected Behavior

Returning NA

0     True
1    False
4       NA
dtype: category
Categories (2, object): [True < False]

Installed Versions

INSTALLED VERSIONS ------------------ commit : 01b86ed python : 3.8.12.final.0 python-bits : 64 OS : Linux OS-release : 5.11.0-38-generic Version : #42~20.04.1-Ubuntu SMP Tue Sep 28 20:41:07 UTC 2021 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8

pandas : 1.4.0.dev0+1085.g01b86edbbb
numpy : 1.21.3
pytz : 2021.1
dateutil : 2.8.2
pip : 21.2.4
setuptools : 58.0.4
Cython : 0.29.24
pytest : 6.2.5
hypothesis : 6.23.1
sphinx : 4.2.0
blosc : None
feather : None
xlsxwriter : 3.0.1
lxml.etree : 4.6.3
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.0.1
IPython : 7.28.0
pandas_datareader: None
bs4 : 4.10.0
bottleneck : 1.3.2
fsspec : 2021.11.0
fastparquet : 0.7.1
gcsfs : 2021.05.0
matplotlib : 3.4.3
numexpr : 2.7.3
odfpy : None
openpyxl : 3.0.9
pandas_gbq : None
pyarrow : 5.0.0
pyxlsb : None
s3fs : 2021.11.0
scipy : 1.7.2
sqlalchemy : 1.4.25
tables : 3.6.1
tabulate : 0.8.9
xarray : 0.18.0
xlrd : 2.0.1
xlwt : 1.3.0
numba : 0.53.1
None

Process finished with exit code 0

Metadata

Metadata

Assignees

No one assigned

    Labels

    BugCategoricalCategorical Data TypeDuplicate ReportDuplicate issue or pull requestMissing-datanp.nan, pd.NaT, pd.NA, dropna, isnull, interpolateNA - MaskedArraysRelated to pd.NA and nullable extension arrays

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions