BUG: duplicated() returns different results on consecutive runs!

[emsi]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

# Unfortunately I have to provide a downloadable dataframe example (51MB).
# I guess either the size or encoding of the data contributes to the problem.
# Dataset can be downloaded like this:
# !wget https://github.com/emsi/artifacts/raw/master/sources.jl

import joblib as jl

df = jl.load("sources.jl")

# multiple calls to duplicate() yield different results!
pd.Series([len(df[df.duplicated()]) for _ in range(500)]).describe()

Issue Description

When I call duplicated() on a given dataframe it returns different results like it has some stochasticity built in. The number of found duplicates differs. Sometimes it's 5507 sometimes its 5509. I even replicated this on another computer as I suspected some CPU/RAM problem.

>>> pd.Series([len(df[df.duplicated()]) for _ in range(500)]).describe()

count     500.000000
mean     5507.884000
std         0.957276
min      5507.000000
25%      5507.000000
50%      5507.000000
75%      5509.000000
max      5509.000000
dtype: float64

Expected Behavior

I'd expect duplicated() to produce reproducible results. :)

Installed Versions

INSTALLED VERSIONS

commit : 4bfe3d0
python : 3.9.0.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.0-100-generic
Version : #113-Ubuntu SMP Thu Feb 3 18:43:29 UTC 2022
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.4.2
numpy : 1.19.2
pytz : 2021.3
dateutil : 2.8.2
pip : 20.2.3
setuptools : 50.3.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.7.1
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.0.3
IPython : 8.0.1
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : None
fsspec : 2022.01.0
gcsfs : None
markupsafe : 2.0.1
matplotlib : 3.5.1
numba : None
numexpr : 2.8.1
odfpy : None
openpyxl : 3.0.9
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.7.3
snappy : None
sqlalchemy : None
tables : 3.7.0
tabulate : None
xarray : None
xlrd : 2.0.1
xlwt : None
zstandard : None

[rhshadrach]

Can you provide a reproducible example demonstrating the issue.

[emsi]

I have narrowed it down a bit. You can replicate it with:

import io
import pandas as pd

df_txt=',source,rule.id\n458093,"\tpublic static void main(String[] args) {\n\t\tlong t = System.currentTimeMillis();\n\t\ttry {\n\t\t\n\t\t\tlogger.info(""distributed transaction manager  init....."");\n\t\t\tSystem.setProperty(""dubbo.application.logger"", ""slf4j"");\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t//\udcc0\udcad\udcc6\udcf0spring\udcb7\udcfe\udcce\udcf1\udcd6\udcf7\udcc5\udce4\udcd6\udcc3\udcceļ\udcfe\udcba\udccddb-service module\udcd6е\udcc4\udcd2\udcd4spring-\udcbf\udcaaͷ\udcb5\udcc4mybatis\udcc5\udce4\udcd6\udcc3\udcceļ\udcfe:spring-dbservice-mybatis.xml\n\t\t\tfinal ClassPathXmlApplicationContext context = new ClassPathXmlApplicationContext(\n\t\t\t\t\t""classpath*:distributed-transaction-service.xml"");\n\t\t\t\n\t\t\t\n\t\t\tlogger.info(""user.dir: "" + System.getProperty(""user.dir""));\n\t\t\t\n\t\t\tif (""true"".equals(System.getProperty(SHUTDOWN_HOOK_KEY))) {\n\t\t\t\tRuntime.getRuntime().addShutdownHook(new Thread() {\n\t\t\t\t\tpublic void run() {\n\t\t\t\t\t\ttry {\n\t\t\t\t\t\t\tcontext.stop();\n\t\t\t\t\t\t\tlogger.warn(""server stopped!!!"");\n\t\t\t\t\t\t} catch (Exception e) {\n\t\t\t\t\t\t\tlogger.error(e.getMessage(), e);\n\t\t\t\t\t\t} catch (Throwable t) {\n\t\t\t\t\t\t\tlogger.error(t.getMessage());\n\t\t\t\t\t\t}\n\t\t\t\t\t\tsynchronized (TransactionServiceLauncher.class) {\n\t\t\t\t\t\t\trunning = false;\n\t\t\t\t\t\t\tTransactionServiceLauncher.class.notify();\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t});\n\t\t\t}\n\t\t\t\n\t\t\t//\udcc0\udcad\udcc6\udcf0\udcb7\udcfe\udcce\udcf1\n\t\t\tcontext.start();\n\t\t\tlogger.info(""distributed transaction manager Started!  take "" + (System.currentTimeMillis() - t) + "" ms"");\n\t\t} catch (RuntimeException e) {\n\t\t\tlogger.error(e.getMessage(), e);\n\t\t\tSystem.exit(1);\n\t\t}\n\t\tsynchronized (TransactionServiceLauncher.class) {\n\t\t\twhile (running) {\n\t\t\t\ttry {\n\t\t\t\t\tTransactionServiceLauncher.class.wait();\n\t\t\t\t} catch (Throwable e) {\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\n\t}\n",NAMING.UUVN\n458094,"\tpublic static void main(String[] args) {\n\t\tlong t = System.currentTimeMillis();\n\t\ttry {\n\t\t\n\t\t\tlogger.info(""distributed transaction manager  init....."");\n\t\t\tSystem.setProperty(""dubbo.application.logger"", ""slf4j"");\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t//\udcc0\udcad\udcc6\udcf0spring\udcb7\udcfe\udcce\udcf1\udcd6\udcf7\udcc5\udce4\udcd6\udcc3\udcceļ\udcfe\udcba\udccddb-service module\udcd6е\udcc4\udcd2\udcd4spring-\udcbf\udcaaͷ\udcb5\udcc4mybatis\udcc5\udce4\udcd6\udcc3\udcceļ\udcfe:spring-dbservice-mybatis.xml\n\t\t\tfinal ClassPathXmlApplicationContext context = new ClassPathXmlApplicationContext(\n\t\t\t\t\t""classpath*:distributed-transaction-service.xml"");\n\t\t\t\n\t\t\t\n\t\t\tlogger.info(""user.dir: "" + System.getProperty(""user.dir""));\n\t\t\t\n\t\t\tif (""true"".equals(System.getProperty(SHUTDOWN_HOOK_KEY))) {\n\t\t\t\tRuntime.getRuntime().addShutdownHook(new Thread() {\n\t\t\t\t\tpublic void run() {\n\t\t\t\t\t\ttry {\n\t\t\t\t\t\t\tcontext.stop();\n\t\t\t\t\t\t\tlogger.warn(""server stopped!!!"");\n\t\t\t\t\t\t} catch (Exception e) {\n\t\t\t\t\t\t\tlogger.error(e.getMessage(), e);\n\t\t\t\t\t\t} catch (Throwable t) {\n\t\t\t\t\t\t\tlogger.error(t.getMessage());\n\t\t\t\t\t\t}\n\t\t\t\t\t\tsynchronized (TransactionServiceLauncher.class) {\n\t\t\t\t\t\t\trunning = false;\n\t\t\t\t\t\t\tTransactionServiceLauncher.class.notify();\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t});\n\t\t\t}\n\t\t\t\n\t\t\t//\udcc0\udcad\udcc6\udcf0\udcb7\udcfe\udcce\udcf1\n\t\t\tcontext.start();\n\t\t\tlogger.info(""distributed transaction manager Started!  take "" + (System.currentTimeMillis() - t) + "" ms"");\n\t\t} catch (RuntimeException e) {\n\t\t\tlogger.error(e.getMessage(), e);\n\t\t\tSystem.exit(1);\n\t\t}\n\t\tsynchronized (TransactionServiceLauncher.class) {\n\t\t\twhile (running) {\n\t\t\t\ttry {\n\t\t\t\t\tTransactionServiceLauncher.class.wait();\n\t\t\t\t} catch (Throwable e) {\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\n\t}\n",TRS.NSYN\n458131,"\tpublic static void main(String[] args) {\n\t\tlong t = System.currentTimeMillis();\n\t\ttry {\n\t\t\n\t\t\tlogger.info(""DubboWithDistributedTransactionAgentService1Launcher  init....."");\n\t\t\tSystem.setProperty(""dubbo.application.logger"", ""slf4j"");\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t//\udcc0\udcad\udcc6\udcf0spring\udcb7\udcfe\udcce\udcf1\udcd6\udcf7\udcc5\udce4\udcd6\udcc3\udcceļ\udcfe\udcba\udccddb-service module\udcd6е\udcc4\udcd2\udcd4spring-\udcbf\udcaaͷ\udcb5\udcc4mybatis\udcc5\udce4\udcd6\udcc3\udcceļ\udcfe:spring-dbservice-mybatis.xml\n\t\t\tfinal ClassPathXmlApplicationContext context = new ClassPathXmlApplicationContext(\n\t\t\t\t\t""classpath*:test-dubbo-service1.xml"");\n\t\t\t\n\t\t\t\n\t\t\tlogger.info(""user.dir: "" + System.getProperty(""user.dir""));\n\t\t\t\n\t\t\tif (""true"".equals(System.getProperty(SHUTDOWN_HOOK_KEY))) {\n\t\t\t\tRuntime.getRuntime().addShutdownHook(new Thread() {\n\t\t\t\t\tpublic void run() {\n\t\t\t\t\t\ttry {\n\t\t\t\t\t\t\tcontext.stop();\n\t\t\t\t\t\t\tlogger.warn(""server stopped!!!"");\n\t\t\t\t\t\t} catch (Exception e) {\n\t\t\t\t\t\t\tlogger.error(e.getMessage(), e);\n\t\t\t\t\t\t} catch (Throwable t) {\n\t\t\t\t\t\t\tlogger.error(t.getMessage());\n\t\t\t\t\t\t}\n\t\t\t\t\t\tsynchronized (TestService1Launcher.class) {\n\t\t\t\t\t\t\trunning = false;\n\t\t\t\t\t\t\tTestService1Launcher.class.notify();\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t});\n\t\t\t}\n\t\t\t\n\t\t\t//\udcc0\udcad\udcc6\udcf0\udcb7\udcfe\udcce\udcf1\n\t\t\tcontext.start();\n\t\t\tlogger.info(""DubboWithDistributedTransactionAgentService1Launcher Started!  take "" + (System.currentTimeMillis() - t) + "" ms"");\n\t\t} catch (RuntimeException e) {\n\t\t\tlogger.error(e.getMessage(), e);\n\t\t\tSystem.exit(1);\n\t\t}\n\t\tsynchronized (TestService1Launcher.class) {\n\t\t\twhile (running) {\n\t\t\t\ttry {\n\t\t\t\t\tTestService1Launcher.class.wait();\n\t\t\t\t} catch (Throwable e) {\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\n\t}\n",NAMING.UUVN\n458132,"\tpublic static void main(String[] args) {\n\t\tlong t = System.currentTimeMillis();\n\t\ttry {\n\t\t\n\t\t\tlogger.info(""DubboWithDistributedTransactionAgentService1Launcher  init....."");\n\t\t\tSystem.setProperty(""dubbo.application.logger"", ""slf4j"");\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t//\udcc0\udcad\udcc6\udcf0spring\udcb7\udcfe\udcce\udcf1\udcd6\udcf7\udcc5\udce4\udcd6\udcc3\udcceļ\udcfe\udcba\udccddb-service module\udcd6е\udcc4\udcd2\udcd4spring-\udcbf\udcaaͷ\udcb5\udcc4mybatis\udcc5\udce4\udcd6\udcc3\udcceļ\udcfe:spring-dbservice-mybatis.xml\n\t\t\tfinal ClassPathXmlApplicationContext context = new ClassPathXmlApplicationContext(\n\t\t\t\t\t""classpath*:test-dubbo-service1.xml"");\n\t\t\t\n\t\t\t\n\t\t\tlogger.info(""user.dir: "" + System.getProperty(""user.dir""));\n\t\t\t\n\t\t\tif (""true"".equals(System.getProperty(SHUTDOWN_HOOK_KEY))) {\n\t\t\t\tRuntime.getRuntime().addShutdownHook(new Thread() {\n\t\t\t\t\tpublic void run() {\n\t\t\t\t\t\ttry {\n\t\t\t\t\t\t\tcontext.stop();\n\t\t\t\t\t\t\tlogger.warn(""server stopped!!!"");\n\t\t\t\t\t\t} catch (Exception e) {\n\t\t\t\t\t\t\tlogger.error(e.getMessage(), e);\n\t\t\t\t\t\t} catch (Throwable t) {\n\t\t\t\t\t\t\tlogger.error(t.getMessage());\n\t\t\t\t\t\t}\n\t\t\t\t\t\tsynchronized (TestService1Launcher.class) {\n\t\t\t\t\t\t\trunning = false;\n\t\t\t\t\t\t\tTestService1Launcher.class.notify();\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t});\n\t\t\t}\n\t\t\t\n\t\t\t//\udcc0\udcad\udcc6\udcf0\udcb7\udcfe\udcce\udcf1\n\t\t\tcontext.start();\n\t\t\tlogger.info(""DubboWithDistributedTransactionAgentService1Launcher Started!  take "" + (System.currentTimeMillis() - t) + "" ms"");\n\t\t} catch (RuntimeException e) {\n\t\t\tlogger.error(e.getMessage(), e);\n\t\t\tSystem.exit(1);\n\t\t}\n\t\tsynchronized (TestService1Launcher.class) {\n\t\t\twhile (running) {\n\t\t\t\ttry {\n\t\t\t\t\tTestService1Launcher.class.wait();\n\t\t\t\t} catch (Throwable e) {\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\n\t}\n",TRS.NSYN\n'

df = pd.read_csv(io.StringIO(df_txt), encoding_errors="surrogateescape", index_col=0)

pd.Series([len(df[df.duplicated(subset=["source", "rule.id"], keep=False)]) for _ in range(100000)]).describe()

As you can see it calculates the length of the dataset constructed from duplicated rows. It should always return the same number of rows, yet sometimes it's 0 and sometimes it's 4!
Here are example values that I've got:

count    100000.000000
mean          2.388840
std           1.961847
min           0.000000
25%           0.000000
50%           4.000000
75%           4.000000
max           4.000000

count    100000.00000
mean          1.15424
std           1.81238
min           0.00000
25%           0.00000
50%           0.00000
75%           4.00000
max           4.00000

count    100000.000000
mean          3.967680
std           0.358102
min           0.000000
25%           4.000000
50%           4.000000
75%           4.000000
max           4.000000

This is super strange.

[emsi]

Additional note:
I was not able to replicate that on google colab (with py3.9 and pd 1.4.2 installed) but I replicated it on yet another (3rd) different computer, this time a laptop with Ubuntu 22.04 and python 3.10, so it might be related to some CPU optimization.

[rhshadrach]

Thanks, attempting to reduce your example, I'm able to reproduce with:

df['source'] = df['source'].str[1083:1084]

This is the bytes: b'\\udcc0'; however if I try to use this to create a DataFrame from scratch using these bytes, I'm not able to reproduce. I wonder if this is related to #34551.

Further investigations and PRs to fix are welcome!