BUG: read_csv with engine='pyarrow' doesn't respect na_values unless entire column has na values

[eli-b]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
from io import StringIO
data = 'A\n' + '\n'.join(['CAT'] * 1)  # Number of CAT lines needs to be at least 1, works with 10,000 too
df = pd.read_csv(StringIO(data), engine='pyarrow', na_values='CAT')
assert df['A'].isna()[0]  # Doesn't raise AssertionError, as expected
# Give the cat some food:
data = 'A\n' + '\n'.join(['CAT'] * 1) + '\n' + 'food'
df = pd.read_csv(StringIO(data), engine='pyarrow', na_values='CAT')
assert df['A'].isna()[0]  # Raises AssertionError unexpectedly

Issue Description

When using read_csv with engine='pyarrow', na_values is not respected unless the whole column has NA values.

This does not reproduce without the engine='pyarrow' argument.

Expected Behavior

na_values should always be respected.

Installed Versions

INSTALLED VERSIONS

commit : 37ea63d
python : 3.11.1.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19043
machine : AMD64
processor : Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : Hebrew_Israel.1255
pandas : 2.0.1
numpy : 1.24.3
pytz : 2023.3
dateutil : 2.8.2
setuptools : 65.7.0
pip : 22.3.1
Cython : None
pytest : 7.3.1
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.12.0
pandas_datareader: None
bs4 : 4.12.2
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : 11.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : 2.3.1
pyqt5 : None

[phofl]

Hi, thanks for your report. This is fixed on main, but might need a test, not sure. cc @lithomas1

[natmokval]

I would like to work on this.

[eli-b]

No arrow label? It doesn't reproduce without using the pyarrow backend.

Even if it's fixed on main, I guess it's good that the issue would be documented for versions 2.0.0 and 2.0.1.

[phofl]

We don't document bugs

[anshuman0123]

Hey , I want to work on this issue

[lithomas1]

I think this has been fixed by me in #52087, and it should be tested, so I'm going to close this.
(The issue was that the strings_can_be_null flags wasn't set, so NA values in string columns weren't picked up).

@natmokval @anshuman0123
If you're still interested in working on the Arrow engine for read_csv, you can try fixing some of the tests marked as skip_pyarrow or xfail_pyarrow in the codebase.

(Just a warning, though: Some may be easier than others, and others are unfixable due to missing features in Pyarrow).

You can also try filtering by the "Arrow" and "IO CSV" labels to find more similar issues.