BUG: read_fwf fails with comments='#'

[naught101]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

With this file:

test.txt:

# file :: atlas/series/CMIP5one/rcp85/monthly/world/time_tas_Amon_onemean_rcp85_000_world.dat
   93.5000       17.9877     2093   1 2093   1
   94.5000       18.0374     2094   1 2094   1
   95.5000       18.0906     2095   1 2095   1
   96.5000       18.1404     2096   1 2096   1
   97.5000       18.2044     2097   1 2097   1
   98.5000       18.2412     2098   1 2098   1
   99.5000       18.2734     2099   1 2099   1
   100.500       18.3431     2100   1 2100   1

In [3]: pd.read_fwf("test.txt", comment='#', header=None)
Out[3]: 
         0        1     2  3     4  5
0   ile ::  eries/C  ne/r  5  mont  /
1  93.5000  17.9877  2093  1  2093  1
2  94.5000  18.0374  2094  1  2094  1
3  95.5000  18.0906  2095  1  2095  1
4  96.5000  18.1404  2096  1  2096  1
5  97.5000  18.2044  2097  1  2097  1
6  98.5000  18.2412  2098  1  2098  1
7  99.5000  18.2734  2099  1  2099  1
8  100.500  18.3431  2100  1  2100  1

on larger files with more header columns, all rows are included in this way.

Issue Description

Comment lines are included in the dataframe

Expected Behavior

Comment lines should NOT be included in the dataframe

Installed Versions

INSTALLED VERSIONS

commit : 0f43794
python : 3.10.10.final.0
python-bits : 64
OS : Linux
OS-release : 5.15.0-73-lowlatency
Version : #80-Ubuntu SMP PREEMPT Wed May 17 13:58:47 UTC 2023
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_AU.UTF-8
LOCALE : en_AU.UTF-8

pandas : 2.0.3
numpy : 1.24.3
pytz : 2023.3
dateutil : 2.8.2
setuptools : 65.6.3
pip : 23.0.1
Cython : None
pytest : 7.3.1
hypothesis : 6.75.2
sphinx : None
blosc : None
feather : None
xlsxwriter : 3.1.2
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : 2.9.3
jinja2 : 3.1.2
IPython : 8.13.2
pandas_datareader: None
bs4 : 4.12.2
bottleneck : None
brotli :
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.7.1
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.10.1
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : 2023.6.0
xlrd : None
zstandard : 0.19.0
tzdata : 2023.3
qtpy : 2.3.1
pyqt5 : None

[rsm-23]

@naught101 this is reproducible but I don't see any parameter called "comment" in the read_fwf documentation