Strict parsing for read_fwf

[eoghanmurray]

I'm importing a fixed width file which has 2 types of records (each with their own definitions).

>>> print "good:\n", pandas.read_fwf(StringIO('T1001\nT1020'), 
    widths=[2,1,1,1], names=['TYPE', 'A', 'B', 'C'])
good:
  TYPE  A  B  C
0   T1  0  0  1
1   T1  0  2  0
>>> print "good:\n", pandas.read_fwf(StringIO('T2XY\nT2XZ'),  
    widths=[2,1,1], names=['TYPE', 'D', 'E'])
good:
  TYPE  D  E
0   T2  X  Y
1   T2  X  Z
>>> print "silently dropped data from first 2 rows:\n", pandas.read_fwf(StringIO('T1001\nT1020\nT2XY\nT2XZ'), 
    widths=[2,1,1], names=['TYPE', 'D', 'E'])
silently dropped data from first 2 rows:
  TYPE  D  E
0   T1  0  0
1   T1  0  2
2   T2  X  Y
3   T2  X  Z
>>> print "unexpected NaN fields:\n", pandas.read_fwf(StringIO('T1001\nT1020\nT2XY\nT2XZ'), 
    widths=[2,1,1,1], names=['TYPE', 'A', 'B', 'C'])
unexpected NaN fields:
  TYPE  A  B    C
0   T1  0  0  1.0
1   T1  0  2  0.0
2   T2  X  Y  NaN
3   T2  X  Z  NaN

Problem description

I expected that lines not matching the passed-in spec would result in a 'bad line' error for that line, and those lines could be ignored.

Expected Output

I expected these lines to raise an error, with the error_bad_lines option available to ignore the lines and show warnings instead (which could be turned off with warn_bad_lines).

>>> pandas.read_fwf(StringIO('T1001\nT1020\nT2XY\nT2XZ'), 
    widths=[2,1,1], names=['TYPE', 'D', 'E'])
ParserError: Error tokenizing data. Expected 4 characters in line 1, saw 5

>>> pandas.read_fwf(StringIO('T1001\nT1020\nT2XY\nT2XZ'), 
    widths=[2,1,1,1], names=['TYPE', 'A', 'B', 'C'])
ParserError: Error tokenizing data. Expected 5 characters in line 3, saw 4

>>> pandas.read_fwf(StringIO('T1001\nT1020\nT2XY\nT2XZ'), 
    widths=[2,1,1], names=['TYPE', 'D', 'E'], error_bad_lines=False)
Skipping Line 1: expected 4 characters, saw 5
Skipping Line 2: expected 4 characters, saw 5

  TYPE  D  E
0   T2  X  Y
1   T2  X  Z

>>> pandas.read_fwf(StringIO('T1001\nT1020\nT2XY\nT2XZ'), 
    widths=[2,1,1,1], names=['TYPE', 'A', 'B', 'C'], error_bad_lines=False)
Skipping Line 3: expected 5 characters, saw 4
Skipping Line 4: expected 5 characters, saw 4

  TYPE  A  B  C
0   T1  0  0  1
1   T1  0  2  0

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 2.7.12.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-87-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_IE.UTF-8
LOCALE: None.None

pandas: 0.20.3
pytest: 2.9.1
pip: 9.0.1
setuptools: 27.3.0
Cython: None
numpy: 1.13.1
scipy: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 1.5
pytz: 2014.4
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.4.1
html5lib: None
sqlalchemy: 1.1.0b1
pymysql: None
psycopg2: 2.6.2 (dt dec pq3 ext lo64)
jinja2: 2.7.2
s3fs: None
pandas_gbq: None
pandas_datareader: None

[gfyoung]

I agree that the case of too many fields is a bug and should explicitly raise. As for cases with too few fields, I'm not sure I would agree. It's being interpreted as empty data (or NULL), which is perfectly acceptable.

[eoghanmurray]

Thanks!
On the 'too few fields' point maybe the example I gave was lacking.
Instead of two sets of widths of [2,1,1,1] and [2,1,1] , the problem would be when there are incompatible widths (& meanings), So e.g. [2,23,14,12,10] and [2,10,11,10]

(In the data I'm looking at, the difference between these two types is signified by the value of the first two characters of the line. Obv. different lines should have been exported to different files, but that is not the legacy data I'm working with.)

[mechif]

I tried the warn_bad_lines and error_bad_lines options - and received a warning that they are being deprecated:

 >>>  FutureWarning: The warn_bad_lines argument has been deprecated and will be removed in a future version

How can I read in a file (these are log files with set column widths) that may sometimes have badly formatted rows? I'd like to skip/delete these rows (but know that they exist) - because the other thousands of lines must be analyzed.

[phofl]

The warning tells you what to do:

FutureWarning: The warn_bad_lines argument has been deprecated and will be removed in a future version. Use on_bad_lines in the future.

[mechif]

Thanks.
This wasn't output in PyCharm 2021.3.1
I'll learn about on_bad_lines (which isn't documented in the read_fwf documentation)