first line comments on a read_csv

[hayd]

related #4505

It seems that commenting on the first line is a little buggy (or perhaps not well-defined):

In [11]: s1 = '# notes\na,b,c\n# more notes\n1,2,3'

In [12]: s2 = 'a,b,c\n# more notes\n1,2,3'

In [13]: pd.read_csv(StringIO(s1), comment='#')
Out[13]: 
        Unnamed: 0
a   b            c
NaN NaN        NaN
1   2            3

In [14]: pd.read_csv(StringIO(s2), comment='#')
Out[14]: 
    a   b   c
0 NaN NaN NaN
1   1   2   3

If you ignore the header:

In [15]: pd.read_csv(StringIO(s1), comment='#', header=None)
CParserError: Error tokenizing data. C error: Expected 1 fields in line 2, saw 3

related #3001 and from this SO question.

[jreback]

see also #4505

[jreback]

of course in the header is special issue

[hayd]

yeah, now I think about it, the first behaviour is ok. Only ignoring header is bug.

[gfyoung]

I don't think this is an issue anymore:

>>> from pandas import read_csv
>>> from pandas.compat import StringIO
>>> data = '# notes\na,b,c\n# more notes\n1,2,3'
>>> read_csv(StringIO(data), engine='c', comment='#')
   a  b  c
0  1  2  3
>>> read_csv(StringIO(data), engine='python', comment='#')
   a  b  c
0  1  2  3
>>> read_csv(StringIO(data), engine='c', comment='#', header=None)
   0  1  2
0  a  b  c
1  1  2  3
>>> read_csv(StringIO(data), engine='python', comment='#', header=None)
   0  1  2
0  a  b  c
1  1  2  3

[jreback]

ok, this looks closable, can you put some tests.