ENH: add read_csv sniffing (sep=None) for C engine

[jorisvandenbossche]

If you now do pd.read_csv(myfile.csv, sep=None), you get the warning:

ParserWarning: Falling back to the 'python' engine because the 'c' engine does not support sep=None with delim_whitespace=False; you can avoid this warning by specifying engine='python'

Is this warning correct that it is not implemented for the c engine?
If so, it could maybe be added.

[iskandr]

👍 I would also appreciate this feature.

[gfyoung]

@jorisvandenbossche : The reason it isn't in the C engine is because we rely on Python's csv library to do the sniffing for us. This could easily be transferred to the C engine, but is that too much coupling between the CParser and the PythonParser? I am not sure. On the one hand, the implementation would be faster in C, but the library's Python implementation is very detailed in terms of its determination of sniffed delimiter. Is it worth trying to replicate such an implementation in C?

cc @jreback

[jreback]

sniffing is quite cheap and you don't have to implement it at all
just use the sniffer
iirc it only reads a few lines

this can be done before the engine actually opens the file as a preprocessing step (and make the interface uniform between the engines)

[gfyoung]

@jreback : PythonParser first iterates through all of the skipped rows before finally reaching a row that should be read after which it sniffs the delimiter from that line. Let me see if something similar can be done with the C engine.

[jreback]

that's fine

this is an explicit action

[gfyoung]

Hmm...this is not as simple to do without duplicating a good amount of code from the PythonParser. Need to see if there is a nice way to refactor this.