ENH: allow engine='pyarrow' in read_csv

[jreback]

we could conditionally use the new pyarrow csv parser as an engine (requires 0.11 IIRC). eventually leading to a replacement path for the existing code. There might be a number of restrictions on what options we can pass as the current parser is more full-featured, but I suspect most of the basic options work.

cc @gfyoung @pitrou @wesm @TomAugspurger

[wesm]

I would expect it to be more limited in functionality in general, so maybe not a full replacement for pandas's read_csv, but it could be used as the "fast path" for well behaved files. Also, as soon as we build support for multi-character and regex delimiters, you could deprecate engine='python'

[pitrou]

I would be cautious here. Parsing or conversions may be stricter, less datatypes may be recognized, or the resulting datatypes may be different. I think it's better to make this an explicit option (engine="arrow" perhaps?).

[wesm]

Yes, I definitely agree with an "opt-in". It would be helpful if the pandas CSV suite could be set up so that we can run those tests using the Arrow reader to test for compatibility (it would be helpful to know for example, if say 50% of the tests work). Compatibility with pandas.read_csv is not our objective at the moment, though

[jreback]

yep would be opt in (as also requires newer pyarrow)

[lithomas1]

take