DIS: Keywords for multi-threading capabilities

[lithomas1]

With the addition of the new pyarrow engine, we now have the option to use multiple threads to read a CSV file. (This is also controllable through the pyarrow.set_cpu_count option).

Should we expose a keyword(such as num_threads maybe) to the user as a keyword, or just add an example in the docs(for this case, redirecting to pyarrow.set_cpu_count? In the case of read_csv, this keyword would probably only apply to the pyarrow engines, however it is worth noting that we have had multiple feature requests for parallel CSV reading (e.g. #37955), and it is probably worth it to be configure the number of threads used if we offer multithreading.

Personally, I would prefer having a keyword, as if we decide to add more I/O engines with multithreading capabilities, it would be more convenient to be able to control this option through a keyword.

cc @pandas-dev/pandas-core

[mroeschke]

What is done for numba capabilities (engine="numba") and is the most flexible IMO is to pair engine with a engine_kwargs keywords which accepts a dict with engine configurations e.g. {"set_cpu_count": 2}

This way the pandas API isn't tied to engine specific constructs while catering to whatever capabilities each engine supports.

[jreback]

we pass thru n_threads
in the parquet reader

so wouldn't object to a matching keyword or @mroeschke suggestion

[lithomas1]

Personally, I think it might be better to just use a standard keyword argument, since it could be confusing if this changes between engines. I think this option would probably be popular enough to warrant this.

Having a standard keyword to control multi-threading could also pave the way for a global option to config this for everything.

(btw, n_threads doesn't exist in pyarrow anymore, you have to use pyarrow.set_cpu_count)

WDYT?

[rhshadrach]

I think we should prefer usage of engine_kwargs when available. This makes it clear to the user that it depends on what engine they are using. It also lessens our technical debt as engines come and go, and change argument names.

[brianpenghe]

Has this been solved? Reading a large csv is too slow with only one thread!

[phofl]

It does not seem so.

[jreback]

you can multi threaded reading with the engine='pyarrow' (in 1,4)

[jbrockmendel]

Out of curiosity, in what scenario with multi-threading available would you not want to crank it up to 11? I guess debugging?

[bashtage]

Out of curiosity, in what scenario with multi-threading available would you not want to crank it up to 11? I guess debugging?

When you use multi-process you almost always want to limit or eliminate multi-threading. For example, when using linear algebra on a spark cluster, failing to set MKL_NUM_THREADS=1 or OMP_NUM_THREADS=1 can lead to a situation where you end up with ncore * ncore threads competing for ncore resources, and basically crippling congestion.

[rhshadrach]

Out of curiosity, in what scenario with multi-threading available would you not want to crank it up to 11? I guess debugging?

I'll also add multi-user servers where you want to be (somewhat) nice to other processes.