ENH: Allow opting in to new dtypes on I/O routines via keyword to I/O routines

[Dr-Irv]

With the new dtypes (IntegerArray, StringArray, etc.), if you want to use them when reading in data, you have to specify the types for all of the columns. It would be nice to have the option to use the new dtypes for all columns as a keyword to read_csv(), read_excel(), etc.

(ref. discussion in pandas dev meeting on 11/20/19)

[TomAugspurger]

@jorisvandenbossche is this likely for 1.0? My initial preference is to add this in the future (as an option in 1.1 say)

[jorisvandenbossche]

It's certainly not release critical, so let's remove from the milestone.

[jorisvandenbossche]

So do have an idea how we would like to tackle this? I think also the constructors (eg DataFrame(..)) could use a similar option.

An option like use_new_dtypes=True/False consistently across functions that create dataframes/series?
Or a better name, as "new" is not very descriptive. use_nullable_dtypes might not be fully covering for eg strings, as those were already nullable before.

[Dr-Irv]

I'd like to argue that if you want to get people to use the new dtypes, and especially pd.NA, then this becomes pretty important, because IMHO, most missing values are introduced when people read in data. So if you don't change the I/O routines, pd.NA is unlikely to get used.

As for names, maybe use_extension_dtypes ??

[jorisvandenbossche]

What I dislike about use_extension_dtypes it that it sounds as extension to pandas, which is here not the case. The hope is that at some point those are the default dtypes.
(I know, the whole thing is called Extension.., but still).

[Dr-Irv]

How about use_distinct_dtypes ?

[Dr-Irv]

@jorisvandenbossche I started to look at this, and if we did it for each reader, it might end up being a lot of work because of all of the different reader implementations. Here's another proposal. What if we created a method DataFrame.as_nullable_types() that would take a DataFrame and convert any column it could to a nullable type. Then if you used any reader, which didn't use the new types, you could convert the entire DataFrame in one line, so you could have df = pd.read_csv('filename.csv').as_nullable_types() or df = pd.read_excel('filename.excel').as_nullable_types(), etc. The rules could look something like this:

1. If dtype is object, convert to string. If it fails, leave it alone.
2. If dtype is float, try conversion to boolean. If that fails, try conversion to Int64. If that fails, leave it alone.

My goal here is to make it easy for people to use the new StringDType, Int64DType, and BooleanDType. If we don't do something like this, I don't think those types will get exercised, because missing values are typically encountered when reading data, and it is painful to have to specify the dtype for each column when reading data with lots of columns.

[jorisvandenbossche]

if we did it for each reader, it might end up being a lot of work because of all of the different reader implementations

If we want to have efficient methods, we will probably need to end up with reader-specific implementation anyway, I think. But, that doesn't mean of course that all of the readers need to support it natively, we can start with the important ones and have others convert it after reading. For example, I was planning to work on a parquet reader that directly gives you nullable integers (which avoids a copy and an unneeded roundtrip to float). All to say: I think it is still useful to have it as a reader option as well in some cases.

But that said, a helper method/function to convert an existing dataframe into a new dataframe using nullable types sounds like a good idea, that will be useful anyway.
Conceptually, it is somewhat similar as DataFrame.infer_objects ("Attempt to infer better dtypes for object columns.", except here we want to infer better dtypes for all columns)

[Dr-Irv]

@jorisvandenbossche I'll work on the helper method, as I think it should be in 1.0, and then later we can figure out how to change the various readers (and in which order) for a later version.