API: .convert_objects is deprecated, do we want a .convert to replace?

[jreback]

xref #11173

or IMHO simply replace by use of pd.to_datetime,pd.to_timedelta,pd.to_numeric.

Having an auto-guesser is ok, but when you try to forcefully coerce things can easily go awry.

[jreback]

cc @bashtage
@jorisvandenbossche @shoyer @TomAugspurger @sinhrks

[bashtage]

There is already _convert which could be promoted.

On Fri, Oct 2, 2015, 10:16 AM Jeff Reback notifications@github.com wrote:

cc @bashtage https://github.com/bashtage
@jorisvandenbossche https://github.com/jorisvandenbossche @shoyer
https://github.com/shoyer @TomAugspurger
https://github.com/TomAugspurger @sinhrks https://github.com/sinhrks

—
Reply to this email directly or view it on GitHub
#11221 (comment).

[bashtage]

The advantage of a well designed convert is that it works on DataFrames. All of to_* are only for 1-d types.

[jreback]

@bashtage oh I agree.

The problem is with coerce, you have to basically not auto-coerce things partially and so leave ambiguous things up to the user (via a 1-d use of the pd.to_*). But assuming we do that then yes, you could make it work.

[bashtage]

I was just thinking of the case where I imported data that should be numeric into a DF, but it has some mixed characters, and I want just numbers or NaNs. This type of conversion is what I ultimately wanted when I started looking at convert_objects when I was surprised that asking to coerce of all strings didn't coerce it to NaN.

[jreback]

but the problem is that of a mixed boolean/nan one is ambiguous (so maybe just need to 'handle' that)

[jorisvandenbossche]

Some comments/observations:

• I actually like convert_objects more than convert, because it more clearly says what it does: try to convert objects dtyped columns to a builtin dtype (convert is rather general).

• If we decide that we something like current the convert objects functionality, I don't really see a reason to deprecate convert_objects for a new converts. I think it should be technically possible to deprecate the old keywords (and not the function) in favor of new keywords. (actually the original approach in the reverted PR).

• I think the functionality of convert_objects is useful (as already is stated above: that you can do something like to_datetime/to_numeric/.. on dataframes). Using the to_.. functions on each series separate will always be the preferable solution for robust code, but as long as convert_objects is very clearly defined (now there are some strange inconsistencies), I think it is useful to have this. It would be very nice if this could just be implemented in terms of the to_.. methods.
  A bit simplified in pseudo code:

  def convert_objects(self, numeric=False, datetime=False, timedelta=False, coerce=False):
      for each column:
          if numeric:
              pd.to_numeric(self, coerce=coerce)
          elif datetime:
              pd.to_datetime(self, coerce=coerce)
          elif timedelta:
              pd.to_timedelta(self, coerce=coerce)
  

• But, the main problem with this is: the reason convert_objects is useful now, is precisely because it has an extra 'rule' that the to_.. methods don't have: only convert the column if there is at least one value that can be converted.
  This is the reason that something like this works:

  In [2]: df = pd.DataFrame({'int_str':['1', '2'], 'real_str':['a', 'b']})
  
  In [3]: df.convert_objects(convert_numeric=True)
  Out[3]:
     int_str real_str
  0        1        a
  1        2        b
  
  In [4]: df.convert_objects(convert_numeric=True).dtypes
  Out[4]:
  int_str      int64
  real_str    object
  dtype: object
  

  and does not give:

  Out[3]:
     int_str   real_str
  0        1        NaN
  1        2        NaN
  

  which would not be really useful (although maybe more predictable). The fact that is not always coerced to NaNs was considered as a bug, for which @bashtage did a PR (and for to_numeric, it is also logical that it returns NaNs). But this made convert_objects also less useful (so it was reverted in the end).
  So I think that in this case, we will have to deviate from the to_.. behaviour

[jorisvandenbossche]

Maybe this could be an extra parameter to convert/convert_objects: whether to coerce non-convertible-columns to NaN or not (meaning: columns for which there is at least not one element convertible and would lead to a full NaN column). @bashtage then you could have the behaviour you want, but the method can still be used for dataframes were not all columns should be considered as numeric.

[jreback]

ok so the question is should we u deprecate convert_objects thrn?

I actually think convert is s much better name snd we certainly could add the options u describe to make it more useful

[bashtage]

convert_objects just seems like a bad API feature since it has this path dependence where it

tries to convert to type a
tries to convert to type b if a fails, but not if a succeeds
tries to convert to type c is a and b fail, but not if either succeed

A better design would only convert a single type which removes any ambiguity if some data is ever convertible to more than one type. to to_* sort of get there, with the caveat that they operate column by column.

[hayd]

Long live convert_objects!