BUG: convert_objects(convert_numeric=True) fails with all strings

[bashtage]

s=pd.Series(['a','a','a'])

s.convert_objects(convert_numeric=True)
Out[78]: 
0    a
1    a
2    a
dtype: object

s[0]=1.0

s
Out[80]: 
0    1
1    a
2    a
dtype: object

s.convert_objects(convert_numeric=True)
Out[81]: 
0     1
1   NaN
2   NaN
dtype: float64

Having a single number changes behavior. Makes convert_objects unreliable when the data type must be numeric.

[jreback]

this was on pupose to avoid indiscrimate converting, hence this is somewhat of a 'soft' option. could add a 'force' which just converts to numeric no matter what.

[bashtage]

coerce might be the most consistent keyword with other functions that do conversions.

[jreback]

sorry, that's what I meant. sounds reasonable to me.

[bashtage]

I would argue that the default behavior feels a bit like an attempt at "magic" -- that is, guessing what I want rather than doing what I say.

[jreback]

@bashtage you can certainly try to change it. I am ok with that. I think the problem was that it was overly aggressive and broke bunch of stuff, but if you want to make it work fine too.

[detroitcoder]

What about using convert_object on a field of strings where only one of the fields can be converted to a numeric?

eg.

converted_df = DataFrame(list("abc1xyz")).convert_objects(convert_numeric=True)
converted_df.info()
>> class 'pandas.core.frame.DataFrame'>
Int64Index: 7 entries, 0 to 6
Data columns (total 1 columns):
0    1 non-null float64
dtypes: float64(1)

Only one numeric value will force this field to float64, set all strings to null, and convert the rest to floats. This seems a bit more like black magic.

[jreback]

@detroitcoder this is exactly the purpose of using convert_objects(convert_numeric=True) to force convert things to numeric, or set them to nan

[detroitcoder]

That seems a bit confusing and dangerous. Say I have a dataframe with three columns, all containing string objects. The first contains no fields that can be converted to int, the second only contains one, and the third can all be converted to ints. If someone uses convert_objects(convert_numeric=True) on this dataframe, the third field is set to int32 and the second is float64.

df = DataFrame(zip(list("abc"), list("xy1"), list("123")))
converted_df = df.convert_objects(convert_numeric=True)
converted_df.info()

>> class 'pandas.core.frame.DataFrame'>
Int64Index: 3 entries, 0 to 2
Data columns (total 3 columns):
0    3 non-null object
1    1 non-null float64
2    3 non-null int64
dtypes: float64(1), int64(1), object(1)
memory usage: 96.0+ bytes

If a user has to know the datatype of the columns before using convert_objects they could just use DataFrame.astype and DataFrame.replace with a regex to convert strings to nulls. I was under the impression that convert_objects could be used to convert objects without having to know what the underlying data consists of.

[jreback]

the point is to force convert something that astype won't work on

what are you trying to do? you rarely actually need to use convert_objects; only use case is what I mentioned above

[detroitcoder]

I want to convert objects from various sources to ndtypes without metadata or knowing what the data consists of. If I could use convert_numeric=True for a soft convert, and convert_numeric="coerce" for a forced convert, similar to convert_dates, that would be perfect. However, convert_numeric=True actually means 'coerce' which means I can't use convert_objects all together.

[jreback]

@detroitcoder well, where is your data coming from? read_csv/read_sql and most of the input converters automatically coerce things for you. You have to work pretty hard to construct a mixed-type object array. I mean you may have it and that's when convert_objects() comes in handy. The default settings do a 'soft' conversion, that is if all of the objects are convertible it will coerce it appropriately. convert_numeric is to perform 'forced' conversions that for some reason you don't/can't do using say read_csv.