cross section coercion with output iterating

[tsu-shiuan]

I'm am trying to call the to_dict function on the following DataFrame:

import pandas as pd

data = {"a": [1,2,3,4,5], "b": [90,80,40,60,30]}

df = pd.DataFrame(data)

df
   a   b
0  1  90
1  2  80
2  3  40
3  4  60
4  5  30

df.reset_index().to_dict("r")
[{'a': 1, 'b': 90, 'index': 0},
 {'a': 2, 'b': 80, 'index': 1},
 {'a': 3, 'b': 40, 'index': 2},
 {'a': 4, 'b': 60, 'index': 3},
 {'a': 5, 'b': 30, 'index': 4}]

However my problem occurs if I perform a float operation on the dataframe, which mutates the index into a float:

(df*1.0).reset_index().to_dict("r")
[{'a': 1.0, 'b': 90.0, 'index': 0.0},  
{'a': 2.0, 'b': 80.0, 'index': 1.0},  
{'a': 3.0, 'b': 40.0, 'index': 2.0},  
{'a': 4.0, 'b': 60.0, 'index': 3.0},  
{'a': 5.0, 'b': 30.0, 'index': 4.0}]

Can anyone explain the above behaviour or recommend a workaround, or verify whether or not this could be a pandas bug? None of the other outtypes in the to_dict method mutates the index as shown above.

I've replicated this on both pandas 0.14 and 0.18 (latest)

Many thanks!

link to stackoverflow: http://stackoverflow.com/questions/36548151/pandas-to-dict-changes-index-type-with-outtype-records

[TomAugspurger]

Nothing to do with the index, just the fact that you have any float dtypes in the data

data = {"a": [1.0,2,3,4,5], "b": [90,80,40,60,30]}
In [19]: df.to_dict("records")
Out[19]:
[{'a': 1.0, 'b': 90.0},
 {'a': 2.0, 'b': 80.0},
 {'a': 3.0, 'b': 40.0},
 {'a': 4.0, 'b': 60.0},
 {'a': 5.0, 'b': 30.0}]

If you look at the code, we use DataFrame.values, which returns a NumPy array, which must have a single dtype (float64 in this case).

We probably don't need to use .values here.

[tsu-shiuan]

Thanks for your response. It there a possible workaround that I can use in the meantime?

[TomAugspurger]

Something like

In [28]: [x._asdict() for x in df.itertuples()]
Out[28]:
[OrderedDict([('Index', 0), ('a', 1.0), ('b', 90)]),
 OrderedDict([('Index', 1), ('a', 2.0), ('b', 80)]),
 OrderedDict([('Index', 2), ('a', 3.0), ('b', 40)]),
 OrderedDict([('Index', 3), ('a', 4.0), ('b', 60)]),
 OrderedDict([('Index', 4), ('a', 5.0), ('b', 30)])]

That's an OrderedDict using namedtuple._asdict, you can write dict comprehension if you want a regular one.

[tsu-shiuan]

Thanks :)

[jreback]

Though one could argue that this result is correct as we don't support mixed types in int-float when doing a cross-section, IOW:

In [10]: (df*1.0).reset_index().iloc[1]
Out[10]: 
index     1.0
a         2.0
b        80.0
Name: 1, dtype: float64

this is somewhat related to #12532, meaning that we should be iterating directly over (which already does the proper coercion), rather that doing a specific coercion in .to_dict().

[jreback]

giong to mark this an an API issue that needs discussion. This would actually be a fairly large change to correctly change this (though to be honest I think the current behavior is fine).