Float conversion makes fillna lossy

[jbrockmendel]

For int64 data near the int64 implementation bounds, astype('float64') or ensure_float64 is lossy. The motivating case is PeriodArray.fillna

dti = pd.date_range(pd.Timestamp.max - pd.Timedelta(nanoseconds=10), periods=5, freq='ns')
pi = dti.to_period('ns')
parr = pi._data
parr[2] = pd.NaT

>>> parr.fillna(method='pad')
<PeriodArray>
['NaT', 'NaT', 'NaT', 'NaT', 'NaT']
Length: 5, dtype: period[N]

[jreback]

In [1]: dti = pd.date_range(pd.Timestamp.max - pd.Timedelta(nanoseconds=10), periods=5, freq='ns')
   ...: pi = dti.to_period('ns')
   ...: parr = pi._data
   ...: parr[2] = pd.NaT
   ...: 

In [2]: parr
Out[2]: 
<PeriodArray>
['2262-04-11 23:47:16.854775797', '2262-04-11 23:47:16.854775798',
                           'NaT', '2262-04-11 23:47:16.854775800',
 '2262-04-11 23:47:16.854775801']
Length: 5, dtype: period[N]

In [4]: Series(parr).fillna(parr[0])
Out[4]: 
0    2262-04-11 23:47:16.854775797
1    2262-04-11 23:47:16.854775798
2    2262-04-11 23:47:16.854775797
3    2262-04-11 23:47:16.854775800
4    2262-04-11 23:47:16.854775801
dtype: period[N]

Series goes thru the correct path, wonder what its dispatching too

actually no, you are right this is lossy.

[jbrockmendel]

(Internet is down, typing with thumbs for a while)

I’ve poked at this a bit for interpolate; if the int64 vals fall inside int32 bounds then we are OK. Otherwise need to cast to float128 to be assured lossless.

[mroeschke]

This looks okay on master now. Could use a test

In [20]: parr.fillna(method='pad')
Out[20]:
<PeriodArray>
['2262-04-11 23:47:16.854775797', '2262-04-11 23:47:16.854775798',
 '2262-04-11 23:47:16.854775798', '2262-04-11 23:47:16.854775800',
 '2262-04-11 23:47:16.854775801']
Length: 5, dtype: period[N]

[KianYang-Lee]

Hi @mroeschke, I would like to contribute. Can you provide a little guideline on what to test on? Thanks

[mroeschke]

@KianYang-Lee would need a test to validate that the code snippet in the original post returns the result in my previous comment.

[KianYang-Lee]

OK taking this. will come up with the test and result soon

[jackgoldsmith4]

@KianYang-Lee are you still working on this?

@jackgoldsmith4
may I take it instead?

[KianYang-Lee]

No, I'm not. Got caught up with work. Sorry and Please proceed

@KianYang-Lee are you still working on this?