BUG: Series.combine_first loss of precision

[jessestone7]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

>>> a = pd.Series([1666880195890293744, 5]).to_frame()
>>> b = pd.Series([6, 7, 8]).to_frame()
>>> a
                     0
0  1666880195890293744
1                    5
>>> b
   0
0  6
1  7
2  8
>>> a.combine_first(b)
                     0
0  1666880195890293760
1                    5
2                    8

Issue Description

I tried this on Pandas version 2.2.2 and I see that there is a loss of precision. This could be related to issue #51777.

Expected Behavior

1666880195890293744 should not get changed to 1666880195890293760

Installed Versions

INSTALLED VERSIONS

commit : d9cdd2e
python : 3.10.9.final.0
python-bits : 64
OS : Darwin
OS-release : 24.0.0
Version : Darwin Kernel Version 24.0.0: Tue Sep 24 23:37:36 PDT 2024; root:xnu-11215.1.12~1/RELEASE_ARM64_T6020
machine : arm64
processor : arm
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.2.2
numpy : 1.26.4
pytz : 2022.7.1
dateutil : 2.8.2
setuptools : 67.3.2
pip : 24.2
Cython : 0.29.33
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.3
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.12.0
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.0
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.6.3
numba : None
numexpr : 2.8.4
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.10.0
sqlalchemy : None
tables : 3.8.0
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2024.1
qtpy : None
pyqt5 : None

[rhshadrach]

Thanks for the report, confirmed on main. PRs to fix are welcome!

[cooolheater]

It's because the dtype changed : int64->float64->int64

To combine with 'b', NaN need to be added to 'a',
so to include NaN, 'int64' was promoted to 'float64' by below flow:

combine_first core/frame.py:8785
combine core/frame.py:8644
align core/generic.py:9447
_align_frame core/generic.py:9524
_reindex_with_indexers core/generic.py:5423
reindex_indexer core/internals/managers.py:832
take_nd core/internals/blocks.py:1020
take_nd core/array_algos/take.py:115
_take_nd_ndarray core/array_algos/take.py:131
_take_preprocess_indexer_and_fill_value core/array_algos/take.py:531
maybe_promote core/dtypes/cast.py:589

IMHO, it's a genaral practice, and converting this 'float64' back to 'int64' seems not natural.

So, I think making the result 'float64' could be a solution. WDYT?

Just for reference> if NaN exists from the first, it could be handled as 'int' :

a = pd.Series([1666880195890293744, 5,pd.NA]).to_frame()
b = pd.Series([6, 7, 8]).to_frame()
a.combine_first(b)

[rhshadrach]

So, I think making the result 'float64' could be a solution. WDYT?

It seems to me we should be able to carry out this operation without passing through floats.

[cooolheater]

The cause of 'passing through floats' is, it tries to insert NaN , while converting 'a' from len:2 to len:3. In this case, should we insert other values (like 0 ) to keep the int64 type ?

[jessestone7]

I am merely a user of Pandas, and the underlying code is far over my head, so maybe I should not be commenting here, but I wonder would it be possible to use here whatever solution was used to solve issue #39051 ?

[cooolheater]

The above patch could solve this issue ( when all the columns are 'int64' ), but could not cover mixed case like below:

a = pd.DataFrame({0:pd.Series([1666880195890293744, 5]),1:pd.Series([1.0,2.0])})
b = pd.Series([6, 7, 8]).to_frame()
a.combine_first(b)

Currently, NDFrame::align simply get just 1 fill_value as a parameter.
IMHO, to solve above case, we need to pass more context as parameters. Is this a right way?

[cooolheater]

It seems this needs more general approach, and it would not suitable for a newbie like me.
So, I'd like to close the PR, hoping someone would take this issue.

[yuanx749]

a = pd.Series([1666880195890293744, 5])
b = pd.Series([6, 7, 8])
print(a.combine_first(b))

Series.combine_first does not lose precision. It has a different implementation from DataFrame.combine_first and does not go through int64->float64->int64. I tried some quick fix using Series.combine_first column-wise but without luck, a couple of tests broken.