BUG: Result datatype is object when concatting decimals of different scales with pyarrow backend.

[rohanjain101]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

>>> pd.options.mode.dtype_backend = 'pyarrow'
>>> df_b = pd.DataFrame({"A":pd.Series([decimal.Decimal("1234.2")], dtype=pd.ArrowDtype(pa.decimal128(6,2)))})
>>> df_a = pd.DataFrame({"A":pd.Series([decimal.Decimal("1.23")], dtype=pd.ArrowDtype(pa.decimal128(3,2)))})
>>> pd.concat([df_a, df_b]).dtypes
A    object
dtype: object
>>>

Issue Description

The resulting dtype is object when using the pyarrow object, when the data type should be decimal with the result rescaled.

Expected Behavior

df_b = pd.DataFrame({"A":pd.Series([decimal.Decimal("2.23")], dtype=pd.ArrowDtype(pa.decimal128(3,2)))})
df_a = pd.DataFrame({"A":pd.Series([decimal.Decimal("1.23")], dtype=pd.ArrowDtype(pa.decimal128(3,2)))})
pd.concat([df_a, df_b]).dtypes
A decimal128(3, 2)[pyarrow]
dtype: object

When they are the same scale/precision, the result remains a pyarrow dtype and is not changed to object.

Installed Versions

pd.show_versions()
C:\ps_0310_is\lib\site-packages_distutils_hack_init_.py:33: UserWarning: Setuptools is replacing distutils.
warnings.warn("Setuptools is replacing distutils.")

INSTALLED VERSIONS

commit : 1a2e300
python : 3.8.10.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.22621
machine : AMD64
processor : Intel64 Family 6 Model 85 Stepping 7, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252

pandas : 2.0.0rc0
numpy : 1.24.2
pytz : 2022.7.1
dateutil : 2.8.2
setuptools : 67.2.0
pip : 22.3.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : 2023.2.0
fsspec : 2023.3.0
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 11.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : None
qtpy : None
pyqt5 : None

[mroeschke]

Thanks for the report. Currently we don't have any existing logic to resolve Arrow types that don't have a corresponding numpy type. This may suitable functionality for Arrow to implement.

[rohanjain101]

@mroeschke It seems like there's atleast some logic to resolve arrow types that don't have corresponding numpy type. For example, the case where the series concatted have same precision/scale, pandas was able to preserve the arrow dtype, only in case where they differ, it is falling back to object.

[mroeschke]

Right, moreover in your Expected Behavior example, the ArrowDtypes are the same, and concat does not have to resolve the types if all DataFrame columns have the same type.

[rohanjain101]

@mroeschke Is the pandas implementation for concat with arrow types intended to be similar to pa.compute.concat_table? If so, this example would raise an error:

>>> t1 = pa.table([
...     pa.array([decimal.Decimal("1234.23"), decimal.Decimal("1243.46")], type=pa.decimal128(6,2)),
...     pa.array(["Flamingo", "Horse"])
...     ], names=['n_legs', 'animals'])
>>> t2 = pa.table([
...     pa.array([decimal.Decimal("1.12"), decimal.Decimal("2.12")], type=pa.decimal128(3,2)),
...     pa.array(["Parrot", "Dog"])
...     ], names=['n_legs', 'animals'])
>>> pa.concat_tables([t1,t2])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pyarrow\table.pxi", line 5224, in pyarrow.lib.concat_tables
  File "pyarrow\error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow\error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Schema at index 1 was different:
n_legs: decimal128(6, 2)
animals: string
vs
n_legs: decimal128(3, 2)
animals: string
>>>

[mroeschke]

Possibly, in the example above, I think pandas would also not like to resize the decimal precision and raise an error

[rohanjain101]

Just tried it out, seems its converted to object instead of raising:

>>> df_b = pd.DataFrame({"A":pd.Series([decimal.Decimal("1234.2")], dtype=pd.ArrowDtype(pa.decimal128(6,2)))})
>>> df_a = pd.DataFrame({"A":pd.Series([decimal.Decimal("1.23")], dtype=pd.ArrowDtype(pa.decimal128(3,2)))})
>>> pd.concat([df_a, df_b]).dtypes
A    object
dtype: object
>>>

[jbrockmendel]

Concatting decimal128(a1, b1) with decimal128(a2, b2) IIUC we can't in general find a common dtype that is non-lossy, but when b1==b2 can't we use decimal128(max(a1, a2), b1)?