ENH: Remove ArrowStringArray and StringDtype("pyarrow")

[gsheni]

Feature Type

• Adding new functionality to pandas
• Changing existing functionality in pandas
• Removing existing functionality in pandas

Problem Description

I wish I could use pandas to create pyarrow backend Series for strings.
I wish there was a single data type and single extension array for strings (rather than 2).

Currently, we have 2 pyarrow data types & arrays for strings

• StringDtype("pyarrow") backend by arrays.ArrowStringArray
• ArrowDtype(pa.string()) backend by arrays.ArrowExtensionArray

I propose we use ArrowDtype(pa.string()) and ArrowExtensionArray.

Feature Description

import pyarrow as pa
import pandas as pd 

series_str_arry = pd.Series(['red', 'blue', None], dtype="string[pyarrow]")

string_ext_arry = pd.ArrowDtype(pa.string())
series_ext_arry = pd.Series(['red', 'blue', None], dtype=string_ext_arry)

assert series_str_arry.dtype == series_ext_arry.dtype
assert series_str_arry.dtype.construct_array_type() == series_ext_arry.dtype.construct_array_type()

Alternative Solutions

• Keep both data types and arrays

Additional Context

• Docs: https://pandas.pydata.org/docs/dev/reference/arrays.html#pyarrow
• Related PR: TST: Fix unsigned pyarrow types in SIGNED_INT_PYARROW_DTYPES #48457 (comment)

[mroeschke]

Noting that there was discussion to not immediately deprecate StringDtype("pyarrow") when ArrowDtype was introduced.

Also ArrowExtensionArray would need to adopt string manipulation methods for them to be equivalent.

Nonetheless, agreed that ArrowDtype should be the primary way to use pyarrow string types in the future

[jorisvandenbossche]

As I mentioned in #47818 (review), I am personally -1 on moving away from StringDtype(storage="arrow").
I think we should rather consider moving ArrowDtype towards the same model as StringDtype (and I am meaning the user-facing API, not necessarily implementation wise).

[jbrockmendel]

As I mentioned in #47818 (review), I am personally -1 on moving away from StringDtype(storage="arrow").
I think we should rather consider moving ArrowDtype towards the same model as StringDtype (and I am meaning the user-facing API, not necessarily implementation wise).

Is the idea here to introduce storage parameter/option/... for other dtypes so that "int64" could alias to one of, say, "int64[pyarrow]", "int64[masked]", "int64[numpy]"?

[jbrockmendel]

What if we used ArrowDtype(pa.string()) under the hood but patched StringDtype.__instancecheck__ to return True for ArrowDtype(pa.string())?