Skip to content

BUG: preserve dtype to bool[pyarrow] when calling pyarrow backed Series.isna() #59436

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 8 commits into from
Closed

BUG: preserve dtype to bool[pyarrow] when calling pyarrow backed Series.isna() #59436

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
2710225
preserver dtpye to bool[pyarrow] when calling Series.isna()
KevsterAmp Aug 7, 2024
85b3ff3
remove unused type(obj.dtype)
KevsterAmp Aug 7, 2024
c099d9b
add bug on whatsnew doc
KevsterAmp Aug 7, 2024
afa2380
fix issue number on bug addition in whatsnew doc
KevsterAmp Aug 7, 2024
c30fb98
Merge remote-tracking branch 'upstream/main' into return-bool-pyarrow…
KevsterAmp Aug 8, 2024
3b15abc
Merge remote-tracking branch 'upstream/main' into return-bool-pyarrow…
KevsterAmp Aug 21, 2024
9aac9a8
improve pyarrow addition
KevsterAmp Aug 21, 2024
d602fc7
add test to bool[pyarrow] implementation
KevsterAmp Aug 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v3.0.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -539,6 +539,7 @@ Conversion
- Bug in :meth:`DataFrame.update` bool dtype being converted to object (:issue:`55509`)
- Bug in :meth:`Series.astype` might modify read-only array inplace when casting to a string dtype (:issue:`57212`)
- Bug in :meth:`Series.reindex` not maintaining ``float32`` type when a ``reindex`` introduces a missing value (:issue:`45857`)
- Bug in :meth:`Series.isna` returning ``bool`` instead of ``bool[pyarrow]`` for Arrow-based Series (:issue:`59436`)

Strings
^^^^^^^
Expand Down
14 changes: 13 additions & 1 deletion pandas/core/dtypes/missing.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
is_string_or_object_np_dtype,
)
from pandas.core.dtypes.dtypes import (
ArrowDtype,
CategoricalDtype,
DatetimeTZDtype,
ExtensionDtype,
Expand Down Expand Up @@ -206,7 +207,18 @@ def _isna(obj):
elif isinstance(obj, ABCSeries):
result = _isna_array(obj._values)
# box
result = obj._constructor(result, index=obj.index, name=obj.name, copy=False)
if isinstance(obj.dtype, ArrowDtype):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would usually recommend to avoid any additional nested if statements, it makes it extra verbose and harder to read.
If you look at the signature of the constructor you can choose to pass dtype=None if you want to let pandas handle the type internally.

result = obj._constructor(
result,
index=obj.index,
name=obj.name,
copy=False,
dtype="bool[pyarrow]",
)
else:
result = obj._constructor(
result, index=obj.index, name=obj.name, copy=False
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
result, index=obj.index, name=obj.name, copy=False
new_dtype = "bool[pyarrow]" if isinstance(obj.dtype, ArrowDtype) else None
result = obj._constructor(result, index=obj.index, name=obj.name, copy=False, dtype=new_dtype)

Or something of the sort would be more readable.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You will also need to add a test case to make sure your changes are getting covered on a specific situation.

)
return result
elif isinstance(obj, ABCDataFrame):
return obj.isna()
Expand Down
Loading