Skip to content

BUG: Add extra check for failing UTF-8 conversion #32548

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Mar 12, 2020
Merged

BUG: Add extra check for failing UTF-8 conversion #32548

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b6bb3f9
Add extra check for failing UTF-8 conversion
roberthdevries Mar 8, 2020
a53e2f0
Reformatted by black
roberthdevries Mar 8, 2020
dc777e5
Add whatsnew entry
roberthdevries Mar 8, 2020
6612801
Remove setting the exception, does not seem to be needed
roberthdevries Mar 9, 2020
62b0cd5
Add check for expected output
roberthdevries Mar 10, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.1.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -302,6 +302,7 @@ I/O
timestamps with ``version="2.0"`` (:issue:`31652`).
- Bug in :meth:`read_csv` was raising `TypeError` when `sep=None` was used in combination with `comment` keyword (:issue:`31396`)
- Bug in :class:`HDFStore` that caused it to set to ``int64`` the dtype of a ``datetime64`` column when reading a DataFrame in Python 3 from fixed format written in Python 2 (:issue:`31750`)
- Bug in :meth:`read_excel` where a UTF-8 string with a high surrogate would cause a segmentation violation (:issue:`23809`)


Plotting
Expand Down
3 changes: 3 additions & 0 deletions pandas/_libs/src/parse_helper.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ int floatify(PyObject *str, double *result, int *maybe_int) {
data = PyBytes_AS_STRING(str);
} else if (PyUnicode_Check(str)) {
tmp = PyUnicode_AsUTF8String(str);
if (tmp == NULL) {
return -1;
}
data = PyBytes_AS_STRING(tmp);
} else {
PyErr_SetString(PyExc_TypeError, "Invalid object type");
Expand Down
Binary file added pandas/tests/io/data/excel/high_surrogate.xlsx
View file
Open in desktop
Binary file not shown.
8 changes: 8 additions & 0 deletions pandas/tests/io/excel/test_readers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1044,3 +1044,11 @@ def test_excel_read_binary(self, engine, read_ext):

actual = pd.read_excel(data, engine=engine)
tm.assert_frame_equal(expected, actual)

def test_excel_high_surrogate(self, engine):
# GH 23809
expected = pd.DataFrame(["\udc88"], columns=["Column1"])

# should not produce a segmentation violation
actual = pd.read_excel("high_surrogate.xlsx")
tm.assert_frame_equal(expected, actual)