Skip to content

BUG: Integer Overflow in read_json with big number in string #30329

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jan 24, 2020
Merged

BUG: Integer Overflow in read_json with big number in string #30329

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
0f784de
BUG: Integer Overflow in read_json with big number in string
rohitkg98 Dec 18, 2019
68cb4b0
CLN: Removed parametrize from test because single test case
rohitkg98 Dec 18, 2019
7ac3f1b
CLN: rectified test and added issue number
rohitkg98 Dec 19, 2019
0d6f2f4
Merge branch 'master' of github.com:rohitkg98/pandas into handle_json…
Jan 21, 2020
ad590d3
CLN: Added bug fix to release notes
Jan 21, 2020
351a419
Merge branch 'master' of https://github.com/pandas-dev/pandas into ha…
Jan 21, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/source/whatsnew/v1.1.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ MultiIndex

I/O
^^^

- Bug in :meth:`read_json` where integer overflow was occuring when json contains big number strings. (:issue:`30320`)
-
-

Expand Down
2 changes: 1 addition & 1 deletion pandas/io/json/_json.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -942,7 +942,7 @@ def _try_convert_data(self, name, data, use_dtypes=True, convert_dates=True):
if (new_data == data).all():
data = new_data
result = True
except (TypeError, ValueError):
except (TypeError, ValueError, OverflowError):
pass

# coerce ints to 64
Expand Down
7 changes: 7 additions & 0 deletions pandas/tests/io/json/test_pandas.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1640,3 +1640,10 @@ def test_deprecate_numpy_argument_read_json(self):
with tm.assert_produces_warning(FutureWarning):
result = read_json(expected.to_json(), numpy=True)
tm.assert_frame_equal(result, expected)

def test_frame_int_overflow(self):
# GH 30320
encoded_json = json.dumps([{"col": "31900441201190696999"}, {"col": "Text"}])
expected = DataFrame({"col": ["31900441201190696999", "Text"]})
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC from the issue, the expected output is an object column containing a number and a string?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They have written the number in a float form. I'll look into generating an object.

3.190044e+19

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy with the string not being coerced, although I think the issue may need further discussion. see #30320 (comment).

What happens now when the encoded_json is an int? Maybe we should test for this too here?

Copy link
Contributor Author

@rohitkg98 rohitkg98 Jan 21, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@simonjayhawkins Please see and provide guidance on this comment

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@simonjayhawkins Please see and provide guidance on this comment

not sure about this. @WillAyd?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know the answer either. Probably needs to be done in C but if you see a way to make it happen in the error handler works as well

result = read_json(encoded_json)
tm.assert_frame_equal(result, expected)