Skip to content

BUG: Fix json_normalize throwing TypeError (#21536) #21540

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jun 22, 2018
Merged

BUG: Fix json_normalize throwing TypeError (#21536) #21540

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
cc4d5b2
BUG: Fix json_normalize throws TypeError with array of values and ec…
vuminhle Jun 19, 2018
b6a4dc5
Remove inplace in rename
vuminhle Jun 19, 2018
7aaa71d
Update release notes
vuminhle Jun 19, 2018
30aab15
Resolved merge conflict
vuminhle Jun 19, 2018
e28e0c7
Update json_normalize docstring
vuminhle Jun 20, 2018
b0d3a86
Remove unnecessary 1reindex_like1
vuminhle Jun 20, 2018
55e1022
Update release notes
vuminhle Jun 22, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/source/whatsnew/v0.23.2.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ Bug Fixes
**I/O**

- Bug in :func:`read_csv` that caused it to incorrectly raise an error when ``nrows=0``, ``low_memory=True``, and ``index_col`` was not ``None`` (:issue:`21141`)
-
- Bug in :func:`json_normalize` where passing an array of values and a `record_prefix` would raise a `TypeError` (:issue:`21536`)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use double backticks on record_prefix and TypeError

Copy link
Contributor

@jreback jreback Jun 20, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't the issue that this is for non-string columns? say this instead, the fact that it raised the TypeError is the bug here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will change to double backticks.
The issue is not because the column is non-string; it is because the array contains values (i.e., not objects or arrays). Suggestions are welcome.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how so? before concatting integers and strings would fail.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I misunderstood. When you mentioned columns, I thought they were the data columns, not the column names. Yes. The bug is caused by concatenating non-string column names, which happens when we pass an array of values (integers, strings, etc.). I can't think of other paths to trigger the bug.

My note explains how one can reproduce the bug. Looks like you want it to describe the reason.
I'm fine with either. How about this?

  • Bug in :func:json_normalize that caused it to incorrectly raise an error when concatenating non-string columns (:issue:21536)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

closer, though a user won't know what concatenating is (in this context), maybe just say something like

bug in formatting the record_prefix with integer columns

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Bug in :func:json_normalize when formatting the record_prefix with integer columns (:issue:21536)

-

**Plotting**
Expand Down
3 changes: 2 additions & 1 deletion pandas/io/json/normalize.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -259,7 +259,8 @@ def _recursive_extract(data, path, seen_meta, level=0):
result = DataFrame(records)

if record_prefix is not None:
result.rename(columns=lambda x: record_prefix + x, inplace=True)
result = result.rename(
columns=lambda x: "{p}{c}".format(p=record_prefix, c=x))

# Data types, a problem
for k, v in compat.iteritems(meta_vals):
Expand Down
6 changes: 6 additions & 0 deletions pandas/tests/io/json/test_normalize.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,12 @@ def test_simple_normalize_with_separator(self, deep_nested):
'country', 'states_name']).sort_values()
assert result.columns.sort_values().equals(expected)

def test_value_array_record_prefix(self):
# GH 21536
result = json_normalize({'A': [1, 2]}, 'A', record_prefix='Prefix.')
expected = DataFrame([[1], [2]], columns=['Prefix.0'])
tm.assert_frame_equal(result.reindex_like(expected), expected)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't think reindex_like is required here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Don't need it. I followed the code in test_simple_normalize_with_separator.
Btw, do we need docstring for test functions? None of the functions in this file has docstring.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope not for test functions. I was referring to the docstring for json_normalize

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah okay. Done. Thanks.


def test_more_deeply_nested(self, deep_nested):

result = json_normalize(deep_nested, ['states', 'cities'],
Expand Down