BUG: Fix json_normalize throwing TypeError (#21536) #21540

vuminhle · 2018-06-19T09:09:28Z

Fix json_normalize throwing TypeError with array of values and record_prefix (#21536)

closes json_normalize throws TypeError with array of values and record_prefix #21536
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

codecov · 2018-06-19T11:17:48Z

Codecov Report

Merging #21540 into master will decrease coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #21540      +/-   ##
==========================================
- Coverage   91.91%    91.9%   -0.02%     
==========================================
  Files         153      153              
  Lines       49546    49549       +3     
==========================================
- Hits        45542    45539       -3     
- Misses       4004     4010       +6

Flag	Coverage Δ
#multiple	`90.3% <100%> (-0.02%)`	⬇️
#single	`41.78% <0%> (-0.03%)`	⬇️

Impacted Files	Coverage Δ
pandas/io/json/normalize.py	`96.87% <100%> (ø)`	⬆️
pandas/util/testing.py	`85.27% <0%> (-0.7%)`	⬇️
pandas/core/arrays/categorical.py	`95.63% <0%> (-0.06%)`	⬇️
pandas/core/indexing.py	`93.37% <0%> (-0.05%)`	⬇️
pandas/tseries/offsets.py	`97.16% <0%> (-0.04%)`	⬇️
pandas/core/indexes/multi.py	`94.97% <0%> (-0.01%)`	⬇️
pandas/core/indexes/datetimes.py	`95.66% <0%> (ø)`	⬆️
pandas/core/indexes/timedeltas.py	`91.24% <0%> (ø)`	⬆️
pandas/core/indexes/base.py	`96.62% <0%> (ø)`	⬆️
pandas/core/sorting.py	`98.2% <0%> (ø)`	⬆️
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5fbb683...55e1022. Read the comment docs.

jreback

minor comment can you add a note in 0.23.2, bug fixes in io section.

jreback · 2018-06-19T11:16:07Z

pandas/tests/io/json/test_normalize.py

@@ -123,6 +123,12 @@ def test_simple_normalize_with_separator(self, deep_nested):
                          'country', 'states_name']).sort_values()
        assert result.columns.sort_values().equals(expected)

+    def test_value_array_record_prefix(self):
+        #GH 21536


space after the #

jreback · 2018-06-19T11:18:05Z

pandas/io/json/normalize.py

@@ -259,7 +259,8 @@ def _recursive_extract(data, path, seen_meta, level=0):
    result = DataFrame(records)

    if record_prefix is not None:
-        result.rename(columns=lambda x: record_prefix + x, inplace=True)
+        result.rename(columns=lambda x: "{p}{c}".format(p=record_prefix, c=x),


can you change to

result = result.rename(...)

we don't use inplace in library code

vuminhle · 2018-06-19T11:37:00Z

@jreback : How do I add a note? Thanks.

WillAyd · 2018-06-19T15:26:03Z

@vuminhle you can add a note to doc/source/whatsnew/v0.23.2.txt. I think this is fine under "Bug Fixes" -> "Other"

jreback · 2018-06-19T16:19:24Z

should be in IO

vuminhle · 2018-06-19T23:46:45Z

@WillAyd @jreback I added a note. Thanks for the instruction.

WillAyd

Can you also update the docstring for this function and add an example of what it does?

WillAyd · 2018-06-20T00:00:05Z

pandas/tests/io/json/test_normalize.py

+        # GH 21536
+        result = json_normalize({'A': [1, 2]}, 'A', record_prefix='Prefix.')
+        expected = DataFrame([[1], [2]], columns=['Prefix.0'])
+        tm.assert_frame_equal(result.reindex_like(expected), expected)


Don't think reindex_like is required here?

Yes. Don't need it. I followed the code in test_simple_normalize_with_separator.
Btw, do we need docstring for test functions? None of the functions in this file has docstring.

Nope not for test functions. I was referring to the docstring for json_normalize

Ah okay. Done. Thanks.

jreback · 2018-06-20T10:08:00Z

doc/source/whatsnew/v0.23.2.txt

@@ -65,7 +65,7 @@ Bug Fixes
 **I/O**

 - Bug in :func:`read_csv` that caused it to incorrectly raise an error when ``nrows=0``, ``low_memory=True``, and ``index_col`` was not ``None`` (:issue:`21141`)
-
+- Bug in :func:`json_normalize` where passing an array of values and a `record_prefix` would raise a `TypeError` (:issue:`21536`)


use double backticks on record_prefix and TypeError

isn't the issue that this is for non-string columns? say this instead, the fact that it raised the TypeError is the bug here

Will change to double backticks.
The issue is not because the column is non-string; it is because the array contains values (i.e., not objects or arrays). Suggestions are welcome.

how so? before concatting integers and strings would fail.

I misunderstood. When you mentioned columns, I thought they were the data columns, not the column names. Yes. The bug is caused by concatenating non-string column names, which happens when we pass an array of values (integers, strings, etc.). I can't think of other paths to trigger the bug.

My note explains how one can reproduce the bug. Looks like you want it to describe the reason.
I'm fine with either. How about this?

Bug in :func:json_normalize that caused it to incorrectly raise an error when concatenating non-string columns (:issue:21536)

closer, though a user won't know what concatenating is (in this context), maybe just say something like

bug in formatting the record_prefix with integer columns

Bug in :func:json_normalize when formatting the record_prefix with integer columns (:issue:21536)

jreback · 2018-06-21T10:37:08Z

doc/source/whatsnew/v0.23.2.txt

@@ -65,7 +65,7 @@ Bug Fixes
 **I/O**

 - Bug in :func:`read_csv` that caused it to incorrectly raise an error when ``nrows=0``, ``low_memory=True``, and ``index_col`` was not ``None`` (:issue:`21141`)
-
+- Bug in :func:`json_normalize` where passing an array of values and a `record_prefix` would raise a `TypeError` (:issue:`21536`)


closer, though a user won't know what concatenating is (in this context), maybe just say something like

bug in formatting the record_prefix with integer columns

jreback · 2018-06-22T23:07:25Z

thanks @vuminhle

(cherry picked from commit 5fdaa97)

…-dev#21540)

BUG: Fix json_normalize throws TypeError with array of values and eco…

cc4d5b2

…rd_prefix (pandas-dev#21536)

vuminhle force-pushed the fix-json-normalize-type-error branch from 035af20 to cc4d5b2 Compare June 19, 2018 11:17

jreback requested changes Jun 19, 2018

View reviewed changes

jreback added Bug IO JSON read_json, to_json, json_normalize labels Jun 19, 2018

Remove inplace in rename

b6a4dc5

vuminhle added 2 commits June 20, 2018 06:39

Update release notes

7aaa71d

Resolved merge conflict

30aab15

WillAyd requested changes Jun 20, 2018

View reviewed changes

vuminhle added 2 commits June 20, 2018 07:59

Update json_normalize docstring

e28e0c7

Remove unnecessary 1reindex_like1

b0d3a86

jreback requested changes Jun 20, 2018

View reviewed changes

jreback requested changes Jun 21, 2018

View reviewed changes

Update release notes

55e1022

WillAyd approved these changes Jun 22, 2018

View reviewed changes

jreback added this to the 0.23.2 milestone Jun 22, 2018

jreback approved these changes Jun 22, 2018

View reviewed changes

jreback merged commit 5fdaa97 into pandas-dev:master Jun 22, 2018

jorisvandenbossche added Needs Backport and removed Needs Backport labels Jun 26, 2018

jorisvandenbossche pushed a commit that referenced this pull request Jun 29, 2018

BUG: Fix json_normalize throwing TypeError (#21536) (#21540)

d096a7b

(cherry picked from commit 5fdaa97)

jorisvandenbossche pushed a commit that referenced this pull request Jul 2, 2018

BUG: Fix json_normalize throwing TypeError (#21536) (#21540)

cf0a55f

(cherry picked from commit 5fdaa97)

Sup3rGeo pushed a commit to Sup3rGeo/pandas that referenced this pull request Oct 1, 2018

BUG: Fix json_normalize throwing TypeError (pandas-dev#21536) (pandas…

6b33c63

…-dev#21540)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Fix json_normalize throwing TypeError (#21536) #21540

BUG: Fix json_normalize throwing TypeError (#21536) #21540

vuminhle commented Jun 19, 2018 •

edited

Loading

codecov bot commented Jun 19, 2018 •

edited

Loading

jreback left a comment

jreback Jun 19, 2018

jreback Jun 19, 2018

vuminhle commented Jun 19, 2018

WillAyd commented Jun 19, 2018

jreback commented Jun 19, 2018

vuminhle commented Jun 19, 2018

WillAyd left a comment

WillAyd Jun 20, 2018

vuminhle Jun 20, 2018

WillAyd Jun 20, 2018

vuminhle Jun 20, 2018

jreback Jun 20, 2018

jreback Jun 20, 2018 •

edited

Loading

vuminhle Jun 20, 2018

jreback Jun 21, 2018

vuminhle Jun 21, 2018

jreback Jun 21, 2018

vuminhle Jun 21, 2018

jreback Jun 21, 2018

jreback commented Jun 22, 2018

BUG: Fix json_normalize throwing TypeError (#21536) #21540

BUG: Fix json_normalize throwing TypeError (#21536) #21540

Conversation

vuminhle commented Jun 19, 2018 • edited Loading

codecov bot commented Jun 19, 2018 • edited Loading

Codecov Report

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vuminhle commented Jun 19, 2018

WillAyd commented Jun 19, 2018

jreback commented Jun 19, 2018

vuminhle commented Jun 19, 2018

WillAyd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback Jun 20, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Jun 22, 2018

vuminhle commented Jun 19, 2018 •

edited

Loading

codecov bot commented Jun 19, 2018 •

edited

Loading

jreback Jun 20, 2018 •

edited

Loading