Unexpected ValueError when using a json reader to read file from disk using chunksize

[vlbrown]

Sent to pandas-dev@python.org on June 23 (vlb@c....com); posted here as recommended by Marc Garcia.

Code Sample, a copy-pastable example if possible

path = 'file://localhost/Users/vlb/Learn/DSC_Intro/'
filename = path + 'yelp_dataset/review_test.json'

# read the entire file -- this works
reviews = pd.read_json(filename, lines=True)
reviews.info()

# create a reader to read in chunks -- this part seems to work
review_reader = pd.read_json(StringIO(filename), lines=True, chunksize=1)
type(review_reader)

# But trying to read from the reader throws an error
# ValueError: Unexpected character found when decoding 'false'
for chunk in review_reader:
    print(chunk)

Data Samples

Either or both of the following records can be used

{"review_id":"rEITo90tpyKmEfNDp3Ou3A","user_id":"6Fz_nus_OG4gar721OKgZA","business_id":"6lj2BJ4tJeu7db5asGHQ4w","stars":5.0,"useful":0,"funny":0,"cool":0,"text":"We've been a huge Slim's fan since they opened one up in Texas about two years ago when we used to live there. This place never disappoints. They even have great salads and grilled chicken. Plus they have fresh brewed sweet tea, it's the best!","date":"2017-05-26 01:23:19"}

{"review_id":"Amo5gZBvCuPc_tZNpHwtsA","user_id":"DzZ7piLBF-WsJxqosfJgtA","business_id":"qx6WhZ42eDKmBchZDax4dQ","stars":5.0,"useful":1,"funny":0,"cool":0,"text":"Our family LOVES the food here. Quick, friendly, delicious, and a great restaurant to take kids to. 5 stars!","date":"2017-03-27 01:14:37"}

Problem description

Problem description

I am working a tutorial that uses a JSON data file from Yelp. The file is huge, so it needs to be read in chunks.

I get an unexpected error: ValueError: Unexpected character found when decoding 'false'

For testing purposes, I have reduced the dataset to a much smaller file with only 3 lines. I can reproduce the error with that file as well as with a file containing only one (any one) of the three lines.

Note that if I simply read in the entire (test) data set in one go, that works. It's only when I create a reader and try to review the chunks that I get the error.

Expected Output

No errors. A chunk should print.

If there is an error, it should be less opaque than "Unexpected character found when decoding 'false'".

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line]

INSTALLED VERSIONS

commit: None
python: 3.7.3.final.0
python-bits: 64
OS: Darwin
OS-release: 15.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.2
pytest: 4.3.1
pip: 19.0.3
setuptools: 40.8.0
Cython: 0.29.6
numpy: 1.16.2
scipy: 1.2.1
pyarrow: None
xarray: None
IPython: 7.4.0
sphinx: 1.8.5
patsy: 0.5.1
dateutil: 2.8.0
pytz: 2018.9
blosc: None
bottleneck: 1.2.1
tables: 3.5.1
numexpr: 2.6.9
feather: None
matplotlib: 3.0.3
openpyxl: 2.6.1
xlrd: 1.2.0
xlwt: 1.3.0
xlsxwriter: 1.1.5
lxml.etree: 4.3.2
bs4: 4.7.1
html5lib: 1.0.1
sqlalchemy: 1.3.1
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: 0.7.0
gcsfs: None

[WillAyd]

I think the problem is that you have lines=True but you are not escaping the newlines that appear in your data.

To illustrate, this fails:

>>> data = """
{"review_id":"rEITo90tpyKmEfNDp3Ou3A","user_id":"6Fz_nus_OG4gar721OKgZA","business_id":"6lj2BJ4tJeu7db5asGHQ4w","stars":5.0,"useful":0,"funny":0,"cool":0,"text":"We've been a huge Slim's fan since they opened one up in Texas about two years ago when we used to live there. This place never disappoints. They even have great salads and grilled chicken. Plus they have fresh brewed sweet tea, it's the best!","date":"2017-05-26 01:23:19"}
{"review_id":"4bUyL7lzoWzDZaJETAKREg","user_id":"_N7Ndn29bpll_961oPeEfw","business_id":"y-Iw6dZflNix4BdwIyTNGA","stars":3.0,"useful":0,"funny":0,"cool":0,"text":"Good selection of classes of beers and mains. I've been here twice.\n\nFirst time I had the fried chicken. It was delicious, but be warned, extremely salty. I couldn't even finish the last piece of chicken after experiencing a salt overload.\n\nSecond time we came on a wednesday. We didn't know it was BBQ night, where they have a completely different menu, and don't offer anything from their original vegetarian-friendly menu. This menu has one vegetarian-friendly option - an eggplant sandwich. The vegetarian in my party said it was awful. Also, on BBQ night you choose 2 sides. Except they were out of all their sides except 2 - fries and potato salad. I can't say I was thrilled to have carb heavy sides with my carb heavy main. How do you run out of sides so early in the evening?\n\nService not so great.\n\nI'd avoid coming here on wednesdays.","date":"2014-06-27 21:19:23"}
{"review_id":"Amo5gZBvCuPc_tZNpHwtsA","user_id":"DzZ7piLBF-WsJxqosfJgtA","business_id":"qx6WhZ42eDKmBchZDax4dQ","stars":5.0,"useful":1,"funny":0,"cool":0,"text":"Our family LOVES the food here. Quick, friendly, delicious, and a great restaurant to take kids to. 5 stars!","date":"2017-03-27 01:14:37"}"""
>>> list(pd.read_json(data, lines=True, chunksize=1))

Whereas this does not:

>>> data = r"""
{"review_id":"rEITo90tpyKmEfNDp3Ou3A","user_id":"6Fz_nus_OG4gar721OKgZA","business_id":"6lj2BJ4tJeu7db5asGHQ4w","stars":5.0,"useful":0,"funny":0,"cool":0,"text":"We've been a huge Slim's fan since they opened one up in Texas about two years ago when we used to live there. This place never disappoints. They even have great salads and grilled chicken. Plus they have fresh brewed sweet tea, it's the best!","date":"2017-05-26 01:23:19"}
{"review_id":"4bUyL7lzoWzDZaJETAKREg","user_id":"_N7Ndn29bpll_961oPeEfw","business_id":"y-Iw6dZflNix4BdwIyTNGA","stars":3.0,"useful":0,"funny":0,"cool":0,"text":"Good selection of classes of beers and mains. I've been here twice.\n\nFirst time I had the fried chicken. It was delicious, but be warned, extremely salty. I couldn't even finish the last piece of chicken after experiencing a salt overload.\n\nSecond time we came on a wednesday. We didn't know it was BBQ night, where they have a completely different menu, and don't offer anything from their original vegetarian-friendly menu. This menu has one vegetarian-friendly option - an eggplant sandwich. The vegetarian in my party said it was awful. Also, on BBQ night you choose 2 sides. Except they were out of all their sides except 2 - fries and potato salad. I can't say I was thrilled to have carb heavy sides with my carb heavy main. How do you run out of sides so early in the evening?\n\nService not so great.\n\nI'd avoid coming here on wednesdays.","date":"2014-06-27 21:19:23"}
{"review_id":"Amo5gZBvCuPc_tZNpHwtsA","user_id":"DzZ7piLBF-WsJxqosfJgtA","business_id":"qx6WhZ42eDKmBchZDax4dQ","stars":5.0,"useful":1,"funny":0,"cool":0,"text":"Our family LOVES the food here. Quick, friendly, delicious, and a great restaurant to take kids to. 5 stars!","date":"2017-03-27 01:14:37"}"""
>>> list(pd.read_json(data, lines=True, chunksize=1))

So tricky but closing as not a pandas bug.

[vlbrown]

OK, but.... 1) Why does chunksize fail but reading without chunksize works perfectly? 2) shouldn't quoted \n be considered "escaped"? 3) I still think there's a pandas issue in that the error message is not helpful. 4) How do I escape the newlines in the reader? (I can't put a 6 million-line file into """)

…

On Jun 24, 2019, at 18:31 , William Ayd ***@***.***> wrote: I think the problem is that you have lines=True but you are not escaping the newlines that appear in your data. To illustrate, this fails: >>> data = """ {"review_id":"rEITo90tpyKmEfNDp3Ou3A","user_id":"6Fz_nus_OG4gar721OKgZA","business_id":"6lj2BJ4tJeu7db5asGHQ4w","stars":5.0,"useful":0,"funny":0,"cool":0,"text":"We've been a huge Slim's fan since they opened one up in Texas about two years ago when we used to live there. This place never disappoints. They even have great salads and grilled chicken. Plus they have fresh brewed sweet tea, it's the best!","date":"2017-05-26 01:23:19"} {"review_id":"4bUyL7lzoWzDZaJETAKREg","user_id":"_N7Ndn29bpll_961oPeEfw","business_id":"y-Iw6dZflNix4BdwIyTNGA","stars":3.0,"useful":0,"funny":0,"cool":0,"text":"Good selection of classes of beers and mains. I've been here twice.\n\nFirst time I had the fried chicken. It was delicious, but be warned, extremely salty. I couldn't even finish the last piece of chicken after experiencing a salt overload.\n\nSecond time we came on a wednesday. We didn't know it was BBQ night, where they have a completely different menu, and don't offer anything from their original vegetarian-friendly menu. This menu has one vegetarian-friendly option - an eggplant sandwich. The vegetarian in my party said it was awful. Also, on BBQ night you choose 2 sides. Except they were out of all their sides except 2 - fries and potato salad. I can't say I was thrilled to have carb heavy sides with my carb heavy main. How do you run out of sides so early in the evening?\n\nService not so great.\n\nI'd avoid coming here on wednesdays.","date":"2014-06-27 21:19:23"} {"review_id":"Amo5gZBvCuPc_tZNpHwtsA","user_id":"DzZ7piLBF-WsJxqosfJgtA","business_id":"qx6WhZ42eDKmBchZDax4dQ","stars":5.0,"useful":1,"funny":0,"cool":0,"text":"Our family LOVES the food here. Quick, friendly, delicious, and a great restaurant to take kids to. 5 stars!","date":"2017-03-27 01:14:37"}""" >>> list(pd.read_json(data, lines=True, chunksize=1)) Whereas this does not: >>> data = r""" {"review_id":"rEITo90tpyKmEfNDp3Ou3A","user_id":"6Fz_nus_OG4gar721OKgZA","business_id":"6lj2BJ4tJeu7db5asGHQ4w","stars":5.0,"useful":0,"funny":0,"cool":0,"text":"We've been a huge Slim's fan since they opened one up in Texas about two years ago when we used to live there. This place never disappoints. They even have great salads and grilled chicken. Plus they have fresh brewed sweet tea, it's the best!","date":"2017-05-26 01:23:19"} {"review_id":"4bUyL7lzoWzDZaJETAKREg","user_id":"_N7Ndn29bpll_961oPeEfw","business_id":"y-Iw6dZflNix4BdwIyTNGA","stars":3.0,"useful":0,"funny":0,"cool":0,"text":"Good selection of classes of beers and mains. I've been here twice.\n\nFirst time I had the fried chicken. It was delicious, but be warned, extremely salty. I couldn't even finish the last piece of chicken after experiencing a salt overload.\n\nSecond time we came on a wednesday. We didn't know it was BBQ night, where they have a completely different menu, and don't offer anything from their original vegetarian-friendly menu. This menu has one vegetarian-friendly option - an eggplant sandwich. The vegetarian in my party said it was awful. Also, on BBQ night you choose 2 sides. Except they were out of all their sides except 2 - fries and potato salad. I can't say I was thrilled to have carb heavy sides with my carb heavy main. How do you run out of sides so early in the evening?\n\nService not so great.\n\nI'd avoid coming here on wednesdays.","date":"2014-06-27 21:19:23"} {"review_id":"Amo5gZBvCuPc_tZNpHwtsA","user_id":"DzZ7piLBF-WsJxqosfJgtA","business_id":"qx6WhZ42eDKmBchZDax4dQ","stars":5.0,"useful":1,"funny":0,"cool":0,"text":"Our family LOVES the food here. Quick, friendly, delicious, and a great restaurant to take kids to. 5 stars!","date":"2017-03-27 01:14:37"}""" >>> list(pd.read_json(data, lines=True, chunksize=1)) So tricky but closing as not a pandas bug. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#27022?email_source=notifications&email_token=AAFLU3H72L5SYRXSNSZEBD3P4FYOBA5CNFSM4H3BAMEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYOV3RI#issuecomment-505241029>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFLU3ARVY25JRGJ74IILL3P4FYOBANCNFSM4H3BAMEA>.

-- Vicki Vicki Brown cfcl.com/vlb

[vlbrown]

"For testing purposes, I have reduced the dataset to a much smaller file with only 3 lines. I can reproduce the error with that file as well as with a file containing only one (any one) of the three lines."

Of those three records, only one has embedded "\n". S, I removed that record from my test dataset.

I get the same failure when my data file is only one JSON record (no embedded newlines) or two records (neither of which have embedded newlines) and that dataset passes your

data = """
<<<

test

data27022.txt

[vlbrown]

On Jun 24, 2019, at 18:31 , William Ayd ***@***.***> wrote: Closed #27022 <#27022>.

Please reopen. Removing the one record that includes embedded "\n' does not change the outcome. I still get the error, and this data passes your test:

>> data = """

{"review_id":"rEITo90tpyKmEfNDp3Ou3A","user_id":"6Fz_nus_OG4gar721OKgZA","business_id":"6lj2BJ4tJeu7db5asGHQ4w","stars":5.0,"useful":0,"funny":0,"cool":0,"text":"We've been a huge Slim's fan since they opened one up in Texas about two years ago when we used to live there. This place never disappoints. They even have great salads and grilled chicken. Plus they have fresh brewed sweet tea, it's the best!","date":"2017-05-26 01:23:19"} {"review_id":"Amo5gZBvCuPc_tZNpHwtsA","user_id":"DzZ7piLBF-WsJxqosfJgtA","business_id":"qx6WhZ42eDKmBchZDax4dQ","stars":5.0,"useful":1,"funny":0,"cool":0,"text":"Our family LOVES the food here. Quick, friendly, delicious, and a great restaurant to take kids to. 5 stars!","date":"2017-03-27 01:14:37"}"""

…

>> list(pd.read_json(data, lines=True, chunksize=1))

-- Vicki Vicki Brown cfcl.com/vlb

[TomAugspurger]

What's the minimal example here? Can we update the original post with it?

[vlbrown]

What's the minimal example here? Can we update the original post with it?

Done.

[TomAugspurger]

Do you have a reproducer that doesn't rely on an external dataset? (in case the original one goes away) Ideally something small, so it's easy to tell if it's been fixed. http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports

…

On Wed, Jun 26, 2019 at 3:10 PM Vicki Brown ***@***.***> wrote: What's the minimal example here? Can we update the original post with it? Done. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#27022?email_source=notifications&email_token=AAKAOIS2LPZ2O47YRQEEHI3P4PELJA5CNFSM4H3BAMEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYUVTBY#issuecomment-506026375>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKAOIRNRJDIJY2UUJ3YBIDP4PELJANCNFSM4H3BAMEA> .

[vlbrown]

On Jun 26, 2019, at 13:52 , Tom Augspurger ***@***.***> wrote: Do you have a reproducer that doesn't rely on an external dataset? (in case the original one goes away) Ideally something small, so it's easy to tell if it's been fixed.

I don't understand. It only seems to fail with an external dataset (i.e. data read in from a file). However, that file can be recreated at any time using either or both of the JSON records that are pasted into the issue report. If that doesn't work, I'm afraid I don't understand what you're asking.

…

-- Vicki Vicki Brown cfcl.com/vlb

[TomAugspurger]

The following doesn't raise.

import pandas as pd
import io


f = io.StringIO("""\
{"review_id":"Amo5gZBvCuPc_tZNpHwtsA","user_id":"DzZ7piLBF-WsJxqosfJgtA","business_id":"qx6WhZ42eDKmBchZDax4dQ","stars":5.0,"useful":1,"funny":0,"cool":0,"text":"Our family LOVES the food here. Quick, friendly, delicious, and a great restaurant to take kids to. 5 stars!","date":"2017-03-27 01:14:37"}""")

reader = pd.read_json(f, lines=True, chunksize=1)
for chunk in reader:
    print(chunk)

I read

Either or both of the following records can be used

As saying that the pandas reader had issues with those lines when chunksize=1 and lines=True. Am I misunderstanding?

[vlbrown]

As far as I have been able to work out, the issue appears if the data is read in from a file (It only seems to fail with an external dataset). Embedding the data records into the code as strings doesn't seem to trip whatever is going wrong. (Also, given that the actual dataset is enormous, knowing that there is no problem if the data is embedded with the code doesn't help.) Please run my test case as is; you're making changes that affect its behaviour. path = 'file://localhost/Users/vlb/Learn/DSC_Intro/' filename = path + 'yelp_dataset/review_test.json' # read the entire file -- this works reviews = pd.read_json(filename, lines=True) reviews.info() # create a reader to read in chunks -- this part seems to work review_reader = pd.read_json(StringIO(filename), lines=True, chunksize=1) type(review_reader) # But trying to read from the reader throws an error # ValueError: Unexpected character found when decoding 'false' for chunk in review_reader: print(chunk)

…

On Jun 26, 2019, at 14:52 , Tom Augspurger ***@***.***> wrote: The following doesn't raise. import pandas as pd import io f = io.StringIO("""\ {"review_id":"Amo5gZBvCuPc_tZNpHwtsA","user_id":"DzZ7piLBF-WsJxqosfJgtA","business_id":"qx6WhZ42eDKmBchZDax4dQ","stars":5.0,"useful":1,"funny":0,"cool":0,"text":"Our family LOVES the food here. Quick, friendly, delicious, and a great restaurant to take kids to. 5 stars!","date":"2017-03-27 01:14:37"}""") reader = pd.read_json(f, lines=True, chunksize=1) for chunk in reader: print(chunk) I read Either or both of the following records can be used As saying that the pandas reader had issues with those lines when chunksize=1 and lines=True. Am I misunderstanding? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#27022?email_source=notifications&email_token=AAFLU3ESEBJCVDXTYCJ3J2DP4PQKNA5CNFSM4H3BAMEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYU5RRY#issuecomment-506058951>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFLU3DRDDREJWMV37PAL3DP4PQKNANCNFSM4H3BAMEA>.

-- Vicki Vicki Brown cfcl.com/vlb