read_json with lines=True not using buff/cache memory

[louispotok]

I have a 3.2 GB json file that I am trying to read into pandas using pd.read_json(lines=True). When I run that, I get a MemoryError, even though my system has >12GB of available memory. This is Pandas version 0.20.2.

I'm on Ubuntu, and the free command shows >12GB of "Available" memory, most of which is "buff/cache".

I'm able to read the file into a dataframe by iterating over the file like so:

dfs = []
with open(fp, 'r') as f:
    while True:
        lines = list(itertools.islice(f, 1000))
        
        if lines:
            lines_str = ''.join(lines)
            dfs.append(pd.read_json(StringIO(lines_str), lines=True))
        else:
            break

df = pd.concat(dfs)

You'll notice that at the end of this I have the original data in memory twice (in the list and in the final df), but no problems.

It seems that pd.read_json with lines=True doesn't use the available memory, which looks to me like a bug.

[gfyoung]

@louispotok : that behavior does sound buggy to me, but before I label it as such, could you provide a minimal reproducible example for us?

[louispotok]

Happy to, but what exactly would constitute an example here? I can provide an example json file, but how would you suggest I reproduce the memory capacity and allocation on my machine?

[gfyoung]

I can provide an example json file, but how would you suggest I reproduce the memory capacity and allocation on my machine?

Just provide the smallest possible JSON file that causes this MemoryError to occur.

[jreback]

The lines=True impl is currently not designed this way. If you subsittue your soln into the current impl does it pass the test suite?

[jreback]

cc @aterrel

[louispotok]

@gfyoung I'm still not sure exactly what would be most helpful for you here.

I tried doing head -n 10 path/to/file | testing.py, where testing.py contains: df = pd.read_json(sys.stdin, lines=True), and then varying how many lines to pass.

Results:
Every million lines is about .8 GB, according to head -n 1000000 path/to/file | wc -c. And I did these each a few times in varying orders, always the same results.

• 1Million lines: success.
• 1.3M lines: success
• 2M lines: got "Killed" and also killed a watch in another terminal window, with message "unable to fork process: Cannot allocate memory"
• 3M lines: got "MemoryError" (I had a watch running here too, no problems at all)
• Full file: got "MemoryError"

[louispotok]

@jreback I think your question was for me, but I don't know how to do what you described. Are there instructions you could point me to?

[gfyoung]

1.3M lines: success

Yikes! That's a pretty massive file. That does certainly help us with regards to what we would need to do to reproduce this issue.

[gfyoung]

I think your question was for me, but I don't know how to do what you described.

Here is the documentation for making contributions to the repository. Essentially @jreback is asking if you could somehow incorporate your workaround in your issue description into the implementation of read_json, which you can find in pandas/io/json/json.py.

A quick glance there indicates what might be the issue: we're putting ALL of the lines into a list in memory! Your workaround might be able to address that.

[louispotok]

Thanks! I added it for one of the possible input types. You can see it here. It passes all the existing tests, and I'm now able to use it to load that file.

I think this is much slower than the previous implementation, and I don't know whether it can be extended to other input types. We could make it faster by increasing the chunk size or doing fewer concats, but at the cost of more memory usage.

[gfyoung]

We could make it faster by increasing the chunk size or doing fewer concats, but at the cost of more memory usage.

I think it would make sense to add such a parameter. We have it for read_csv. Try adding that and let us know how that works! This looks pretty good so far.

[louispotok]

Using the chunksize param in read_csv returns a TextFileReader, though, right? Won't that be confusing?

[gfyoung]

@louispotok : IMO, it would not because there's more confusion when people try to pass in the same parameters to one read_* function that they're used to passing in for another and found out they don't work or don't exist. Thus, you would be doing all read_json users a service by adding a similar parameter as exists in read_csv. 😄