Jsonlines export error

[TevenLeScao]

Describe the bug

When exporting large datasets in jsonlines (c4 in my case) the created file has an error every 9999 lines: the 9999th and 10000th are concatenated, thus breaking the jsonlines format. This sounds like it is related to batching, which is by 10000 by default

Steps to reproduce the bug

This what I'm running:

in python:

from datasets import load_dataset
ptb = load_dataset("ptb_text_only")
ptb["train"].to_json("ptb.jsonl")

then out of python:

head -10000 ptb.jsonl

Expected results

Properly separated lines

Actual results

The last line is a concatenation of two lines

Environment info

• datasets version: 1.9.1.dev0
• Platform: Linux-5.4.0-1046-gcp-x86_64-with-Ubuntu-18.04-bionic
• Python version: 3.6.9
• PyArrow version: 4.0.1

[albertvillanova]

Thanks for reporting @TevenLeScao! I'm having a look...

[TevenLeScao]

(not sure what just happened on the assignations sorry)

[TevenLeScao]

For some reason this happens (both datasets version are on master) only on Python 3.6 and not Python 3.8.

[albertvillanova]

@TevenLeScao we are using pandas to serialize the dataset to JSON Lines. So it must be due to pandas. Could you please check the pandas version causing the issue?

[albertvillanova]

@TevenLeScao I have just checked it: this was a bug in pandas and it was fixed in version 1.2: pandas-dev/pandas#36898

[lhoestq]

Thanks ! I'm creating a PR

[albertvillanova]

Well I though it was me who has taken on this issue... 😅

[lhoestq]

Sorry, I was also talking to teven offline so I already had the PR ready before noticing x)

[albertvillanova]

I was also already working in my PR... Nevermind. Next time we should pay attention if there is somebody (self-)assigned to an issue and if he/she is still working on it before overtaking it... 😄