Skip to content

BUG: to_json(lines=True) does not add a trailing newline #36888

Closed
@lleeoo

Description

@lleeoo
  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • (optional) I have confirmed this bug exists on the master branch of pandas.


Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

df = pd.DataFrame({'AAA': [4, 5, 6, 7],
                   'BBB': [10, 20, 30, 40],
                   'CCC': [100, 50, -30, -50]})
df.to_json('foo.jsonl', orient='records', lines=True)                                       
open('foo.jsonl').read()                                                                    

output (Notice - no newline after the last line):

'{"AAA":4,"BBB":10,"CCC":100}\n{"AAA":5,"BBB":20,"CCC":50}\n{"AAA":6,"BBB":30,"CCC":-30}\n{"AAA":7,"BBB":40,"CCC":-50}'

Problem description

When outputting JSON lines, i.e. lines=True, every record should end with a newline, including the last record. Otherwise, files cannot be concatenated, the number of records will be counted incorrectly by wc -l, etc.

Expected Output

There should be a newline at the end, i.e.:

'{"AAA":4,"BBB":10,"CCC":100}\n{"AAA":5,"BBB":20,"CCC":50}\n{"AAA":6,"BBB":30,"CCC":-30}\n{"AAA":7,"BBB":40,"CCC":-50}\n'

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit : d9fff27 python : 3.7.7.final.0 python-bits : 64 OS : Linux OS-release : 3.10.0-1127.19.1.el7.x86_64 Version : #1 SMP Tue Aug 11 19:12:04 EDT 2020 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8

pandas : 1.1.0
numpy : 1.19.1
pytz : 2020.1
dateutil : 2.8.1
pip : 20.2.2
setuptools : 49.6.0.post20200814
Cython : None
pytest : None
hypothesis : None
sphinx : 3.2.1
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.17.0
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : 0.8.0
fastparquet : 0.4.1
gcsfs : None
matplotlib : 3.3.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 0.15.1
pytables : None
pyxlsb : None
s3fs : None
scipy : 1.5.2
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : 1.2.0
xlwt : None
numba : 0.50.1

Metadata

Metadata

Assignees

Labels

BugIO JSONread_json, to_json, json_normalize

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions