Skip to content

Suppress UnicodeEncodeError when executing to_csv method #27750

Closed
@shigemk2

Description

@shigemk2

Code Sample, a copy-pastable example if possible

# error pattern
import pandas as pd

unicode_data = [["key", "\u070a"]]
df = pd.DataFrame(unicode_data)
df.to_csv("./test.csv", encoding="cp932") # UnicodeEncodeError: 'cp932' codec can't encode character '\u070a' in position 6: illegal multibyte sequence
# good pattern
import pandas as pd

unicode_data = [["key", "\u070a"]]
df = pd.DataFrame(unicode_data)
with open("./test.csv", mode="w", encoding="cp932", errors="ignore") as f:
    df.to_csv(f)

Problem description

UnicodeEncodeError occurs when executing to_csv with eoncode parameter SHIFT-JIS or cp932.
We are able to avoid this error using with open(good pattern), this code is redundant.
So I want to suppress UnicodeEncodeError with to_csv's parameter.

Expected Output

# good pattern
import pandas as pd

unicode_data = [["key", "\u070a"]]
df = pd.DataFrame(unicode_data)
df.to_csv("./test.csv", encoding="cp932", ignore_error=True)

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.7.2.final.0
python-bits : 64
OS : Darwin
OS-release : 18.6.0
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : ja_JP.UTF-8
LOCALE : ja_JP.UTF-8

pandas : 0.25.0
numpy : 1.16.2
pytz : 2018.9
dateutil : 2.8.0
pip : 19.0.3
setuptools : 40.8.0
Cython : None
pytest : 4.3.1
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : 2.6.1
pandas_gbq : None
pyarrow : None
pytables : None
s3fs : None
scipy : None
sqlalchemy : 1.3.1
tables : None
xarray : None
xlrd : 1.2.0
xlwt : None
xlsxwriter : None

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions