Skip to content

BUG: Accept unicode quotechars again in pd.read_csv #14492

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v0.19.1.txt
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ Bug Fixes



- Bug in ``pd.read_csv`` for Python 2.x in which Unicode quote characters were no longer being respected (:issue:`14477`)
- Bug in localizing an ambiguous timezone when a boolean is passed (:issue:`14402`)
- Bug in ``TimedeltaIndex`` addition with a Datetime-like object where addition overflow in the negative direction was not being caught (:issue:`14068`, :issue:`14453`)

Expand Down
3 changes: 3 additions & 0 deletions pandas/io/parsers.py
Original file line number Diff line number Diff line change
Expand Up @@ -1759,6 +1759,9 @@ def __init__(self, f, **kwds):
self.delimiter = kwds['delimiter']

self.quotechar = kwds['quotechar']
if isinstance(self.quotechar, compat.text_type):
self.quotechar = str(self.quotechar)

self.escapechar = kwds['escapechar']
self.doublequote = kwds['doublequote']
self.skipinitialspace = kwds['skipinitialspace']
Expand Down
15 changes: 14 additions & 1 deletion pandas/io/tests/parser/quoting.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
import pandas.util.testing as tm

from pandas import DataFrame
from pandas.compat import StringIO
from pandas.compat import PY3, StringIO, u


class QuotingTests(object):
Expand Down Expand Up @@ -138,3 +138,16 @@ def test_double_quote(self):
result = self.read_csv(StringIO(data), quotechar='"',
doublequote=False)
tm.assert_frame_equal(result, expected)

def test_quotechar_unicode(self):
# See gh-14477
data = 'a\n1'
expected = DataFrame({'a': [1]})

result = self.read_csv(StringIO(data), quotechar=u('"'))
tm.assert_frame_equal(result, expected)

# Compared to Python 3.x, Python 2.x does not handle unicode well.
if PY3:
result = self.read_csv(StringIO(data), quotechar=u('\u0394'))
tm.assert_frame_equal(result, expected)
3 changes: 2 additions & 1 deletion pandas/parser.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -570,7 +570,8 @@ cdef class TextReader:
if not QUOTE_MINIMAL <= quoting <= QUOTE_NONE:
raise TypeError('bad "quoting" value')

if not isinstance(quote_char, (str, bytes)) and quote_char is not None:
if not isinstance(quote_char, (str, compat.text_type,
bytes)) and quote_char is not None:
dtype = type(quote_char).__name__
raise TypeError('"quotechar" must be string, '
'not {dtype}'.format(dtype=dtype))
Expand Down