Description
Pandas version checks
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd
a = {'abc':[1,2,3],'xy （z）':[4,5,6]}
df = pd.DataFrame(a)
query = "`xy （z）` == 2"
new_df = df.query(query)Issue Description
A error has happened as follow.
Traceback (most recent call last):
File ~/pandas-check/.venv/lib/python3.8/site-packages/IPython/core/interactiveshell.py:3433 in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
Cell In [1], line 7
new_df = df.query(query)
File ~/pandas-check/.venv/lib64/python3.8/site-packages/pandas/util/_decorators.py:331 in wrapper
return func(*args, **kwargs)
File ~/pandas-check/.venv/lib64/python3.8/site-packages/pandas/core/frame.py:4463 in query
res = self.eval(expr, **kwargs)
File ~/pandas-check/.venv/lib64/python3.8/site-packages/pandas/util/_decorators.py:331 in wrapper
return func(*args, **kwargs)
File ~/pandas-check/.venv/lib64/python3.8/site-packages/pandas/core/frame.py:4601 in eval
return _eval(expr, inplace=inplace, **kwargs)
File ~/pandas-check/.venv/lib64/python3.8/site-packages/pandas/core/computation/eval.py:353 in eval
parsed_expr = Expr(expr, engine=engine, parser=parser, env=env)
File ~/pandas-check/.venv/lib64/python3.8/site-packages/pandas/core/computation/expr.py:813 in __init__
self.terms = self.parse()
File ~/pandas-check/.venv/lib64/python3.8/site-packages/pandas/core/computation/expr.py:832 in parse
return self._visitor.visit(self.expr)
File ~/pandas-check/.venv/lib64/python3.8/site-packages/pandas/core/computation/expr.py:405 in visit
clean = self.preparser(node)
File ~/pandas-check/.venv/lib64/python3.8/site-packages/pandas/core/computation/expr.py:166 in _preparse
return tokenize.untokenize(f(x) for x in tokenize_string(source))
File /opt/rh/rh-python38/root/usr/lib64/python3.8/tokenize.py:278 in untokenize
out = ut.untokenize(iterable)
File /opt/rh/rh-python38/root/usr/lib64/python3.8/tokenize.py:187 in untokenize
for t in it:
File ~/pandas-check/.venv/lib64/python3.8/site-packages/pandas/core/computation/expr.py:166 in <genexpr>
return tokenize.untokenize(f(x) for x in tokenize_string(source))
File ~/pandas-check/.venv/lib64/python3.8/site-packages/pandas/core/computation/expr.py:124 in <lambda>
return lambda *args, **kwargs: f(g(*args, **kwargs))
File ~/pandas-check/.venv/lib64/python3.8/site-packages/pandas/core/computation/parsing.py:92 in clean_backtick_quoted_toks
return tokenize.NAME, create_valid_python_identifier(tokval)
File ~/pandas-check/.venv/lib64/python3.8/site-packages/pandas/core/computation/parsing.py:65 in create_valid_python_identifier
raise SyntaxError(f"Could not convert '{name}' to a valid Python identifier.")
File <string>
SyntaxError: Could not convert 'BACKTICK_QUOTED_STRING_xy_（z）' to a valid Python identifier.
Expected Behavior
We expect the rows to be selected appropriately by the query method without error.
This error would be related to #28576.
Indeed, this does not happens if the parenthesis is 1-bytes code.
Installed Versions
INSTALLED VERSIONS
commit : 91111fd
python : 3.8.13.final.0
python-bits : 64
OS : Linux
OS-release : 3.10.0-1160.76.1.el7.x86_64
Version : #1 SMP Wed Aug 10 16:21:17 UTC 2022
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : None.None
pandas : 1.5.1
numpy : 1.23.4
pytz : 2022.6
dateutil : 2.8.2
setuptools : 63.4.1
pip : 22.2.2
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 8.6.0
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None
tzdata : None