Skip to content

Commit 14e976e

Browse files
appeltelbenjaminp
authored andcommitted
closes bpo-30117: fix lib2to3 ParserIdempotency test (GH-1242)
Fix two (in my opinion) spurious failure conditions in the lib2to3.tests.test_parser.TestParserIdempotency test_parser test. Use the same encoding found in the initial file to write a temp file for a diff. This retains the BOM if the encoding was initially utf-8-sig. If the file cannot be parsed using the normal grammar, try again with no print statement which should succeed for valid files using future print_function For case (1), the driver was correctly handling a BOM in a utf-8 file, but then the test was not writing a comparison file using 'utf-8-sig' to diff against, so the BOM got removed. I don't think that is the fault of the parser, and lib2to3 will retain the BOM. For case (2), lib2to3 pre-detects the use of from __future__ import print_function or allows the user to force this interpretation with a -p flag, and then selects a different grammar with the print statement removed. That makes the test cases unfair to this test as the driver itself doesn't know which grammar to use. As a minimal fix, the test will try using a grammar with the print statement, and if that fails fall back on a grammar without it. A more thorough handling of the idempotency test would to be to parse all files using both grammars and ignore if one of the two failed but otherwise check both. I didn't think this was necessary but can change.
1 parent ce0f33d commit 14e976e

File tree

2 files changed

+15
-10
lines changed

2 files changed

+15
-10
lines changed

Lib/lib2to3/tests/support.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,13 @@
1515
proj_dir = os.path.normpath(os.path.join(test_dir, ".."))
1616
grammar_path = os.path.join(test_dir, "..", "Grammar.txt")
1717
grammar = pgen2_driver.load_grammar(grammar_path)
18+
grammar_no_print_statement = pgen2_driver.load_grammar(grammar_path)
19+
del grammar_no_print_statement.keywords["print"]
1820
driver = pgen2_driver.Driver(grammar, convert=pytree.convert)
21+
driver_no_print_statement = pgen2_driver.Driver(
22+
grammar_no_print_statement,
23+
convert=pytree.convert
24+
)
1925

2026
def parse_string(string):
2127
return driver.parse_string(reformat(string), debug=True)

Lib/lib2to3/tests/test_parser.py

Lines changed: 9 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88

99
# Testing imports
1010
from . import support
11-
from .support import driver
11+
from .support import driver, driver_no_print_statement
1212
from test.support import verbose
1313

1414
# Python imports
@@ -413,8 +413,6 @@ class TestParserIdempotency(support.TestCase):
413413

414414
"""A cut-down version of pytree_idempotency.py."""
415415

416-
# Issue 13125
417-
@unittest.expectedFailure
418416
def test_all_project_files(self):
419417
for filepath in support.all_project_files():
420418
with open(filepath, "rb") as fp:
@@ -425,12 +423,13 @@ def test_all_project_files(self):
425423
source = fp.read()
426424
try:
427425
tree = driver.parse_string(source)
428-
except ParseError as err:
429-
if verbose > 0:
430-
warnings.warn('ParseError on file %s (%s)' % (filepath, err))
431-
continue
426+
except ParseError:
427+
try:
428+
tree = driver_no_print_statement.parse_string(source)
429+
except ParseError as err:
430+
self.fail('ParseError on file %s (%s)' % (filepath, err))
432431
new = str(tree)
433-
x = diff(filepath, new)
432+
x = diff(filepath, new, encoding=encoding)
434433
if x:
435434
self.fail("Idempotency failed: %s" % filepath)
436435

@@ -481,9 +480,9 @@ def test_trailing_comma_after_generator_expression_argument_works(self):
481480
self.validate("set(x for x in [],)")
482481

483482

484-
def diff(fn, result):
483+
def diff(fn, result, encoding='utf-8'):
485484
try:
486-
with open('@', 'w') as f:
485+
with open('@', 'w', encoding=encoding, newline='\n') as f:
487486
f.write(str(result))
488487
fn = fn.replace('"', '\\"')
489488
return subprocess.call(['diff', '-u', fn, '@'], stdout=(subprocess.DEVNULL if verbose < 1 else None))

0 commit comments

Comments
 (0)