Parts of strings that look like comments being stripped out of nested strings with debug specifier

[tom-pytel]

Bug report

Bug description:

Is this desired behavior?

$ ./python -m ast
f'{""" # this is part of the string, it shouldn't be stripped out
"""=}'
Module(
   body=[
      Expr(
         value=JoinedStr(
            values=[
               Constant(value='""" \n"""='),
               FormattedValue(
                  value=Constant(value=" # this is part of the string, it shouldn't be stripped out\n"),
                  conversion=114)]))])

$ ./python -m ast
t'{""" # this is part of the string, it shouldn't be stripped out
"""=}'
Module(
   body=[
      Expr(
         value=TemplateStr(
            values=[
               Constant(value='""" \n"""='),
               Interpolation(
                  value=Constant(value=" # this is part of the string, it shouldn't be stripped out\n"),
                  str='""" \n"""',
                  conversion=114)]))])

Even if they are not actually multiline:

$ python -m ast
t'{" # nooo "=}'
Module(
   body=[
      Expr(
         value=TemplateStr(
            values=[
               Constant(value='" '),
               Interpolation(
                  value=Constant(value=' # nooo '),
                  str='"',
                  conversion=114)]))])

Seems to go back to 3.12

CPython versions tested on:

3.14, CPython main branch

Operating systems tested on:

No response

Linked PRs

• gh-135148: Correctly handle f/t strings with comments and debug expressions #135198

[JelleZijlstra]

Yeah this is clearly wrong. cc @pablogsal @lysnikolaou

[JelleZijlstra]

This is actually a regression in 3.12, likely due to PEP-701:

$ uv run --python=3.12 python
Python 3.12.2 (main, May  6 2024, 20:50:52) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> f'{" # no "=}'
'" \' # no \''
>>> 
$ uv run --python=3.11 python
Python 3.11.11 (main, Feb 12 2025, 15:06:01) [Clang 19.1.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> f'{" # no "=}'
'" # no "=\' # no \''

Removing the release blocker label as it's old.

[tom-pytel]

Not just debug specifier, note the .str field:

$ ./python -m ast
t'{" # no "}'
Module(
   body=[
      Expr(
         value=TemplateStr(
            values=[
               Interpolation(
                  value=Constant(value=' # no '),
                  str='"',
                  conversion=-1)]))])

Could this be problematic?

>>> t'{" ! important information ! "}'.interpolations
(Interpolation(' ! important information ! ', '" ! important information ! "', None, ''),)

>>> t'{" # important information # "}'.interpolations
(Interpolation(' # important information # ', '"', None, ''),)

[lysnikolaou]

Oof, yeah, this was probably introduced in #112284. We'll need to fix it, but the fix is not exactly straight-forward. I'll take a deeper look tomorrow.

[tom-pytel]

the fix is not exactly straight-forward.

Not sure if this would apply in the context of the parser but worked for me before I realized it wasn't matching up.

1. In range of interest exclude all lines which have multiline strings which do not end on the line.

2. For each remaining line store maximum end column of expressions that end on that line and restrict comment checks to past this column.