strings should use an old_str / old_unicode type, similar to old_div

[bwo]

Consider the following module:

$ cat test.py
import helper

helper.discriminate(str(3))
helper.discriminate('d')
helper.discriminate(str(u'd'))
helper.discriminate('d' + str(3))


print((chr(123) + chr(125)).format('1'))
helper.discriminate('{}'.format(1))
helper.discriminate(chr(10))

(Don't worry about helper for now.)

I would expect the following:

• in Python 2, str(3) returns not-unicode, so following the transformations, that behavior will be preserved (i.e. in Python 3 I'll get either bytes or some kind of emulation of Python 2's str; in Python 2, I'll still get str).
• similarly, since there's no from __future__ import unicode_literals heading the file, the string literal 'd' will become the literal b'd', or some kind of emulation object.
• since I'm using str to explicitly coerce u'd' to str, this will continue to work.
• for the format calls, in Python 3 a bytes object obviously won't work, since it won't have format()—so, again, some kind of from past import str is what I'd expect.

None of this is what happens, though.

Here's helper.py, followed by the results of running test.py:

$ cat helper.py
def discriminate(o):
    if isinstance(o, unicode):
        print('barf')
    elif isinstance(o, str):
        print('ok good')
$ python2 test.py
ok good
ok good
ok good
ok good
1
ok good
ok good

Now let's try translating first test.py, and then also helper.py:

$ futurize -w -0 test.py 2>/dev/null
--- test.py     (original)
+++ test.py     (refactored)
@@ -1,3 +1,6 @@
+from __future__ import print_function
+from builtins import str
+from builtins import chr
 import helper

 helper.discriminate(str(3))
$ python test.py
barf
ok good
barf
barf
1
ok good
barf
$ futurize -w -0 helper.py 2>/dev/null
--- helper.py   (original)
+++ helper.py   (refactored)
@@ -1,5 +1,6 @@
+from __future__ import print_function
 def discriminate(o):
-    if isinstance(o, unicode):
+    if isinstance(o, str):
         print('barf')
     elif isinstance(o, str):
         print('ok good')
$ python test.py
barf
1
barf

So, calls to str and chr now produce unicode, but the string literal 'd' is still a Python 2 str. If we translate module-by-module, the behavior of the script obviously changes. Even using str to cast between types doesn't work anymore! Moreover, the isinstance checks in helper are completely wrong.

AFAICT the only way to actually mechanically translate code using strings without introducing semantic differences is just to use a compatibility layer, including one around every literal. I know you all have spent a lot more time on this than I have (and also that such a layer would be a PITA to get right), but the above complications bit me hard in trying futurize out on part of the codebase for a reasonably large project (and translating the entirety at once wouldn't fix it). It seems as if even with futurize, without such a compatibility layer, there's no alternative but to check almost every string :(.

[edschofield]

Thanks for your comments. Yes, futurize currently has safe transformations in --stage1 and potentially unsafe transformations in --stage2. The str/bytes boundary is the most difficult to get right. The main goal of --stage2 is to help with modernizing your code base by producing a diff that is Py3-like while highlighting parts of the code that may need manual attention. Stage 2 could conceivably work differently, invoking the past module's str object etc. to try to produce code that does a perfect translation of Py2 to Py2/3, but (a) this would still not work, because there are so many functions in the standard library and third-party libraries whose bytes/str handling has changed in different directions, and (b) it would introduce compatibility hacks which make the code less maintainable. For your use-case, check out the Py2-like str object and others in the past module; this preserves Py2 str semantics in your Py3 code ... but this may not be a good long-term strategy for your code base...

…

________________________________ From: bwo <notifications@github.com> Sent: Thursday, June 1, 2017 5:30:55 AM To: PythonCharmers/python-future Cc: Subscribed Subject: [PythonCharmers/python-future] strings should use an old_str / old_unicode type, similar to old_div (#285) Consider the following module: $ cat test.py import helper helper.discriminate(str(3)) helper.discriminate('d') helper.discriminate(str(u'd')) helper.discriminate('d' + str(3)) print((chr(123) + chr(125)).format('1')) helper.discriminate('{}'.format(1)) helper.discriminate(chr(10)) (Don't worry about helper for now.) I would expect the following: * in Python 2, str(3) returns not-unicode, so following the transformations, that behavior will be preserved (i.e. in Python 3 I'll get either bytes or some kind of emulation of Python 2's str; in Python 2, I'll still get str). * similarly, since there's no from __future__ import unicode_literals heading the file, the string literal 'd' will become the literal b'd', or some kind of emulation object. * since I'm using str to explicitly coerce u'd' to str, this will continue to work. * for the format calls, in Python 3 a bytes object obviously won't work, since it won't have format()—so, again, some kind of from past import str is what I'd expect. None of this is what happens, though. Here's helper.py, followed by the results of running test.py: $ cat helper.py def discriminate(o): if isinstance(o, unicode): print('barf') elif isinstance(o, str): print('ok good') $ python2 test.py ok good ok good ok good ok good 1 ok good ok good Now let's try translating first test.py, and then also helper.py: $ futurize -w -0 test.py 2>/dev/null --- test.py (original) +++ test.py (refactored) @@ -1,3 +1,6 @@ +from __future__ import print_function +from builtins import str +from builtins import chr import helper helper.discriminate(str(3)) $ python test.py barf ok good barf barf 1 ok good barf $ futurize -w -0 helper.py 2>/dev/null --- helper.py (original) +++ helper.py (refactored) @@ -1,5 +1,6 @@ +from __future__ import print_function def discriminate(o): - if isinstance(o, unicode): + if isinstance(o, str): print('barf') elif isinstance(o, str): print('ok good') $ python test.py barf 1 barf So, calls to str and chr now produce unicode, but the string literal 'd' is still a Python 2 str. If we translate module-by-module, the behavior of the script obviously changes. Even using str to cast between types doesn't work anymore! Moreover, the isinstance checks in helper are completely wrong. AFAICT the only way to actually mechanically translate code using strings without introducing semantic differences is just to use a compatibility layer, including one around every literal. I know you all have spent a lot more time on this than I have (and also that such a layer would be a PITA to get right), but the above complications bit me hard in trying futurize out on part of the codebase for a reasonably large project (and translating the entirety at once wouldn't fix it). It seems as if even with futurize, without such a compatibility layer, there's no alternative but to check almost every string :(. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#285>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAEx6vO6hGF4DYDJHgtETPqND2vonYjJks5r_b_vgaJpZM4NsFfC>.