[SR-13280] Migrate Swift to Python 3

[compnerd]

Previous ID	SR-13280
Radar	None
Original Reporter	@compnerd
Type	Bug

Additional Detail from JIRA

Votes	2
Component/s
Labels	Bug
Assignee	None
Priority	Medium

md5: 8dfbb0580c079fb8d47eec5326b3108a

Sub-Tasks:

• SR-13281 Migrate build to Python 3
• SR-13282 Switch LLVM to versioned Python
• SR-13284 Remove workaround for Python 3 to Python 2 fallback
• SR-13285 Migrate test suite to Python 3
• SR-13286 Migrate validation test suite to Python 3
• SR-13287 Switch test suite to python 3 by default
• SR-13288 Add Python 3.7 to Linux builders
• SR-13290 Add flake8/flake8_import_order to builders
• SR-13291 Make LLVM lit python 3 compatible for Swift
• SR-13292 Ensure integration tests work with Python 3
• SR-13304 Remove explicit use of Python2
• SR-13319 Install python3-devel on CI builders
• SR-16025 Delete usage of the "six" Python library
• SR-16064 Remove ambiguous reference to 'python'
• SR-16073 Remove future imports related to Python 3

Issue Description:

This issue tracks the work to migrate the Swift repositories to migrate to Python 3

[compnerd]

+ @shahmishal @JDevlieghere najacque (JIRA User) @hyp tachoknight (JIRA User) @drodriguez @tbkka @drexin plotfi (JIRA User)

[tbkka]

The Swift test suite, validation test suite, and benchmark suite all seem to work with Python 3 now. I need to run some more tests on systems where `python` is Python 3 and ones where `python` is Python 2.7 to make sure everything is stable in both scenarios.

[tbkka]

I think PR#33146 fixes the very last python_lint failure for Python 3.

[tbkka]

I'm now locally seeing repeated successes with `build-script -rTB` on both macOS and Linux with python == python2.7 and with python == python3.8. So now I'm trying to expand the scope of my testing:

Using `build-script -rTB --lldb` and with python == python2.7 on Ubuntu 20.04
I'm now getting failures from Runtime/linux-fatal-backtrace.swift

The error claims that `_lldb.so` lacks a `PyInit_lldb` function so it cannot be imported as a module. I'm not sure how the Python version could be causing this, but I'll dig into it...

I wonder if the swift build needs an explicit "Python_LLDB" to match whatever Python was used by LLDB for those cases where it matters? Or would it make more sense to try to get both LLDB and Swift to agree to always use the default `python`? (The latter seems troublesome, as there are still a lot of systems where python == python2.7 and I expect some of our scripts will soon start requiring Python 3 features.)

[tbkka]

Ah. Apparently Python3 and Python2 have very different conventions for how C modules get linked. I somehow ended up with a module built for Python2 trying to be run with Python3, due to some Python version mismatch between LLDB and Swift builds. I'll try to puzzle out where that came from.

[tbkka]

I believe #33181 is sufficient to ensure that any Swift tests that use the LLDB module get run with the same Python version that LLDB used.

Testing help: Running broad sets of tests (including validation tests, benchmarks, sourcekit, foundation, swiftpm, lldb, etc) on systems with multiple Python versions installed. CI runs particular combinations, but it cannot try every combination that people may be using for regular development.

[swift-ci]

Comment by solo turn (JIRA)

@tbkka, on arch linux python = pyton3. i installed python2 as well, built swift from master, run the test suite, and the following are python2 and fail because they are called with "python":

• swift-integration-tests/test-sourcekit-lsp/test-sourcekit-lsp.py

• swift-integration-tests/test-snapshot-binaries/linux_load_commands.py

[swift-ci]

Comment by solo turn (JIRA)

for building, i needed to patch CMakeLists to not require python2 otherwise the build would fail on arch linux, see SR-13201.

[swift-ci]

Comment by solo turn (JIRA)

on arch, with both, python3 and python2 installed, the build is successful. not working is ocaml build for a call to "ar" which does not work, SR-13083, and the 2 integration tests mentioned above. and, most of the time it gets "redefinition of 'realpath'" because of the arch linux default of "CPPFLAGS="-D_FORTIFY_SOURCE=2", SR-13082, which is kind of blocking.

i switched the github build to "every 3 hours":
https://github.com/soloturn/swift-aur/actions

if you want i can remove python2 from the PKGBUILD and only build it with python3, or you commit there if you want?
https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=swift-language-git

patch files are here:
https://aur.archlinux.org/cgit/aur.git/tree/?h=swift-language-git

[swift-ci]

Comment by solo turn (JIRA)

@gottesmm, here open python2 tasks. @compnerd is still working somebody on python3?

[dduan]

I believe this PR is a good symbolic point that signifies the completion of this epic.