Compiler selects invalid `--lldb-python` path

[Dante-Broggi]

For my mac (macOS Mojave), the --lldb-python path is being set to /usr/bin/python3, which doesn't exist.
Furthermore, /usr/bin is immutable, so I cannot simply symlink to /usr/local/bin/python3, the result of which python3.

This is causing the debuginfo test suite to fail.

Code

./x.py -i test src/test/debuginfo

Meta

rust version:
Top of master, specifically:
3dbade6

Error output

running 115 tests
iFFFFFFFiFFFFFiFFFFFFFFFFFiFFFFiFFiiFFiFFiFFiiFiFFFFFFFFFFFFFiFiFiiFFFFFFFFiFFFFFiFiiFFFF.iFFFFiFFiF 100/115
FFFFiiFiFFFFFFF
failures:

---- [debuginfo-lldb] debuginfo/basic-types-globals-metadata.rs stdout ----
NOTE: compiletest thinks it is using LLDB version 1100
NOTE: compiletest thinks it is using LLDB without native rust support

error: Failed to setup Python process for LLDB script: No such file or directory (os error 2)
[ERROR compiletest::runtest] fatal error, panic: "Failed to setup Python process for LLDB script: No such file or directory (os error 2)"
thread 'main' panicked at 'fatal error', src/tools/compiletest/src/runtest.rs:2133:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
// …

Backtrace

running 115 tests
iFFFFFFFiFFFFFiFFFFFFFFFFFiFFFFiFFiiFFiFFiFFiiFiFFFFFFFFFFFFFiFiFiiFFFFFFFFiFFFFFiFiiFFFF.iFFFFiFFiF 100/115
FFFFiiFiFFFFFFF
failures:

---- [debuginfo-lldb] debuginfo/basic-types-globals-metadata.rs stdout ----
NOTE: compiletest thinks it is using LLDB version 1100
NOTE: compiletest thinks it is using LLDB without native rust support

error: Failed to setup Python process for LLDB script: No such file or directory (os error 2)
[ERROR compiletest::runtest] fatal error, panic: "Failed to setup Python process for LLDB script: No such file or directory (os error 2)"
thread 'main' panicked at 'fatal error', src/tools/compiletest/src/runtest.rs:2133:9
stack backtrace:
   0: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
   1: core::fmt::write
   2: std::io::Write::write_fmt
   3: std::io::impls::<impl std::io::Write for alloc::boxed::Box<W>>::write_fmt
   4: std::sys_common::backtrace::print
   5: std::panicking::default_hook::{{closure}}
   6: std::panicking::default_hook
   7: std::panicking::rust_panic_with_hook
   8: std::panicking::begin_panic
   9: compiletest::runtest::TestCx::fatal
  10: compiletest::runtest::TestCx::cmd2procres
  11: compiletest::runtest::TestCx::run_revision
  12: compiletest::runtest::run
  13: core::ops::function::FnOnce::call_once{{vtable.shim}}
  14: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
  15: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
  16: __rust_maybe_catch_panic
  17: std::panicking::try
  18: test::run_test_in_process
  19: test::run_test::run_test_inner
  20: test::run_test
  21: test::run_tests
  22: test::console::run_tests_console
  23: compiletest::main
  24: std::rt::lang_start::{{closure}}
  25: std::panicking::try::do_call
  26: __rust_maybe_catch_panic
  27: std::panicking::try
  28: std::rt::lang_start_internal
  29: main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[ehuss]

I seem to recall having problems with these tests, even before #68863.

Here's an overview of how I understand the problem:

The lldb that ships with Xcode 11 uses python3. Since 10.14 doesn't include python3, AFAIK it is not possible to use xcode's Python lldb module on Mojave (using an external interpreter). Using homebrew doesn't work either, because it is not compatible with the _lldb.so module.

EDIT: Oh, and there is a Python2 version of the lldb module at /Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python. The only way to discover this path would be to run lldb -P and see if it ends with "Python3" and modify the path. That might work, but seems fragile.

One option is to build lldb yourself by editing config.toml and enabling lldb. This will build lldb with python2. The bootstrap code would have to also be changed to use /usr/bin/python instead.

Another option is to use an older version of xcode (and also change the bootstrap hard-coded python3 path).

I think those options are kinda terrible. My preference would be to disable the debuginfo tests on macos if the system version is older than 10.15. I don't know how feasible that is.

@pietroalbini Does that sound like a reasonable summary and possible solution?

It's also not clear whether or not python3 will be removed in 10.16. The release notes just say python will be removed in "Future versions of macOS", but don't say which future versions. If /usr/bin/python3 is removed, I'm not sure how we'll be able to run the tests. It looks like the lldb_batchmode.py script fundamentally needs to use the lldb python module outside of lldb itself. If Apple removes python, then I think the only solution to continue running the debuginfo tests is to build lldb locally and not use the one that ships with xcode. It could also be possible that future Xcode versions will install its own python3?

[pietroalbini]

That explaination sounds plausible based on my limited understanding of the situation (not having any Apple system made writing and debugging that PR so much harder).

cc @Mark-Simulacrum on what the best approach for bootstrap would be

[Mark-Simulacrum]

I'm pretty much down for any solution in bootstrap, detecting the macOS version sounds finicky but also reasonable. I am unlikely to invest time into fixing this myself, though I would be happy to review a PR.

[golddranks]

I'm also encountering this bug. Building rustc fails on Mojave 10.14 with the same error message.

Just to get unstuck with developing rustc, is there a quick option to disable the affected tests?

[ehuss]

@golddranks What are you working on? I think most people do not run all tests, but instead run the specific test-suites that are relevant to what you are doing. I think you can also pass the --exclude flag to skip over the offending suite. Or you can try one of the solutions outlined above if you're actually working on debug-related stuff, the changes should be relatively small.

[golddranks]

@ehuss On stdlib. --exclude helped, and now I feel stupid for not finding it myself :D Thanks.