Running `x.py build` (after selecting 'c' to modify LLVM/codegen in setup) requires installing ninja, and it is not obvious how to do that

[daira]

I'm trying to build the compiler for the first time, following the instructions at https://rustc-dev-guide.rust-lang.org/building/how-to-build-and-run.html

daira@katava:~$ git clone https://github.com/rust-lang/rust.git
Cloning into 'rust'...
[...]
daira@katava:~$ cd rust
daira@katava:~/rust$ ls
Cargo.lock          config.toml.example  library         RELEASES.md     x.py
Cargo.toml          configure            LICENSE-APACHE  rustfmt.toml
CODE_OF_CONDUCT.md  CONTRIBUTING.md      LICENSE-MIT     src
compiler            COPYRIGHT            README.md       triagebot.toml
daira@katava:~/rust$ ./x.py setup
Updating only changed submodules
Updating submodule src/tools/rust-installer [...]
Updating submodule src/tools/cargo [...]
Updating submodule src/tools/rls [...]
Updating submodule src/tools/miri [...]
Updating submodule library/backtrace [...]
Updating submodule library/stdarch [...]
Submodules updated in 20.69 seconds
downloading https://static.rust-lang.org/dist/2021-09-08/rust-std-beta-x86_64-unknown-linux-gnu.tar.xz
######################################################################### 100.0%
extracting /home/daira/rust/build/cache/2021-09-08/rust-std-beta-x86_64-unknown-linux-gnu.tar.xz
downloading https://static.rust-lang.org/dist/2021-09-08/rustc-beta-x86_64-unknown-linux-gnu.tar.xz
######################################################################### 100.0%
extracting /home/daira/rust/build/cache/2021-09-08/rustc-beta-x86_64-unknown-linux-gnu.tar.xz
downloading https://static.rust-lang.org/dist/2021-09-08/cargo-beta-x86_64-unknown-linux-gnu.tar.xz
######################################################################### 100.0%
extracting /home/daira/rust/build/cache/2021-09-08/cargo-beta-x86_64-unknown-linux-gnu.tar.xz
downloading https://static.rust-lang.org/dist/2021-09-08/rustfmt-nightly-x86_64-unknown-linux-gnu.tar.xz
######################################################################### 100.0%
extracting /home/daira/rust/build/cache/2021-09-08/rustfmt-nightly-x86_64-unknown-linux-gnu.tar.xz
  Downloaded toml v0.5.7
  Downloaded opener v0.5.0
  Downloaded merge_derive v0.1.0
  Downloaded ignore v0.4.17
  Downloaded globset v0.4.5
  Downloaded autocfg v1.0.0
  Downloaded cmake v0.1.44
  Downloaded merge v0.1.0
  Downloaded 8 crates (189.4 KB) in 0.40s
   Compiling proc-macro2 v1.0.24
   Compiling unicode-xid v0.2.2
   Compiling syn v1.0.65
   Compiling version_check v0.9.3
   Compiling memchr v2.4.1
   Compiling autocfg v1.0.0
   Compiling lazy_static v1.4.0
   Compiling libc v0.2.99
   Compiling serde_derive v1.0.125
   Compiling cfg-if v1.0.0
   Compiling log v0.4.14
   Compiling serde v1.0.125
   Compiling regex-automata v0.1.10
   Compiling ryu v1.0.5
   Compiling regex-syntax v0.6.25
   Compiling same-file v1.0.6
   Compiling serde_json v1.0.59
   Compiling fnv v1.0.7
   Compiling unicode-width v0.1.8
   Compiling itoa v0.4.6
   Compiling cc v1.0.69
   Compiling bootstrap v0.0.0 (/home/daira/rust/src/bootstrap)
   Compiling once_cell v1.7.2
   Compiling build_helper v0.1.0 (/home/daira/rust/src/build_helper)
   Compiling thread_local v1.0.1
   Compiling walkdir v2.3.1
   Compiling getopts v0.2.21
   Compiling proc-macro-error-attr v1.0.4
   Compiling proc-macro-error v1.0.4
   Compiling num-traits v0.2.12
   Compiling crossbeam-utils v0.8.3
   Compiling cmake v0.1.44
   Compiling aho-corasick v0.7.18
   Compiling bstr v0.2.13
   Compiling quote v1.0.7
   Compiling time v0.1.43
   Compiling num_cpus v1.13.0
   Compiling filetime v0.2.14
   Compiling opener v0.5.0
   Compiling regex v1.5.4
   Compiling globset v0.4.5
   Compiling ignore v0.4.17
   Compiling merge_derive v0.1.0
   Compiling merge v0.1.0
   Compiling toml v0.5.7
    Finished dev [unoptimized + debuginfo] target(s) in 28.54s
Welcome to the Rust project! What do you want to do with x.py?
a) library: Contribute to the standard library
b) compiler: Contribute to the compiler itself
c) codegen: Contribute to the compiler, and also modify LLVM or codegen
d) tools: Contribute to tools which depend on the compiler, but do not modify it directly (e.g. rustdoc, clippy, miri)
e) user: Install Rust from source
Please choose one (a/b/c/d/e): c
`x.py` will now use the configuration at /home/daira/rust/src/bootstrap/defaults/config.codegen.toml

Rust's CI will automatically fail if it doesn't pass `tidy`, the internal tool for ensuring code quality.
If you'd like, x.py can install a git hook for you that will automatically run `tidy --bless` on each commit
to ensure your code is up to par. If you decide later that this behavior is undesirable,
simply delete the `pre-commit` file from .git/hooks.
Would you like to install the git hook?: [y/N] y
Linked `src/etc/pre-commit.sh` to `.git/hooks/pre-commit`

To get started, try one of the following commands:
- `x.py check`
- `x.py build`
- `x.py test`
For more suggestions, see https://rustc-dev-guide.rust-lang.org/building/suggested.html
Build completed successfully in 0:01:50
daira@katava:~/rust$ ./x.py build
Updating only changed submodules
Submodules updated in 0.01 seconds
    Finished dev [unoptimized + debuginfo] target(s) in 0.12s

Couldn't find required command: ninja
You should install ninja, or set `ninja=false` in config.toml in the `[llvm]` section.

Build completed unsuccessfully in 0:00:00
daira@katava:~/rust$ sudo apt-get install ninja
[sudo] password for daira: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Package ninja is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'ninja' has no installation candidate
daira@katava:~/rust$ cargo install ninja
    Updating crates.io index
  Downloaded ninja v0.0.1
  Downloaded 1 crate (1.7 KB) in 0.85s
  Installing ninja v0.0.1
   Compiling ninja v0.0.1
    Finished release [optimized] target(s) in 3.16s
  Installing /home/daira/.cargo/bin/ninja
   Installed package `ninja v0.0.1` (executable `ninja`)
daira@katava:~/rust$ ninja
thread 'main' panicked at 'Failed to open build.ninja: Os { code: 2, kind: NotFound, message: "No such file or directory" }', /home/daira/.cargo/registry/src/github.com-1ecc6299db9ec823/ninja-0.0.1/src/bin/ninja.rs:9:46
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
daira@katava:~/rust$ ./x.py build
Updating only changed submodules
Submodules updated in 0.01 seconds
    Finished dev [unoptimized + debuginfo] target(s) in 0.41s
Updating submodule src/llvm-project
Submodule 'src/llvm-project' (https://github.com/rust-lang/llvm-project.git) registered for path 'src/llvm-project'
Cloning into '/home/daira/rust/src/llvm-project'...
[...]
    Finished release [optimized] target(s) in 32.37s
Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building LLVM for x86_64-unknown-linux-gnu
running: "cmake" "/home/daira/rust/src/llvm-project/llvm" "-G" "Ninja" "-DLLVM_ENABLE_ASSERTIONS=ON" "-DLLVM_ENABLE_PLUGINS=OFF" "-DLLVM_TARGETS_TO_BUILD=AArch64;ARM;BPF;Hexagon;MSP430;Mips;NVPTX;PowerPC;RISCV;Sparc;SystemZ;WebAssembly;X86" "-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=AVR" "-DLLVM_INCLUDE_EXAMPLES=OFF" "-DLLVM_INCLUDE_DOCS=OFF" "-DLLVM_INCLUDE_BENCHMARKS=OFF" "-DLLVM_INCLUDE_TESTS=OFF" "-DLLVM_ENABLE_TERMINFO=OFF" "-DLLVM_ENABLE_LIBEDIT=OFF" "-DLLVM_ENABLE_BINDINGS=OFF" "-DLLVM_ENABLE_Z3_SOLVER=OFF" "-DLLVM_PARALLEL_COMPILE_JOBS=12" "-DLLVM_TARGET_ARCH=x86_64" "-DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu" "-DLLVM_INSTALL_UTILS=ON" "-DLLVM_ENABLE_ZLIB=ON" "-DLLVM_ENABLE_LIBXML2=OFF" "-DLLVM_VERSION_SUFFIX=-rust-dev" "-DCMAKE_INSTALL_MESSAGE=LAZY" "-DCMAKE_C_COMPILER=cc" "-DCMAKE_CXX_COMPILER=c++" "-DCMAKE_ASM_COMPILER=cc" "-DCMAKE_C_FLAGS=-ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_CXX_FLAGS=-ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_INSTALL_PREFIX=/home/daira/rust/build/x86_64-unknown-linux-gnu/llvm" "-DCMAKE_ASM_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_BUILD_TYPE=Release"
CMake Error at CMakeLists.txt:44 (project):
  Running

   '/home/daira/.cargo/bin/ninja' '--version'

  failed with:

   thread 'main' panicked at 'Failed to open build.ninja: Os { code: 2, kind: NotFound, message: "No such file or directory" }', /home/daira/.cargo/registry/src/github.com-1ecc6299db9ec823/ninja-0.0.1/src/bin/ninja.rs:9:46

  note: run with `RUST_BACKTRACE=1` environment variable to display a
  backtrace



-- Configuring incomplete, errors occurred!
See also "/home/daira/rust/build/x86_64-unknown-linux-gnu/llvm/build/CMakeFiles/CMakeOutput.log".
thread 'main' panicked at '
command did not execute successfully, got: exit status: 1

build script failed, must exit now', /home/daira/.cargo/registry/src/github.com-1ecc6299db9ec823/cmake-0.1.44/src/lib.rs:885:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
	finished in 0.102 seconds
Build completed unsuccessfully in 0:07:15

Possible solutions

One of:

• documentation at https://rustc-dev-guide.rust-lang.org/building/how-to-build-and-run.html explaining how to install ninja, or
• documentation at https://rustc-dev-guide.rust-lang.org/building/how-to-build-and-run.html saying to set ninja=false in the [llvm] section of config.toml, or
• ninja to be installed automatically by ./x.py setup, or
• ninja=false to be set automatically by default by ./x.py setup;

and

• a solution for the Failed to open build.ninja error.

[daira]

Here's the backtrace:

thread 'main' panicked at '
command did not execute successfully, got: exit status: 1

build script failed, must exit now', /home/daira/.cargo/registry/src/github.com-1ecc6299db9ec823/cmake-0.1.44/src/lib.rs:885:5
stack backtrace:
   0: rust_begin_unwind
             at /rustc/e784c962ea252f0874a4305168077e7048cb39e9/library/std/src/panicking.rs:517:5
   1: std::panicking::begin_panic_fmt
             at /rustc/e784c962ea252f0874a4305168077e7048cb39e9/library/std/src/panicking.rs:460:5
   2: cmake::fail
             at /home/daira/.cargo/registry/src/github.com-1ecc6299db9ec823/cmake-0.1.44/src/lib.rs:885:5
   3: cmake::run
             at /home/daira/.cargo/registry/src/github.com-1ecc6299db9ec823/cmake-0.1.44/src/lib.rs:863:9
   4: cmake::Config::build
             at /home/daira/.cargo/registry/src/github.com-1ecc6299db9ec823/cmake-0.1.44/src/lib.rs:698:13
   5: <bootstrap::native::Llvm as bootstrap::builder::Step>::run
             at ./src/bootstrap/native.rs:359:9
   6: bootstrap::builder::Builder::ensure
             at ./src/bootstrap/builder.rs:1580:23
   7: bootstrap::compile::rustc_cargo_env
             at ./src/bootstrap/compile.rs:700:27
   8: bootstrap::compile::rustc_cargo
             at ./src/bootstrap/compile.rs:652:5
   9: <bootstrap::compile::Rustc as bootstrap::builder::Step>::run
             at ./src/bootstrap/compile.rs:588:9
  10: bootstrap::builder::Builder::ensure
             at ./src/bootstrap/builder.rs:1580:23
  11: <bootstrap::compile::Assemble as bootstrap::builder::Step>::run
             at ./src/bootstrap/compile.rs:1075:9
  12: bootstrap::builder::Builder::ensure
             at ./src/bootstrap/builder.rs:1580:23
  13: bootstrap::builder::Builder::compiler
             at ./src/bootstrap/builder.rs:627:9
  14: <bootstrap::compile::Std as bootstrap::builder::Step>::make_run
             at ./src/bootstrap/compile.rs:52:23
  15: bootstrap::builder::StepDescription::maybe_run
             at ./src/bootstrap/builder.rs:175:13
  16: bootstrap::builder::StepDescription::run
             at ./src/bootstrap/builder.rs:211:25
  17: bootstrap::builder::Builder::run_step_descriptions
             at ./src/bootstrap/builder.rs:619:9
  18: bootstrap::builder::Builder::execute_cli
             at ./src/bootstrap/builder.rs:599:9
  19: bootstrap::Build::build
             at ./src/bootstrap/lib.rs:624:13
  20: bootstrap::main
             at ./src/bootstrap/bin/main.rs:33:5
  21: core::ops::function::FnOnce::call_once
             at /rustc/e784c962ea252f0874a4305168077e7048cb39e9/library/core/src/ops/function.rs:227:5

[daira]

Adding an [llvm] section with ninja=false to config.toml in the root does not appear to make any difference:

daira@katava:~/rust$ cat config.toml
# Includes one of the default files in src/bootstrap/defaults
profile = "codegen"
changelog-seen = 2

[llvm]
ninja = false

[daira]

The next thing I tried was to uninstall ninja and install ninja-build, which resulted in this error instead:

running: "cmake" "/home/daira/rust/src/llvm-project/llvm" "-G" "Ninja" "-DLLVM_ENABLE_ASSERTIONS=ON" "-DLLVM_ENABLE_PLUGINS=OFF" "-DLLVM_TARGETS_TO_BUILD=AArch64;ARM;BPF;Hexagon;MSP430;Mips;NVPTX;PowerPC;RISCV;Sparc;SystemZ;WebAssembly;X86" "-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=AVR" "-DLLVM_INCLUDE_EXAMPLES=OFF" "-DLLVM_INCLUDE_DOCS=OFF" "-DLLVM_INCLUDE_BENCHMARKS=OFF" "-DLLVM_INCLUDE_TESTS=OFF" "-DLLVM_ENABLE_TERMINFO=OFF" "-DLLVM_ENABLE_LIBEDIT=OFF" "-DLLVM_ENABLE_BINDINGS=OFF" "-DLLVM_ENABLE_Z3_SOLVER=OFF" "-DLLVM_PARALLEL_COMPILE_JOBS=12" "-DLLVM_TARGET_ARCH=x86_64" "-DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu" "-DLLVM_INSTALL_UTILS=ON" "-DLLVM_ENABLE_ZLIB=ON" "-DLLVM_ENABLE_LIBXML2=OFF" "-DLLVM_VERSION_SUFFIX=-rust-dev" "-DCMAKE_INSTALL_MESSAGE=LAZY" "-DCMAKE_C_COMPILER=cc" "-DCMAKE_CXX_COMPILER=c++" "-DCMAKE_ASM_COMPILER=cc" "-DCMAKE_C_FLAGS=-ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_CXX_FLAGS=-ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_INSTALL_PREFIX=/home/daira/rust/build/x86_64-unknown-linux-gnu/llvm" "-DCMAKE_ASM_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_BUILD_TYPE=Release"
-- The C compiler identification is GNU 10.2.1
-- The CXX compiler identification is GNU 10.2.1
-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc
CMake Error:
  The detected version of Ninja (ninja 1.8.2-rs-git) is less than the version
  of Ninja required by CMake (1.3).

[klensy]

You can set download-ci-llvm = true if you don't want to play with llvm.

[daira]

I do want to potentially play with (especially, instrument) llvm.

Removing the [llvm] section, doing rm -rf build/x86_64-unknown-linux-gnu/ and running ./x.py build again fixed it. (This Stack Overflow answer gave me a hint that this might be useful thing to try, since CMakeCache.txt was under the build/x86_64-unknown-linux-gnu directory.) I also installed the ninja-build Debian package; I don't know whether that was necessary/relevant.

[jyn514]

I also installed the ninja-build Debian package

Yes, this is what fixed it - I don't know what the cargo package you installed is, but the Debian package is the one rustbuild expects.

documentation at https://rustc-dev-guide.rust-lang.org/building/how-to-build-and-run.html explaining how to install ninja, or

Hmm, I would rather have bootstrap tell you that directly when it gives an error that ninja isn't found, that seems more helpful (and easier to keep up to date). Are you interested in making a PR for that? :)

[jyn514]

The error checking for this lives in

rust/src/bootstrap/lib.rs

Line 1498 in 7a3d1a5

You should install ninja, or set `ninja=false` in config.toml in the `[llvm]` section.

.

[daira]

Yes I can make a PR for that.

[daira]

https://github.com/ninja-build/ninja/wiki/Pre-built-Ninja-packages describes how to install ninja packages on a variety of platforms, so I'll have the error message link to that.

[daira]

One thing I'm confused about is that adding

[llvm]
ninja=false

to config.toml was not sufficient to fix the problem originally. I would have expected that to work regardless of whether ninja was installed.