`x.py` panics when trying to build "stage3" compiler

[camelid]

I don't think there even is a stage3, so I think this is a bug in bootstrap. I haven't encountered this before. Rerunning the command produces the same panic. Despite the panic, rustc +r2stage2 --help works, presumably because the build has finished and x.py is trying to do extra steps by building a non-existent stage.

In case it helps, my branch's merge-base is 00d5e42. My branch only modifies the Miri engine, so I'm pretty sure it's unrelated to this bug.

$ x build --stage 2 compiler/rustc
[...]
Copying stage1 rustc from stage1 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Assembling stage2 compiler (x86_64-apple-darwin)
Uplifting stage1 std (x86_64-apple-darwin -> x86_64-apple-darwin)
Copying stage2 std from stage1 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Uplifting stage1 rustc (x86_64-apple-darwin -> x86_64-apple-darwin)
Copying stage2 rustc from stage1 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Assembling stage3 compiler (x86_64-apple-darwin)
thread 'main' panicked at 'fs::read(stamp) failed with No such file or directory (os error 2) ("/Users/user/rust2/build/x86_64-apple-darwin/stage2-rustc/x86_64-apple-darwin/release/.librustc.stamp")', src/bootstrap/lib.rs:1296:24
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Build completed unsuccessfully in 0:29:07

Backtrace

thread 'main' panicked at 'fs::read(stamp) failed with No such file or directory (os error 2) ("/Users/user/rust2/build/x86_64-apple-darwin/stage2-rustc/x86_64-apple-darwin/release/.librustc.stamp")', src/bootstrap/lib.rs:1296:24
stack backtrace:
   0: rust_begin_unwind
             at /rustc/e784c962ea252f0874a4305168077e7048cb39e9/library/std/src/panicking.rs:517:5
   1: core::panicking::panic_fmt
             at /rustc/e784c962ea252f0874a4305168077e7048cb39e9/library/core/src/panicking.rs:101:14
   2: bootstrap::Build::read_stamp_file
             at ./src/bootstrap/lib.rs:1296:24
   3: <bootstrap::compile::Assemble as bootstrap::builder::Step>::run
             at ./src/bootstrap/compile.rs:1107:27
   4: bootstrap::builder::Builder::ensure
             at ./src/bootstrap/builder.rs:1592:23
   5: bootstrap::builder::Builder::compiler
             at ./src/bootstrap/builder.rs:627:9
   6: <bootstrap::compile::Assemble as bootstrap::builder::Step>::make_run
             at ./src/bootstrap/compile.rs:1034:30
   7: bootstrap::builder::StepDescription::maybe_run
             at ./src/bootstrap/builder.rs:175:13
   8: bootstrap::builder::StepDescription::run
             at ./src/bootstrap/builder.rs:231:21
   9: bootstrap::builder::Builder::run_step_descriptions
             at ./src/bootstrap/builder.rs:619:9
  10: bootstrap::builder::Builder::execute_cli
             at ./src/bootstrap/builder.rs:599:9
  11: bootstrap::Build::build
             at ./src/bootstrap/lib.rs:624:13
  12: bootstrap::main
             at ./src/bootstrap/bin/main.rs:33:5
  13: core::ops::function::FnOnce::call_once
             at /rustc/e784c962ea252f0874a4305168077e7048cb39e9/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Build completed unsuccessfully in 0:00:05

[Mark-Simulacrum]

cc @jyn514 -- I think this is caused by the move of compiler assembly into the rustc step (or similar).

I think the fix here is likely to make sure the code is reading from stage1-rustc's stamp file if it's reusing those libraries. We may also prefer to just skip the copy entirely -- stage3/bin/rustc is not generally useful -- but that should get gated on full-bootstrap if it still exists (I forget).

[jyn514]

stage3/bin/rustc is not generally useful -- but that should get gated on full-bootstrap if it still exists

My understanding is that stage3 is still useful for ensuring we have reproducible builds. Yes, full-bootstrap still exists (although I'm not sure what the use case is).

[RalfJung]

Independent of whether stage 3 exists or not, the command here explicitly asks for stage 2 so something seems to have gone wrong when stage 3 steps are happening.

This problem still persists in current master, I just ran into it with ./x.py build compiler/rustc --stage 2.

[jyn514]

@RalfJung for context this is likely caused by #89759 - I'm ok with reverting that for now if bootstrap has other bugs, but I don't think assembling the compiler before you've asked for it is inherently a bug.

[Mark-Simulacrum]

Since you've already built the stage 3 compiler (in all but copying it into place), it makes some sense to assemble it. At least, this more closely matches the likely-desired behavior at earlier stages, IMO.

I'd be fine with us just adding a stage == 3 and not full-bootstrap bailout to this code for now, which should mitigate this error, or a more complete patch -- I think reverting #89759 is probably not the right patch, though.

[RalfJung]

Since you've already built the stage 3 compiler (in all but copying it into place), it makes some sense to assemble it.

Have I? All I want is a stage 2 that I can use to build Miri. build --stage 2 compiler/rustc seemed like the easiest way to do that. I guess this will also build the 'stage2 compiler artifacts' (aka the stage 3 compiler), but that is just a copy so I don't mind this extra work.

From what you said it sounds like ./x.py build library/test --stage 2 should suffice for my usecase?

[RalfJung]

From what you said it sounds like ./x.py build library/test --stage 2 should suffice for my usecase?

Hm, no, it does not since Miri needs the rustc libs themselves. build --stage 2 compiler/rustc does the extra work of putting those into the stage 2 sysroot, it seems.

[Mark-Simulacrum]

Without full-bootstrap, stages >= 2 don't actually have distinct libraries (we uplift the stage 1 compiler artifacts). So in some sense any stage is "built" and copy-able into the relevant directories once you have a stage2/bin/rustc.

But, yes, you do need to copy the rustc libraries into the appropriate sysroot. I suspect that your particular use case may be painful/unworkable with the stage 3 assembly step being broken (as noted in this issue), but am not sure.

[RalfJung]

Well, running build --stage 2 compiler/rustc and then just ignoring the fact that it panics seems to be good enough to build Miri. But yeah something is wrong, as the panic shows.^^

[smijolovic]

What is the proper way to build the stage2 compiler now?
With 1.57.0,
x.py build --stage 2 compiler/rustc

Copying stage1 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Assembling stage2 compiler (x86_64-unknown-linux-gnu)
Uplifting stage1 std (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage2 std from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Uplifting stage1 rustc (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage2 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Assembling stage3 compiler (x86_64-unknown-linux-gnu)
thread 'main' panicked at 'fs::read(stamp) failed with No such file or directory (os error 2) ("/home/chaasmadmin/nimbus8/rust/rust-chaasm-1.57.0/rust/build/x86_64-unknown-linux-gnu/stage2-rustc/x86_64-unknown-linux-gnu/release/.librustc.stamp")', src/bootstrap/lib.rs:1296:24
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
Build completed unsuccessfully in 5:08:19

[RalfJung]

x.py build --stage 2 compiler/rustc works for me, you just have to ignore the error.

[camelid]

Yeah, the error is just annoying; it doesn't actually break anything.