Incremental compilation error building rustc (`-i --keep-stage 0`)

[alexreg]

I manage to successfully build the latest master revision of rustc, but then when I try to build it incrementally as ./x.py build -i --keep-stage 0, I get the following error. Apparently, this is broken for others as well, as has been for a while.

$ ./x.py build -i --keep-stage 0
Updating only changed submodules
Submodules updated in 0.08 seconds
    Finished dev [unoptimized] target(s) in 0.45s
Warning: Using a potentially old libstd. This may not behave well.
Copying stage0 std from stage0 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Warning: Using a potentially old libtest. This may not behave well.
Copying stage0 test from stage0 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Warning: Using a potentially old librustc. This may not behave well.
Copying stage0 rustc from stage0 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Warning: Using a potentially old codegen backend. This may not behave well.
Assembling stage1 compiler (x86_64-apple-darwin)
Building stage1 std artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
   Compiling cc v1.0.28
   Compiling libc v0.2.46
   Compiling core v0.0.0 (/Users/alex/Software/rust-devel/src/libcore)
   Compiling unwind v0.0.0 (/Users/alex/Software/rust-devel/src/libunwind)
   Compiling build_helper v0.1.0 (/Users/alex/Software/rust-devel/src/build_helper)
   Compiling compiler_builtins v0.1.5
   Compiling cmake v0.1.33
   Compiling backtrace-sys v0.1.27
   Compiling std v0.0.0 (/Users/alex/Software/rust-devel/src/libstd)
   Compiling rustc_asan v0.0.0 (/Users/alex/Software/rust-devel/src/librustc_asan)
   Compiling rustc_tsan v0.0.0 (/Users/alex/Software/rust-devel/src/librustc_tsan)
   Compiling rustc-std-workspace-core v1.0.0 (/Users/alex/Software/rust-devel/src/tools/rustc-std-workspace-core)
   Compiling alloc v0.0.0 (/Users/alex/Software/rust-devel/src/liballoc)
   Compiling panic_abort v0.0.0 (/Users/alex/Software/rust-devel/src/libpanic_abort)
   Compiling rustc-demangle v0.1.10
   Compiling panic_unwind v0.0.0 (/Users/alex/Software/rust-devel/src/libpanic_unwind)
    Finished release [optimized] target(s) in 1m 12s
Copying stage1 std from stage1 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Building stage1 test artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
   Compiling getopts v0.2.17
   Compiling proc_macro v0.0.0 (/Users/alex/Software/rust-devel/src/libproc_macro)
   Compiling term v0.0.0 (/Users/alex/Software/rust-devel/src/libterm)
   Compiling test v0.0.0 (/Users/alex/Software/rust-devel/src/libtest)
    Finished release [optimized] target(s) in 21.00s
Copying stage1 test from stage1 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Building stage1 compiler artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
   Compiling nodrop v0.1.12
   Compiling cfg-if v0.1.6
   Compiling rand_core v0.3.0
   Compiling memoffset v0.2.1
   Compiling scopeguard v0.3.3
   Compiling lazy_static v1.2.0
   Compiling void v1.0.2
   Compiling stable_deref_trait v1.1.0
   Compiling either v1.5.0
   Compiling unicode-width v0.1.5
   Compiling byteorder v1.2.7
   Compiling bitflags v1.0.4
   Compiling graphviz v0.0.0 (/Users/alex/Software/rust-devel/src/libgraphviz)
   Compiling scoped-tls v1.0.0
   Compiling lazy_static v0.2.11
   Compiling termcolor v1.0.4
   Compiling rustc-demangle v0.1.10
   Compiling datafrog v2.0.1
   Compiling remove_dir_all v0.5.1
   Compiling fmt_macros v0.0.0 (/Users/alex/Software/rust-devel/src/libfmt_macros)
   Compiling rustc_fs_util v0.0.0 (/Users/alex/Software/rust-devel/src/librustc_fs_util)
   Compiling rustc-serialize v0.3.24
   Compiling quick-error v1.2.2
   Compiling scoped-tls v0.1.2
   Compiling cc v1.0.28
   Compiling libc v0.2.46
   Compiling rustc-rayon-core v0.1.2
   Compiling rustc-rayon v0.1.2
   Compiling crossbeam-utils v0.2.2
   Compiling log v0.4.6
   Compiling rustc_target v0.0.0 (/Users/alex/Software/rust-devel/src/librustc_target)
   Compiling crc32fast v1.1.2
   Compiling syntax v0.0.0 (/Users/alex/Software/rust-devel/src/libsyntax)
   Compiling arrayvec v0.4.7
   Compiling owning_ref v0.3.3
   Compiling unreachable v1.0.0
   Compiling rustc v0.0.0 (/Users/alex/Software/rust-devel/src/librustc)
   Compiling log_settings v0.1.2
   Compiling rustc_incremental v0.0.0 (/Users/alex/Software/rust-devel/src/librustc_incremental)
   Compiling rustc_metadata v0.0.0 (/Users/alex/Software/rust-devel/src/librustc_metadata)
   Compiling chalk-macros v0.1.0
   Compiling humantime v1.2.0
   Compiling rustc_driver v0.0.0 (/Users/alex/Software/rust-devel/src/librustc_driver)
   Compiling rand_xorshift v0.1.0
   Compiling rand_isaac v0.1.1
   Compiling rand_hc v0.1.0
   Compiling rand_core v0.2.2
   Compiling miniz-sys v0.1.11
   Compiling rustc-hash v1.0.1
   Compiling smallvec v0.6.7
   Compiling ena v0.11.0
   Compiling rustc_cratesio_shim v0.0.0 (/Users/alex/Software/rust-devel/src/librustc_cratesio_shim)
   Compiling lock_api v0.1.3
   Compiling polonius-engine v0.6.2
   Compiling crossbeam-epoch v0.3.1
   Compiling chalk-engine v0.9.0
   Compiling num_cpus v1.8.0
   Compiling jobserver v0.1.12
   Compiling atty v0.2.11
   Compiling backtrace v0.3.11
   Compiling memmap v0.6.2
   Compiling rand v0.5.5
   Compiling serialize v0.0.0 (/Users/alex/Software/rust-devel/src/libserialize)
   Compiling rand_chacha v0.1.0
   Compiling rand_pcg v0.1.1
   Compiling rand v0.6.1
   Compiling parking_lot_core v0.4.0
   Compiling rustc_apfloat v0.0.0 (/Users/alex/Software/rust-devel/src/librustc_apfloat)
   Compiling env_logger v0.5.13
   Compiling flate2 v1.0.6
   Compiling crossbeam-deque v0.2.0
   Compiling rustc_macros v0.1.0 (/Users/alex/Software/rust-devel/src/librustc_macros)
error[E0460]: found possibly newer version of crate `std` which `synstructure` depends onialize...
 --> src/librustc_macros/src/lib.rs:4:5
  |
4 | use synstructure::decl_derive;
  |     ^^^^^^^^^^^^
  |
  = note: perhaps that crate needs to be recompiled?
  = note: the following crate versions were found:
          crate `std`: /Users/alex/Software/rust-devel/build/x86_64-apple-darwin/stage1/lib/rustlib/x86_64-apple-darwin/lib/libstd-082537f2f194063c.rlib
          crate `std`: /Users/alex/Software/rust-devel/build/x86_64-apple-darwin/stage1/lib/rustlib/x86_64-apple-darwin/lib/libstd-082537f2f194063c.dylib
          crate `synstructure`: /Users/alex/Software/rust-devel/build/x86_64-apple-darwin/stage1-rustc/release/deps/libsynstructure-923fcb34d21ecfbf.rlib

error: aborting due to previous error

For more information about this error, try `rustc --explain E0460`.
error: Could not compile `rustc_macros`.
warning: build failed, waiting for other jobs to finish...
error: build failed
command did not execute successfully: "/Users/alex/Software/rust-devel/build/x86_64-apple-darwin/stage0/bin/cargo" "build" "--target" "x86_64-apple-darwin" "-j" "12" "--release" "--features" "" "--manifest-path" "/Users/alex/Software/rust-devel/src/rustc/Cargo.toml" "--message-format" "json"
expected success, got: exit code: 101
failed to run: /Users/alex/Software/rust-devel/build/bootstrap/debug/bootstrap build -i --keep-stage 0
Build completed unsuccessfully in 0:01:59

[estebank]

Stopgap: ./x.py clean helps you get to a good state.

[alexreg]

@estebank Yeah I know, but then I have to compile from scratch, and it takes forever. :-(

[ehuss]

I think I see one way this happens. I think the issue is that clear_if_dirty only knows about one output directory, but it does not consider the host directory (for stage1) where proc-macros and build scripts are located.

Here is one way to repro this:

1. ./x.py build
2. Make a change to libstd that will modify the SVH. I added a module, but there's probably an easier way.
3. Force a proc macro to get rebuilt: touch src/librustc_macros/src/lib.rs
4. ./x.py build

The last step should fail in stage1 to build rustc_macros because one of its dependencies (synstructure) was compiled with an older libstd, but synstructure itself was not rebuilt.

I think the solution is to clear the host directory, although care should be taken so it is not cleared multiple times if there are multiple targets. I'm not super familiar with rustbuild, but maybe @Mark-Simulacrum or @collin5 know more about this.

One way to get this particular example back on track is to rm -rf build/*/stage1-rustc/release.

Some similar issues: #49979, #50481

EDIT: Other things like codegen have to be cleaned out too, like rm -rf build/*/stage1-codegen/release.

[alexreg]

@estebank Do you know who works on incremental compilation in the compiler team? This seems like a pretty high-priority bug, so maybe we should tag them. :-)

[estebank]

I believe @alexcrichton, @nikomatsakis, @Zoxc, @michaelwoerister and @MarkSimulacrum have been involved with incremental compilation at one point or another.

[Mark-Simulacrum]

Based on the error message, this likely has nothing to do with the compiler -- it's most likely purely a problem inside rustbuild (i.e., rustbuild is broken).

[ehuss]

Yea, this is only related to rustbuild. The issue is that cargo does not track libstd as a dependency, so rustbuild needs to be careful to force rebuilding of crates when libstd changes. It does that currently (via clear_if_dirty), but it is missing the directory for proc-macros and build scripts.

[alexreg]

Ah okay. So it should be quite a straightforward fix for someone who knows how rustbuild works, eh?

[Mark-Simulacrum]

Actually quite difficult, but yes, in theory it's a fairly easy fix.