Regular incremental (?) build failures during stage 0 build

[RalfJung]

Regularly, when I do a git pull that skips over a few days worth of commits, or switch between branches based on different master commits, a ./x.py check fails afterwards. It fails with errors like "lifetime may not live long enough" or "this associated function takes 2 generic arguments but 1 generic argument was supplied" -- regular build errors, not ICEs. The errors are completely bogus though. Here's an example of the most recent one that I experienced today:

Checking stage0 std test/bench/example targets (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
package:   /home/r/src/rust/rustc.2/src/tools/miri/Cargo.toml
workspace: /home/r/src/rust/rustc.2/Cargo.toml
    Checking core v0.0.0 (/home/r/src/rust/rustc.2/library/core)
error[E0107]: this associated function takes 2 generic arguments but 1 generic argument was supplied
   --> library/core/tests/any.rs:145:14
    |
145 |             .provide_value::<String>(|| "bye".to_owned());
    |              ^^^^^^^^^^^^^   ------ supplied 1 generic argument
    |              |
    |              expected 2 generic arguments
    |
help: add missing generic argument
    |
145 |             .provide_value::<String, F>(|| "bye".to_owned());
    |                                    +++

error[E0107]: this function takes 2 generic arguments but 1 generic argument was supplied
   --> library/core/tests/any.rs:185:9
    |
185 |         request_ref::<T>(self)
    |         ^^^^^^^^^^^   - supplied 1 generic argument
    |         |
    |         expected 2 generic arguments
    |
help: add missing generic argument
    |
185 |         request_ref::<T, P>(self)
    |                        +++

error[E0107]: this function takes 2 generic arguments but 1 generic argument was supplied
   --> library/core/tests/any.rs:154:19
    |
154 |     assert_eq!(&**request_ref::<String>(obj).unwrap(), "hello");
    |                   ^^^^^^^^^^^   ------ supplied 1 generic argument
    |                   |
    |                   expected 2 generic arguments
    |
help: add missing generic argument
    |
154 |     assert_eq!(&**request_ref::<String, P>(obj).unwrap(), "hello");
    |                                       +++

error[E0107]: this function takes 2 generic arguments but 1 generic argument was supplied
   --> library/core/tests/any.rs:155:18
    |
155 |     assert_eq!(&*request_value::<String>(obj).unwrap(), "bye");
    |                  ^^^^^^^^^^^^^   ------ supplied 1 generic argument
    |                  |
    |                  expected 2 generic arguments
    |
help: add missing generic argument
    |
155 |     assert_eq!(&*request_value::<String, P>(obj).unwrap(), "bye");
    |                                        +++

error[E0107]: this function takes 2 generic arguments but 1 generic argument was supplied
   --> library/core/tests/any.rs:156:16
    |
156 |     assert_eq!(request_value::<u8>(obj), None);
    |                ^^^^^^^^^^^^^   -- supplied 1 generic argument
    |                |
    |                expected 2 generic arguments
    |
help: add missing generic argument
    |
156 |     assert_eq!(request_value::<u8, P>(obj), None);
    |                                  +++

error[E0107]: this function takes 2 generic arguments but 1 generic argument was supplied
   --> library/core/tests/any.rs:164:19
    |
164 |     assert_eq!(&**request_ref::<String>(&*obj).unwrap(), "hello");
    |                   ^^^^^^^^^^^   ------ supplied 1 generic argument
    |                   |
    |                   expected 2 generic arguments
    |
help: add missing generic argument
    |
164 |     assert_eq!(&**request_ref::<String, P>(&*obj).unwrap(), "hello");
    |                                       +++

error[E0107]: this function takes 2 generic arguments but 1 generic argument was supplied
   --> library/core/tests/any.rs:165:18
    |
165 |     assert_eq!(&*request_value::<String>(&*obj).unwrap(), "bye");
    |                  ^^^^^^^^^^^^^   ------ supplied 1 generic argument
    |                  |
    |                  expected 2 generic arguments
    |
help: add missing generic argument
    |
165 |     assert_eq!(&*request_value::<String, P>(&*obj).unwrap(), "bye");
    |                                        +++

error[E0107]: this function takes 2 generic arguments but 1 generic argument was supplied
   --> library/core/tests/any.rs:166:16
    |
166 |     assert_eq!(request_value::<u8>(&*obj), None);
    |                ^^^^^^^^^^^^^   -- supplied 1 generic argument
    |                |
    |                expected 2 generic arguments
    |
help: add missing generic argument
    |
166 |     assert_eq!(request_value::<u8, P>(&*obj), None);
    |                                  +++

error[E0107]: this function takes 2 generic arguments but 1 generic argument was supplied
   --> library/core/tests/any.rs:174:19
    |
174 |     assert_eq!(&**request_ref::<String>(&obj).unwrap(), "hello");
    |                   ^^^^^^^^^^^   ------ supplied 1 generic argument
    |                   |
    |                   expected 2 generic arguments
    |
help: add missing generic argument
    |
174 |     assert_eq!(&**request_ref::<String, P>(&obj).unwrap(), "hello");
    |                                       +++

error[E0107]: this function takes 2 generic arguments but 1 generic argument was supplied
   --> library/core/tests/any.rs:175:18
    |
175 |     assert_eq!(&*request_value::<String>(&obj).unwrap(), "bye");
    |                  ^^^^^^^^^^^^^   ------ supplied 1 generic argument
    |                  |
    |                  expected 2 generic arguments
    |
help: add missing generic argument
    |
175 |     assert_eq!(&*request_value::<String, P>(&obj).unwrap(), "bye");
    |                                        +++

error[E0107]: this function takes 2 generic arguments but 1 generic argument was supplied
   --> library/core/tests/any.rs:176:16
    |
176 |     assert_eq!(request_value::<u8>(&obj), None);
    |                ^^^^^^^^^^^^^   -- supplied 1 generic argument
    |                |
    |                expected 2 generic arguments
    |
help: add missing generic argument
    |
176 |     assert_eq!(request_value::<u8, P>(&obj), None);
    |                                  +++

For more information about this error, try `rustc --explain E0107`.
error: could not compile `core` due to 11 previous errors

These always go away when I do ./x.py clean. This started some time not too long before Jun 28 and has been going on since then.

[ehuss]

Can you share your config.toml? I tried building every commit with incremental for the last 3 weeks in sequence and randomly, and it didn't seem to have any issues.

The particular error you posted seems to imply that it is loading an outdated rmeta or rlib file when running the second cargo with --all-targets. I can't imagine how that might be happening, since the stage0-sysroot gets rebuilt.

[RalfJung]

Here re the non-comment non-empty lines of my config.toml:

changelog-seen = 2
[llvm]
download-ci-llvm = true
ninja = true
[build]
submodules = false
extended = true
tools = ["miri"]
[rust]
codegen-units = 0
debug-assertions = true
debug-assertions-std = false
debuginfo-level = 1
incremental = true
deny-warnings = false

[RalfJung]

Now that the bootstrap compiler was bumped, I am getting new exciting errors:

error[E0514]: found crate `unic_langid_impl` compiled by an incompatible version of rustc
 --> /home/r/.cargo/registry/src/github.com-1ecc6299db9ec823/unic-langid-macros-0.9.0/src/lib.rs:2:9
  |
2 | pub use unic_langid_impl::{subtags, LanguageIdentifier};
  |         ^^^^^^^^^^^^^^^^
  |
  = help: please recompile that crate using this compiler (rustc 1.63.0-beta.6 (efd358333 2022-07-16)) (consider running `cargo clean` first)
  = note: the following crate versions were found:
          crate `unic_langid_impl` compiled by rustc 1.63.0-beta.2 (6c1f14289 2022-06-28): /home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage0-rustc/release/deps/libunic_langid_impl-ca98cc3868906402.rmeta

error[E0514]: found crate `unicode_xid` compiled by an incompatible version of rustc
  --> /home/r/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-1.0.37/src/fallback.rs:16:5
   |
16 | use unicode_xid::UnicodeXID;
   |     ^^^^^^^^^^^
   |
   = help: please recompile that crate using this compiler (rustc 1.63.0-beta.6 (efd358333 2022-07-16)) (consider running `cargo clean` first)
   = note: the following crate versions were found:
           crate `unicode_xid` compiled by rustc 1.63.0-beta.2 (6c1f14289 2022-06-28): /home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage0-rustc/release/deps/libunicode_xid-9976faca336fa7c2.rmeta

[ehuss]

I don't suppose you could look back through your history just before the error to see what steps you took? Like did you run commands A, B, C, switch from commit X to commit Y, and then run D and immediately there was an error?

Also, it can be helpful to post the entire output from x.py, to at least know what steps it is taking. Like, did it try to build unicode_xid? Or, did for some reason, did it think it wasn't out-of-date? Why did that happen but for some reason it thought proc-macro2 was out-of-date?

Is this reproducible? Like, if you run ./x.py check again, do you get the exact same errors? If so, it could be helpful to collect some debugging information (like running with -v, using CARGO_LOG, and the contents of the deps directories).

Are you using something like rust-analyzer? If so, how is that set up? Perhaps that is interfering?

I tried for a while with your config to reproduce what you are seeing, but I haven't been able to. I'm having a hard time imagining how that could happen. For example, the second error:

1. It was compiling proc-macro2 for some proc-macro. Presumably because the compiler version changed.
2. proc-macro2 depends on unicode_xid.
3. unicode_xid was not recompiled, and rustc picked up a stale copy.

Normally, when the compiler version changes, cargo detects the fingerprint change from a hash of rustc -Vv. Since the two beta versions are different, it should have noticed that unicode_xid was out of date and rebuilt it.

Perhaps it is possible that the crate loader is rejecting the correct unicode_xid, but only reporting the error for some other stale file? For example, if for some reason there was a "stale" .rlib and a "current" .rmeta, rustc will (IIRC) prefer the rlib over the rmeta. However, your error message is mentioning the .rmeta is stale... Regardless, it could really help to know roughly which steps were taken to lead to this point. Like, are you running combinations of ./x.py check and ./x.py build? Does it happen immediately after you update git? etc.

[RalfJung]

Here is the full output that ./x.py check compiler/rustc produced:

Building rustbuild
    Finished dev [unoptimized] target(s) in 0.04s
Checking stage0 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
    Finished release [optimized + debuginfo] target(s) in 0.26s
Checking stage0 std test/bench/example targets (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
    Finished release [optimized + debuginfo] target(s) in 0.16s
Checking stage0 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
    Checking proc-macro2 v1.0.37
    Checking unic-langid-macros v0.9.0
    Checking rustc_ast v0.0.0 (/home/r/src/rust/rustc/compiler/rustc_ast)
    Checking rustc_error_messages v0.0.0 (/home/r/src/rust/rustc/compiler/rustc_error_messages)
    Checking rustc_target v0.0.0 (/home/r/src/rust/rustc/compiler/rustc_target)
    Checking rustc_feature v0.0.0 (/home/r/src/rust/rustc/compiler/rustc_feature)
    Checking rustc_log v0.0.0 (/home/r/src/rust/rustc/compiler/rustc_log)
error[E0514]: found crate `unic_langid_impl` compiled by an incompatible version of rustc
 --> /home/r/.cargo/registry/src/github.com-1ecc6299db9ec823/unic-langid-macros-0.9.0/src/lib.rs:2:9
  |
2 | pub use unic_langid_impl::{subtags, LanguageIdentifier};
  |         ^^^^^^^^^^^^^^^^
  |
  = help: please recompile that crate using this compiler (rustc 1.63.0-beta.6 (efd358333 2022-07-16)) (consider running `cargo clean` first)
  = note: the following crate versions were found:
          crate `unic_langid_impl` compiled by rustc 1.63.0-beta.2 (6c1f14289 2022-06-28): /home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage0-rustc/release/deps/libunic_langid_impl-ca98cc3868906402.rmeta

error[E0514]: found crate `unicode_xid` compiled by an incompatible version of rustc
  --> /home/r/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-1.0.37/src/fallback.rs:16:5
   |
16 | use unicode_xid::UnicodeXID;
   |     ^^^^^^^^^^^
   |
   = help: please recompile that crate using this compiler (rustc 1.63.0-beta.6 (efd358333 2022-07-16)) (consider running `cargo clean` first)
   = note: the following crate versions were found:
           crate `unicode_xid` compiled by rustc 1.63.0-beta.2 (6c1f14289 2022-06-28): /home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage0-rustc/release/deps/libunicode_xid-9976faca336fa7c2.rmeta

error: could not compile `unic-langid-macros` due to previous error
warning: build failed, waiting for other jobs to finish...
error: could not compile `proc-macro2` due to previous error
Build completed unsuccessfully in 0:00:04

Is this reproducible? Like, if you run ./x.py check again, do you get the exact same errors?

Yes. But I have no idea how to get my checkout into that state. (And this checkout has been ./x.py cleaned since then. I needed to get a patch written...)

Are you using something like rust-analyzer? If so, how is that set up? Perhaps that is interfering?

I am increasingly suspecting that this might be the case. That would explain why it never happens when I just chain some commands on the terminal. rust-analyzer does some weird things for its proc-macro support (useRustcWrapper is a thing, it injects itself deep into the build process it seems), maybe that is going wrong.

My config pretty much matches the rustc-dev-guide recommendations, I think:

{
    "rust-analyzer.rustc.source": "./Cargo.toml",
    "rust-analyzer.linkedProjects": [
        "./Cargo.toml",
        //"./src/bootstrap/Cargo.toml"
    ],
    "rust-analyzer.checkOnSave.overrideCommand": [
        "python3",
        "x.py",
        "check",
        "--json-output",
        "library/std",
        "compiler/rustc",
    ],
    // This also affects proc macros
    "rust-analyzer.cargo.buildScripts.overrideCommand": [
        "python3",
        "x.py",
        "check",
        "--json-output",
        "compiler/rustc",
    ],
    "rust-analyzer.rustfmt.overrideCommand": [
        "./build/x86_64-unknown-linux-gnu/stage0/bin/rustfmt",
        "--edition=2021"
    ],
    "files.watcherExclude": {
        "*rustc*/src/llvm-project/**": true,
        "*rustc*/build/**": true,
    },
    "files.exclude": {
        "src/llvm-project/**": true,
        "build/**": true
    },
}

Today I also globally disabled build scrips and proc macros, but it is possible that I did that after I got the error mentioned above.

Regardless, it could really help to know roughly which steps were taken to lead to this point.

What I did today in that checkout is to switch from some old branch (forgot which one) to latest master, open vscode, and I also ran ./x.py fmt. Then I wondered why vscode didn't show compilation errors for clearly ill-typed code, and ran the command I quoted above.

[bjorn3]

I had the same issue for cg_clif in rust-lang/rust-analyzer#12845

[ehuss]

OK, I think this almost certainly an issue with rust-analyzer. Here are some reproduction steps using the config you posted:

1. Exit VSCode
2. git co be9cfb307e6928600a9769695e02d6296e8e1174
3. ./x.py clean
4. ./x.py check compiler
5. git co 22d25f21dc008785f52e7c2833de4f4236b1066b
6. code .
7. Wait for r-a to do its indexing stuff.
8. ./x.py check compiler

[RalfJung]

I have not seen this problem for a while, since I disabled proc macro and build script support in RA.

[jyn514]

The upstream issue has been closed; I think this particular bug is no longer a problem. Thank you for reporting incremental bugs ❤️ I know it's easy to dismiss them as "rust-lang/rust has a weird build system" but the reports really are helpful.