building beta with vendor=true fail due to network dependencies

[semarie]

Trying to build beta using downloaded tarball and vendor=true (in order to test and prepare distribution for OpenBSD packages), I have the following error:

$ python2.7 /data/semarie/build-rust/build_dir/rustc/rustc-beta-src/x.py build
    Updating git repository `https://github.com/rust-lang-nursery/rustfmt`
error: failed to load source for a dependency on `rustfmt`

Caused by:
  Unable to update https://github.com/rust-lang-nursery/rustfmt?branch=libsyntax#6c1de769

Caused by:
  failed to fetch into /home/semarie/.cargo/git/db/rustfmt-5390e0ead582d971

Caused by:
  attempting to update a git repository, but --frozen was specified
failed to run: /usr/local/bin/cargo build --manifest-path /data/semarie/build-rust/build_dir/rustc/rustc-beta-src/src/bootstrap/Cargo.toml --frozen
Build completed unsuccessfully in 0:00:00

My config.toml is:

[build]
rustc = "/usr/local/bin/rustc"
cargo = "/usr/local/bin/cargo"
prefix = "/data/semarie/build-rust/install_dir/beta"
docs = false
vendor = true

[dist]
src-tarball = false

[rust]
channel = "beta"
codegen-tests = false

[target.x86_64-unknown-openbsd]
llvm-config = "/usr/local/bin/llvm-config"

$ rustc -vV
rustc 1.18.0
binary: rustc
commit-hash: unknown
commit-date: unknown
host: x86_64-unknown-openbsd
release: 1.18.0
LLVM version: 4.0

$ cargo -vV
cargo 0.19.0
release: 0.19.0

The error could be about rustfmt or cargo dependency.

It seems that rls have git dependencies defined for them. From src/tools/rls/Cargo.toml:

[dependencies]
cargo = { git = "https://github.com/rust-lang/cargo" }
rustfmt = { git = "https://github.com/rust-lang-nursery/rustfmt", branch = "libsyntax" }

And even if src/Cargo.toml has a [replace] section, it is only for 0.20.0 (and not git head):

"https://github.com/rust-lang/cargo#0.20.0" = { path = "tools/cargo" }

So when cargo tries to resolv dependencies, it tries to download them from network, but fail due to --frozen.

Please note that when testing, my CARGO_HOME (~/.cargo) is empty.

[cuviper]

I just ran into this too. :/

FWIW, on rust master rls/Cargo.toml now specifies rustfmt-nightly = "0.1", but still git cargo.

[semarie]

Under nightly, I have the same: only cargo error now.

...
DEBUG:cargo::core::registry: load/match    registry https://github.com/rust-lang/crates.io-index
DEBUG:cargo::core::registry: load/match    registry https://github.com/rust-lang/crates.io-index
DEBUG:cargo::core::registry: load/match    registry https://github.com/rust-lang/crates.io-index
DEBUG:cargo::core::registry: load/missing  https://github.com/rust-lang/cargo#50b1c24d
DEBUG:cargo::sources::config: loading: https://github.com/rust-lang/cargo#50b1c24d
    Updating git repository `https://github.com/rust-lang/cargo`
DEBUG:cargo: exit_with_error; err=CliError { error: Some(CargoError(Msg("failed to load source for a dependency on `cargo`"), State { next_error:
 Some(CargoError(Msg("Unable to update https://github.com/rust-lang/cargo#50b1c24d"), State { next_error: Some(CargoError(Msg("failed to fetch into /data/semarie/build-rust/install_dir/crates/git/db/cargo-e7ff1db891893a9e"), State { next_error: Some(CargoError(Msg("attempting to update a git repository, but --frozen was specified"), State { next_error: None, backtrace: None })), backtrace: None })), backtrace: None })), backtrace:
None })), unknown: false, exit_code: 101 }
error: failed to load source for a dependency on `cargo`

Caused by:
  Unable to update https://github.com/rust-lang/cargo#50b1c24d

Caused by:
  failed to fetch into /data/semarie/build-rust/install_dir/crates/git/db/cargo-e7ff1db891893a9e

Caused by:
  attempting to update a git repository, but --frozen was specified
Traceback (most recent call last):
  File "/data/semarie/build-rust/build_dir/rustc/rustc-nightly-src/src/bootstrap/bootstrap.py", line 694, in <module>
    main()
  File "/data/semarie/build-rust/build_dir/rustc/rustc-nightly-src/src/bootstrap/bootstrap.py", line 678, in main
    bootstrap()
  File "/data/semarie/build-rust/build_dir/rustc/rustc-nightly-src/src/bootstrap/bootstrap.py", line 660, in bootstrap
    rb.build_bootstrap()
  File "/data/semarie/build-rust/build_dir/rustc/rustc-nightly-src/src/bootstrap/bootstrap.py", line 413, in build_bootstrap
    run(args, env=env, verbose=self.verbose)
  File "/data/semarie/build-rust/build_dir/rustc/rustc-nightly-src/src/bootstrap/bootstrap.py", line 142, in run
    raise RuntimeError(err)
RuntimeError: failed to run: /data/semarie/build-rust/install_dir/beta/bin/cargo build --manifest-path /data/semarie/build-rust/build_dir/rustc/rustc-nightly-src/src/bootstrap/Cargo.toml --verbose --frozen

[est31]

Either we fix alexcrichton/cargo-vendor#30 or we stop depending on git deps somehow.

[est31]

hmm digging a bit more, it seems the git dependency in Cargo.lock is due to rls requiring it in its Cargo.toml. It is being replaced in the src/Cargo.toml of this repo, but apparently Cargo still needs the git repo for some reason. So that may be a Cargo bug?

[brson]

cc @alexcrichton

This would be a pretty unfortunate thing to hit stable.

Anybody have ideas about how we can automatically test this on CI?

[est31]

Anybody have ideas about how we can automatically test this on CI?

We could create a source tarball and download the stage0, remove network by doing sudo ifdown <interface name> or something similar, and then do a bootstrap of rustc.

[est31]

A possible emergency mitigation of this issue: we could point beta to a branch/fork of rls that points to cargo as a relative path dependency instead of a git dependency. Alternatively we simply remove rls but not sure how much changes of rustbuild that entails.

Long term one can think about either giving cargo-vendor support to git deps or fixing cargo itself to worh with the current situation (not syncing git repos if they get [replace]'d), but I doubt such a fix will make it into beta in time.

[brson]

Are there any hacks we can do to automatically mangle the rls deps during vendoring? e.g. x.py vendor calls cargo vendor while doing hacks on all the resulting tomls.

[alexcrichton]

There are no hacks I know of to get this to work. If we want this fixed I believe we need to remove the rls from the repository on the beta branch.

[brson]

There's also a 1.19 milestone issue that I'd forgotten about to disable RLS packaging: #42357

[brson]

cc @rillian seems like this might impact you.