--emit=dep-info or some flag should include rlibs & externally linked files that will be used

[cramertj]

Currently, --emit=dep-info only shows source files that are included in the current crate. There should be another mode (or the behavior should be extended) to include all files that will be used by the equivalent non-dep-info invocation of rustc. This includes rlibs of all crates used by the build, both explicitly in --extern and brought in through lib search paths, as well as external object files etc. that may be linked in. This is necessary in order to add Rust support to remote compilers and compiler caches, which depend on the exact hashes of all files used in the build.

Note that AFAICT sccache currently works around this by relying on some behaviors of cargo and depending on all the libraries contained in link paths. (@luser / @alexcrichton please correct me if I'm wrong here). I'd like to instead ensure that we can ship the exact set of files needed for compilation, both of libraries, and for the final link step.

i'm sure what I've written above is probably partially incorrect and misrepresentative somehow, but hopefully others will correct my errors and we can work towards a solution together :)

cc @juliehockett @petrhosek

[alexcrichton]

It's right yeah that sccache implicitly relies on this, where if A depends on B which depends on C, sccache's hash for A only includes B, but B's own hash transitively includes C so it ends up working out in practice.

I don't think this is overly hard to do in rustc in principle, but I suspect some refactoring would be necessary to achieve this without engaging the whole compiler on --emit dep-info. Seems like a reasonable addition to me to add! This may need to be coupled with a Cargo change, but we can handle that then.

[luser]

First off, just to be clear, sccache doesn't attempt to cache rustc invocations that output anything but an rlib, so it does not currently attempt to deal with linker inputs.

sccache gets the list of static libs / rlibs that are involved in compilation by parsing the rustc commandline:
https://github.com/mozilla/sccache/blob/05bd9c7944d8c7c6e11f994c8cb5246f7638d887/src/compiler/rust.rs#L787
https://github.com/mozilla/sccache/blob/05bd9c7944d8c7c6e11f994c8cb5246f7638d887/src/compiler/rust.rs#L754

It uses the search paths (also from the commandline) to locate all static libs:
https://github.com/mozilla/sccache/blob/05bd9c7944d8c7c6e11f994c8cb5246f7638d887/src/compiler/rust.rs#L889

I don't think adding rlibs etc to the dep-info output would break sccache in any way--it currently just parses the first line of the file (sort of cheating, but rustc lists all source files as dependencies of the dep file itself there) and hashes all files listed there to use as inputs to the overall hash key. This would change the hash for existing compilations, and sccache would be doing redundant work but it should otherwise be harmless.

[cramertj]

sccache gets the list of static libs / rlibs that are involved in compilation by parsing the rustc commandline:

Yeah, this is just insufficient since everything added to a link search path (-L) could be depended on by the build, but isn't necessarily. Specifically which libraries are necessary is the information we need.

[luser]

This is fair. What we're doing in sccache is more of a heuristic and not 100% precise, but it covers the "rustc as invoked by cargo" use case which is what I aimed to support initially.

[tmandry]

I've been looking into this. One question is how hard it will be to successfully do this across multiple crates that depend on each other. e.g. I want to run rustc with --dep-info-only on crate A, then crate B which depends on crate A.

I know that in the case of cargo check, for instance, we obviously emit enough metadata for our dependent crates. Ideally, I don't want to go through a full cargo check; I'd rather skip the analysis phase altogether. The question is what kind of metadata we need for our dependent crates to also do a --dep-info-only run (we need enough to get through name resolution), and how emit only that.

[bjorn3]

This is supported on nightly with -Zbinary-dep-depinfo. This feature is tracked by #63012.

[zopsicle]

This is supported on nightly with -Zbinary-dep-depinfo. This feature is tracked by #63012.

AFAICT this only includes dependencies on crates, not on libraries linked with e.g. build.rs println!("cargo::rustc-link-lib=static=DirectXTex"); or #[link(name = "SDL2")] unsafe extern {}. #58393 and rust-lang/cargo#12502 were closed as duplicates of this issue, so this use case is considered in scope by some.

Implementing this probably requires cooperation from the linker. But I don’t think I’ve ever seen a linker generate a depfile. GNU ld does accept a --dependency-file option to generate a depfile, but it only seems to include transitive dependencies. So e.g. ld --dependency-file=depfile -lSDL2 main.o won’t include libSDL2.a in the depfile, because it is listed as a direct dependency on the ld command line. 💀