Switching cwd breaks usage of custom target specifications

[robert-w-gries]

Per japaric/xargo#186 (comment), the following no longer works:

$ cargo new --lib foo

$ cd foo

$ echo '#![no_std]' > src/lib.rs

$ cargo add heapless

$ rustc -Z unstable-options --target x86_64-unknown-linux-gnu --print target-spec-json > foo.json

$ cargo build --target foo
   Compiling untagged-option v0.1.1
error: Error loading target specification: Could not find specification for target "foo"

The following conversation was taken from #4865:

@SergioBenitez

Shouldn't cargo be setting cwd to the workspace root (here just the crate root, foo) which is exactly where the target file foo.json is? If so, shouldn't rustc find the file and build properly? Or has something changed in rustc so that it no longer looks for target files in the cwd?

@alexcrichton

oh I believe the issue there is that you're compiling crates from crates.io and Cargo is now switching the cwd even for those crates (as their own workspace root is themselves in Cargo's $HOME/.cargo cache

Ah yeah so to clarify I believe everything is working correctly in cargo/rustc and this is an intentional change on Cargo's behalf. (breaking in the technical/pedantic sense)

Workaround

In order to fix the cargo build error, you need to manually set RUST_TARGET_PATH.

export RUST_TARGET_PATH=`pwd`

Remaining Discussion

I think the following question remains:

Why are users of custom target specifications required to set RUST_TARGET_PATH every time cargo build is invoked?

The error message does not make it clear why the build is failing. Also, it is unintuitive to set an environment variable pointing to the current directory for the build to succeed.

If the current behavior will remain in the long-term, there should probably be documentation somewhere that addresses the breaking change. Also, I think the Cargo error message should hint that RUST_TARGET_PATH is pointing to the wrong directory.

[phil-opp]

We could use absolute paths for target triples, i.e. cargo build --target /home/xyz/foo/target.json. Then the target triple would be found independent of the current working directory. We could even convert all target file paths to absolute paths in cargo, so that cargo build --target target.json works too from the home/xyz/foo folder.

Unfortunately, absolute paths don't work today, since the target triple path is just pushed to the sysroot path:

// in src/librustc/session/filesearch.rs

pub fn relative_target_lib_path(sysroot: &Path, target_triple: &str) -> PathBuf {
    let mut p = PathBuf::from(find_libdir(sysroot).as_ref());
    assert!(p.is_relative());
    p.push(RUST_LIB_DIR);
    p.push(target_triple);        <-- problem if the path is absolute
    p.push("lib");
    p
}

If target_triple is absolute, the sysroot is completely eliminated from the path on push, which means that rustc looks for the sysroot in the project folder. So we should remove the root component from the path before pushing (i.e. convert /home/xyz/foo/target.json to home/xyz/foo/target.json). An alternative might be to store it under a hash of the target.json instead of depending on the passed path. This would also solve rust-lang/rust#24666 and i think this issue too: japaric/xargo#44 (comment).

[phil-opp]

@alexcrichton @carols10cents @matklad

This issue is affecting lots of people (everyone who uses custom targets) and has not received any attention yet. So I thought I'd ping you as members of cargo team, maybe you could take a look.

[alexcrichton]

@phil-opp I think it's unlikely we'll be reverting to the previous behavior, at this point I think it's best to consider how to change custom targets such that they work with Cargo "by default", although I'm not quite sure what that would mean.

[alexcrichton]

One option for Cargo is to automatically pass an absolute path for anything it detects as JSON (aka ends with json and a file exists in the cwd), and then we could fix rustc's logic to work with absolute paths.

[SergioBenitez]

@alexcrichton Why not do something similar, but limit it to a file named $target.json? I believe this would restore the previous functionality.

[alexcrichton]

@SergioBenitez yep that'd also work!

[phil-opp]

@alexcrichton Thanks for the quick reply and showing a path forward!

[phil-opp]

Seems like the target triple is stored in the metadata of libcore rlib (created by xargo). If the targets don't match a "couldn't find crate core with expected target triple" error occurs.

This means that we already have to compile the sysroot for the absolute target and can't just convert the triple to an absolute path in cargo. So I try to adjust xargo now.