Cannot link to third-party Rust libraries in the same prefix `rustc` was installed to

I'm working on a repository for the Paludis package manager, and Rust has trouble with the standard system paths.

<snip for clarity>
root@jurily ~# cave contents dev-rust/sdl
/usr/lib64/libsdl-e351513a-0.3.2.so

root@jurily ~# cave contents dev-lang/rust
/usr/lib64/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-3574b280-0.11-pre.so
/usr/lib64/rustlib/x86_64-unknown-linux-gnu/lib/libstd-aad93cea-0.11-pre.so
/usr/lib64/liblibc-3574b280-0.11-pre.so
/usr/lib64/libstd-aad93cea-0.11-pre.so

With standard invocation:

rustc --opt-level=3 --out-dir /var/tmp/paludis/build/dev-rust-sprocketnes-scm/work/sprocketnes-scm -L /var/tmp/paludis/build/dev-rust-sprocketnes-scm/work/sprocketnes-scm nes.rs
nes.rs:11:1: 11:18 error: can't find crate for `sdl`
nes.rs:11 extern crate sdl;
          ^~~~~~~~~~~~~~~~~

So then I had the wonderful idea to feed it $LDPATH manually:

erustc () { # happy fun times
    OIFS="$IFS"
    IFS=":"
    local LDHACK=""
    local LDTOKENS=""
    read -a LDTOKENS <<< "${LDPATH}"
    for i in "${LDTOKENS[@]}"; do
        LDHACK="$LDHACK -L $i"
    done

    IFS="$OIFS"
    edo rustc --opt-level=3 --out-dir "${WORK}" -L "${WORK}" "${@}" ${LDHACK}
}

---

rustc --opt-level=3 --out-dir /var/tmp/paludis/build/dev-rust-sprocketnes-scm/work/sprocketnes-scm -L /var/tmp/paludis/build/dev-rust-sprocketnes-scm/work/sprocketnes-scm nes.rs -L /usr/local/lib -L /usr/lib64 -L /lib64 -L /usr/lib64/nss -L /usr/lib64/qt4 -L /usr/lib64/mysql -L /usr/lib64/xulrunner
nes.rs:1:1: 1:1 error: multiple dylib candidates for `std` found
nes.rs:1 //
         ^
nes.rs:1:1: 1:1 note: candidate #1: /usr/lib64/rustlib/x86_64-unknown-linux-gnu/lib/libstd-aad93cea-0.11-pre.so
nes.rs:1 //
         ^
nes.rs:1:1: 1:1 note: candidate #2: /usr/lib64/libstd-aad93cea-0.11-pre.so
nes.rs:1 //
         ^
nes.rs:1:1: 1:1 error: multiple dylib candidates for `libc` found
nes.rs:1 //
         ^
nes.rs:1:1: 1:1 note: candidate #1: /usr/lib64/liblibc-3574b280-0.11-pre.so
nes.rs:1 //
         ^
nes.rs:1:1: 1:1 note: candidate #2: /usr/lib64/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-3574b280-0.11-pre.so
nes.rs:1 //

Lose-lose. I assume there are valid reasons for this, but it's still confusing:

1. Why are there multiple shared libraries installed? If rustc needs its own set, wouldn't it make sense to link them statically?
2. Why can't rustc figure out which one to use?
3. Would it actually make a difference? They're the same version, right down to the hash.
4. If I put everything next to the "good" libstd, will it keep working? Can I count on that path to not change?

I could use a completely different directory of course, but that's an ugly workaround and would likely hurt upstream chances.

[thestinger]

1. Why are there multiple shared libraries installed? If rustc needs its own set, wouldn't it make sense to link them statically?

They're built in an earlier stage, so in theory this could break due to either bugs or stage markers. In practice, it's fine to link them and I do it in my Arch Linux package.

https://projects.archlinux.org/svntogit/community.git/tree/trunk?h=packages/rust
https://github.com/thestinger/packages/tree/master/rust-git

2. Why can't rustc figure out which one to use?

Only one of these should be in the library search path. I'm not quite sure what's going wrong, but perhaps it has to do with you adding the rustlib directory to the search path.

but perhaps it has to do with you adding the rustlib directory to the search path.

No, that's the rustc default. I added /usr/lib64. rustc is linked to the ones in /usr/lib64:

jurily@jurily ~> ldd /usr/bin/rustc | grep libstd-
    libstd-aad93cea-0.11-pre.so => /usr/bin/../lib/libstd-aad93cea-0.11-pre.so (0x00007ff49174f000)

jurily@jurily ~> ldd /usr/lib64/libsplay-c8a1a9db-0.0.so | grep libstd-
    libstd-aad93cea-0.11-pre.so => /usr/lib64/../../../../../../../usr/lib64/rustlib/x86_64-unknown-linux-gnu/lib/libstd-aad93cea-0.11-pre.so (0x00007f94d1fad000)

(libsplay was compiled without the extra paths)

[alexcrichton]

This is indeed a bit of a problem with our directory layout. And I've brought it up with @brson before. The gist of it is that:

1. The compiler's runtime libraries are in /usr/lib64.
2. The target libraries (the ones that are linked to) are in /usr/lib64/rustlib/$target/lib

This means the host library (the one in /usr/lib64) was not generated by the compiler living at /usr/bin/rustc, meaning the compiler doesn't link to it by default. When you pass -L /usr/lib64, it's found as a candidate, however.

That's the situation, and one possible solution is to assume convergence and not distribute target artifacts for the host architecture (have it all in /usr/lib64). I think we'd want to think a bit about doing that, however.

make snap-stage3 already produces a static (where it counts) executable. Use that and all problems go away.

I imagine it would also increase sanity in the build system to move to an all-static-ish bootstrap.

[lilyball]

This issue is becoming problematic for me now, because OS X 10.10 no longer searches /usr/local/lib by default for libraries. I can no longer compile rust-lua because it can't find -llua. If I add -L/usr/local/lib then I get the multiple matching dylibs error. I can compile successfully with #[link_args = "-L/usr/local/lib -llua"] but unfortunately these arguments aren't propagated to downstream dependencies of my rlib, so they don't even know to link against -llua (much less where to find it).

How did rustc handle this issue before? Did it simply not search /usr/local/lib for crates, but the system linker would go ahead and search it when linking the final result to find third-party libs?

I think rustc needs to simply prioritize dylibs found inside the target library root over anything else.

[lilyball]

An alternative might be to add a metadata bit to the crate that says "this is a host crate", and use that to skip the crate when looking for target crates.

I still think the solution is to link rustc statically. There are no advantages of dynamic linking for rustc: we're not loading LLVM lazily so startup times are not improved and the libraries must not be linked to by more than one executable. Also, rustc defaults to building rlibs, this is explicitly worked around in the build system up to and including a special compiler flag.

Moving to a prefer-static build would likely do good things to the cross-platform story as well.

For a workaround, FHS says:

Applications may use a single subdirectory under /usr/lib. If an application uses a subdirectory, all architecture-dependent data exclusively used by the application must be placed within that subdirectory.

[lilyball]

@Jurily Syntax extensions need to link against libsyntax, and with the upcoming pluggable lint stuff they may need to link against librustc as well. rustc dynamically loads these things, so dynamic linking is required for both rustc and syntax extensions.

[lilyball]

Maybe we should just use rustlib for both target and host crates from the stdlib? We can use separate directories for host and target libraries.

[lilyball]

FWIW I'm currently testing code to do the prioritize-target-lib-path approach right now.

Hmm. In that case, rustc should be built one more time on top of the set we want to install, but that would make a lot of people unhappy. Can we separate install-fast and install-careful targets?

[lilyball]

@Jurily The compiler is built on top of what we install already. The difference is there's target crates and host crates. We install both, just in separate directories.

[lilyball]

Prioritizing the target lib path makes -L/usr/local/lib work, but unfortunately Rust doesn't embed search paths like that into rlib crate metadata, so rust-lua still has a problem (which is that any client of the liblua rlib has to independently pass the -L/usr/local/lib flag to find -llua).