Tests failing with multiple matching crates for `serde`

[phansch]

rust-lang/rust#60053 broke the Clippy tests. Specifically tests/ui/serde.rs fails with:

error[E0464]: multiple matching crates for `serde`
  --> $DIR/serde.rs:4:1
   |
LL | extern crate serde;
   | ^^^^^^^^^^^^^^^^^^^
   |
   = note: candidates:
           crate `serde`: /home/<user>/code/rust-clippy/target/debug/deps/libserde-3ec33c1c405dbd9a.rlib
           crate `serde`: /home/<user>/.rustup/toolchains/master/lib/rustlib/x86_64-unknown-linux-gnu/lib/libserde-4b8bb77bdaea80db.rlib

error[E0463]: can't find crate for `serde`
  --> $DIR/serde.rs:4:1
   |
LL | extern crate serde;
   | ^^^^^^^^^^^^^^^^^^^ can't find crate

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0463`.

I'm not really sure how fix this. Maybe somewhere in compiletest-rs? The error still happens with a clean Rust master installation. Can we somehow ignore the sysroot serde?

cc Manishearth/compiletest-rs#114, rust-lang/rust#24853

[Xanewok]

First of all, sorry for the breakage.

I'll also paste the Rust CI logs (albeit for a diferent case):

[01:25:18] +error[E0464]: multiple matching crates for `serde`
[01:25:18] +  --> $DIR/used_underscore_binding_macro.rs:11:10
[01:25:18] +   |
[01:25:18] +LL | #[derive(Deserialize)]
[01:25:18] +   |          ^^^^^^^^^^^
[01:25:18] +   |
[01:25:18] +   = note: candidates:
[01:25:18] +           crate `serde`: /checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/libserde-bb2ee07afa357acf.rlib
[01:25:18] +           crate `serde`: /checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/release/deps/libserde-2ef066c730b0470a.rlib
[01:25:18] +
[01:25:18] +error[E0463]: can't find crate for `_serde`
[01:25:18] +  --> $DIR/used_underscore_binding_macro.rs:11:10
[01:25:18] +   |
[01:25:18] +LL | #[derive(Deserialize)]
[01:25:18] +   |          ^^^^^^^^^^^ can't find crate
[01:25:18] +
[01:25:18] +error: aborting due to 2 previous errors
[01:25:18] +
[01:25:18] +For more information about this error, try `rustc --explain E0463`.
[01:25:18] +
[01:25:18] 
[01:25:18] The actual stderr differed from the expected stderr.
[01:25:18] Actual stderr saved to /checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/build/clippy-77007bd62c0dadf4/out/test_build_base/crashes/used_underscore_binding_macro.stderr
[01:25:18] To update references, run this command from build directory:
[01:25:18] tests/ui/update-references.sh '/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/build/clippy-77007bd62c0dadf4/out/test_build_base' 'crashes/used_underscore_binding_macro.rs'
[01:25:18] 
[01:25:18] error: 1 errors occurred comparing output.
[01:25:18] status: exit code: 1
[01:25:18] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools-bin/clippy-driver" "tests/ui/crashes/used_underscore_binding_macro.rs" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/build/clippy-77007bd62c0dadf4/out/test_build_base" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/build/clippy-77007bd62c0dadf4/out/test_build_base/crashes/used_underscore_binding_macro.stage-id" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/release" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/release/deps" "-Dwarnings" "-Zui-testing" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/build/clippy-77007bd62c0dadf4/out/test_build_base/crashes/used_underscore_binding_macro.stage-id.aux" "-A" "unused"

As I understand it, this is because we have 2 version of serde:

• one in the sysroot (compiler uses its own version) at

/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/libserde-bb2ee07afa357acf.rlib

• one for the tools at

/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/release/deps/libserde-2ef066c730b0470a.rlib

which is pulled for the compiletest compiler invocation (wrapped in a clippy_driver) with a
-L /checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/release/deps

I don't have a clear idea how to approach this, so to think out loud:

Obviously we can't drop the -L because clippy-driver needs their deps to run,
...but also we're interested in the serde used in the local build,
...hence we'd like to ignore sysroot serde somehow.

I don't think we can ever let local deps coalesce to the ones used in the compiler, because they may be semver-incompatible and compiler would like to have its own thread-locals and such.

Could we somehow "blacklist" sysroot crates on a lib-searching level if a rustc_private feature isn't enabled? However, if that's enabled (which I believe is for Clippy) should we set a precedence? There are both cases where sysroot is preferred and also where -L ones are preferred...
Could we differentiate sysroot crates on a semantic level (e.g. use sysroot::serde)? (rust-lang/rust#57288)

I hope solving this would also enable using serde_derive from crates.io in the compiler (same problem with shadowed sysroot libs via -L), which we had to workaround by manually expanding the macros in the rls-data (not pretty!)

FWIW RLS and Clippy work correctly with Rust patch on crates that pull serde dependency (tested in the RLS). Worst case, I can revert the patch but would really like to see a path forward with this.

EDIT: It seems we can disambiguate these by using explicit --extern flag

[Xanewok]

Indeed, with the original invocation of

LD_LIBRARY_PATH=$(rustc --print sysroot)/lib "target/debug/clippy-driver" "tests/ui/serde.rs" "-L" "/home/xanewok/repos/rust-clippy/target/debug/test_build_base" "--target=x86_64-unknown-linux-gnu" "-C" "prefer-dynamic" "-o" "/home/xanewok/repos/rust-clippy/target/debug/test_build_base/serde.stage-id" "-L" "target/debug" "-L" "target/debug/deps" "-Dwarnings" "-Zui-testing" "-L" "/home/xanewok/repos/rust-clippy/target/debug/test_build_base/serde.stage-id.aux" "-A" "unused"

I get the following error:

error[E0464]: multiple matching crates for `serde`
  --> tests/ui/serde.rs:4:1
   |
LL | extern crate serde;
   | ^^^^^^^^^^^^^^^^^^^
   |
   = note: candidates:
           crate `serde`: /home/xanewok/repos/rust-clippy/target/debug/deps/libserde-66800e6ad9385298.rlib
           crate `serde`: /home/xanewok/repos/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/libserde-bb2ee07afa357acf.rlib

error[E0463]: can't find crate for `serde`
  --> tests/ui/serde.rs:4:1
   |
LL | extern crate serde;
   | ^^^^^^^^^^^^^^^^^^^ can't find crate

but if I add the disambiguating --extern

--extern serde=/home/xanewok/repos/rust-clippy/target/debug/deps/libserde-66800e6ad9385298.rlib

this correctly compiles.