Nightlies since 2017-07-20 break when linking some libraries targeting Musl

[golddranks]

In my experience, this breaks builds that try to link the crate libpq-sys when targeting Musl. I haven't confirmed any other libraries breaking.

This is a stable-to-nightly regression. The exact breakage happens between nightlies 2017-07-19 and 2017-07-20. Here's a comparison between the commits that the two nightlies were built from: 83c659e...582af6e I haven't yet looked at the commits.

Here's a test case that runs in a docker container. See the scripts build_musl_breaks.sh and build_musl_works.sh: https://github.com/golddranks/diesel_musl_testcase

The containers use this docker image: https://github.com/golddranks/rust_musl_docker

[Mark-Simulacrum]

My guess is #43170, though I seem to recall something about that being disabled on musl... @kyrias -- is this possibly caused by that PR?

[kyrias]

Interesting. Do you have some build logs? Not currently at a computer.

Musl has had support for full RELRO for a few years, but it's certainly possible that it's related. Will look into it when I get to a computer.

[golddranks]

https://gist.github.com/golddranks/45f156e611adbcea24574e0dddaccaef Here are the logs. On the previous nightly it links the same code without problems.

[golddranks]

Hm, seems that the logs there don't show that the strlcpy that failed linking is used in libpq (and the strlcpy it attempts to use is likely to be the one declared here: https://github.com/postgres/postgres/blob/1d25779284fe1ba08ecd57e647292a9deb241376/src/port/strlcpy.c [strlcpy is an unofficial C extension that exists on some platfroms, and Postgres provides polyfills for its own code.]) some of the logs I had show that.

[Mark-Simulacrum]

Nominating for a priority and assignee from the compiler team.

[golddranks]

Btw. some of my comments may seem confused: here the problem seems to manifest when it's apparently trying to run/load the compiler plugin. I think I had earlier some code that refused to link and that showed a better error message than the testcase code. I'll see if I can find that example tomorrow.

[kyrias]

I get the same when I set -Zrelro-level=off though, so it would be quite weird if it is the problem, hm. I can't easily build a musl environment at the moment though, so doing a git bisect would be problematic for me. Any chance you could try it?

[aidanhs]

(When target is musl, the compiler plugins are built for musl too, so we need dynamically-linked musl-libc for the plugins)

(from your dockerfile)

This is not true - compiler plugins need to be compatible with the compiler, which means they link against the system libc (which is glibc on debian). As a rule of thumb, you should basically never set LD_LIBRARY_PATH to a path filled with musl libraries on debian (it's also generally not wise to set CC=musl-gcc globally on a non-musl system, just selectively set it for the commands that you need it for).

So it's not really a surprise that diesel_codegen is failing to load - musl does provide strlcpy but glibc does not, so when the diesel_codegen is loaded by rustc, the statically linked postgres library within it is expecting to be linked against musl. But compiler plugins are dynamic libraries so the executable chooses the libc, and rustc uses glibc!

root@01ae1d1cce84:/workdir# nm /workdir/target/release/deps/libdiesel_codegen-5bbd71c5b82e9c69.so | grep strlcpy
                 U strlcpy
root@01ae1d1cce84:/workdir# nm -D /lib/x86_64-linux-gnu/libc.so.6 | grep strlcpy
root@01ae1d1cce84:/workdir#

So wait, why did this 'work' previously? Well, this comment is helpful: "Full RELRO makes the dynamic linker resolve all symbols at startup". So, although I've not verified this, my strong suspicion is that strlcpy wasn't being used and so you got away with it because the symbol never needed resolving.

Why did I put 'work' in quotes above? Whether you can get away with this kind of thing is purely down to chance, and even if it compiles you can get pretty terrible subtle corruptions in data due to libc mismatches. It's good that you've hit this, because it means you now know there's a problem.

No perfect solution immediately springs to mind. Fundamentally you need two versions of libpq - one dynamically linked against glibc for the compiler plugin, one statically linked against musl for your actual binary. Unfortunately, I am not aware of any lever you can pull in pkg-config-rs that lets you set static library preferences per target (perhaps @alexcrichton has some thoughts). One thing I think you can do for the time being is to do something like LIB_PQ_DYNAMIC=true cargo build -p diesel_codegen && LIB_PQ_STATIC=true cargo build - the idea is to build up to and including diesel_codegen with a dynamic libpq, then do the last step static with the musl pkg-config (so you'll probably need to point to the different pkg-config directory for the second command).

[codyps]

Ah, the pkg-config-rs-can't cross issue. I've had a note about it in meta-rust for a while as it seemed likely we'd hit it there.

Likely solution is to allow 2 PKG_CONFIG_PATHs with the same override mechanism used for CC, etc. Probably a pkg-config-rs bug anyhow.