ThinLTO and `-o` compiler option can lead to duplicate object file inclusion in staticlib

[nikhilm]

I've verified this on macOS 10.14/Xcode 10.3 as well as Ubuntu 14.04/GCC 4.8
rustc -V: rustc 1.36.0-nightly (50a0def 2019-05-21) (although the code i mention is the same in master)

tl;dr object files (.o) prefixes are inferred from -o, but lto object file exclusion uses crate name. External build systems which use -o can face build failures.

Setup:

1. A staticlib that depends on a rlib (standard crate)
2. Build using lto=thin.
3. The rlib is built with -o that causes it to have a custom output name.

Sample: https://gist.github.com/nikhilm/a72d89002553ecab4511fbe77df223cc
Run cargo build --release -vv to get the command lines.

First, just building the dependent rlib (by copying the cargo invocation, no -o passed)

$ ar t target/release/deps/libbitflags-9e9e338798b0f64c.rlib
__.SYMDEF
bitflags-9e9e338798b0f64c.bitflags.e90hrf8b-cgu.0.rcgu.o
rust.metadata.bin
bitflags-9e9e338798b0f64c.bitflags.e90hrf8b-cgu.0.rcgu.bc.z

notice how the prefix is bitflags

Now, tweaking the rustc command for bitflags to add a -o and running ar on that file:

__.SYMDEF
libbitflags-9e9e338798b0f64c.bitflags.e90hrf8b-cgu.0.rcgu.o
rust.metadata.bin
libbitflags-9e9e338798b0f64c.bitflags.e90hrf8b-cgu.0.rcgu.bc.z

notice how the prefix is libbitflags.

This is fine. But when compiling a staticlib comprising this rlib, there is a special case when lto is enabled.

rust/src/librustc_codegen_llvm/back/archive.rs

Line 143 in 6c18a3d

if lto && fname.starts_with(&obj_start) && fname.ends_with(".o") {

, where the obj_start is the name of the crate (bitflags).

Now, if we take a look at the cargo generated staticlib, running ar -t target/release/libhello_world.a:

__.SYMDEF
hello_world-bdf2887e2bd4c34b.3v2guokzdntel6g4.rcgu.o
hello_world-bdf2887e2bd4c34b.alloc.a2n556im-cgu.0.rcgu.o
hello_world-bdf2887e2bd4c34b.backtrace_sys.cepblb1k-cgu.0.rcgu.o
hello_world-bdf2887e2bd4c34b.bitflags.e90hrf8b-cgu.0.rcgu.o
hello_world-bdf2887e2bd4c34b.core.ec2x3rcv-cgu.0.rcgu.o
hello_world-bdf2887e2bd4c34b.hashbrown.894rc464-cgu.0.rcgu.o
hello_world-bdf2887e2bd4c34b.hello_world.7socix02-cgu.0.rcgu.o
hello_world-bdf2887e2bd4c34b.hello_world.7socix02-cgu.1.rcgu.o
hello_world-bdf2887e2bd4c34b.libc.5av1bc7k-cgu.0.rcgu.o
hello_world-bdf2887e2bd4c34b.panic_unwind.354k3km2-cgu.0.rcgu.o
hello_world-bdf2887e2bd4c34b.rustc_demangle.cq0g9zqx-cgu.0.rcgu.o
hello_world-bdf2887e2bd4c34b.rustc_std_workspace_alloc.eehx1vq7-cgu.0.rcgu.o
hello_world-bdf2887e2bd4c34b.rustc_std_workspace_core.bloqy3w9-cgu.0.rcgu.o
hello_world-bdf2887e2bd4c34b.std.7g3y6w1q-cgu.0.rcgu.o
hello_world-bdf2887e2bd4c34b.unwind.5mw5tvii-cgu.0.rcgu.o
...

However, if we tweak the rustc for the staticlib to link against the rlib generated using -o
If this is different from the crate name, we are in trouble because the starts_with match fails (libbitflags does not start with bitflags).

__.SYMDEF
libhello_world-bdf2887e2bd4c34b.3v2guokzdntel6g4.rcgu.o
libhello_world-bdf2887e2bd4c34b.alloc.a2n556im-cgu.0.rcgu.o
libhello_world-bdf2887e2bd4c34b.backtrace_sys.cepblb1k-cgu.0.rcgu.o
libhello_world-bdf2887e2bd4c34b.bitflags.e90hrf8b-cgu.0.rcgu.o  <--------
libhello_world-bdf2887e2bd4c34b.core.ec2x3rcv-cgu.0.rcgu.o
libhello_world-bdf2887e2bd4c34b.hashbrown.894rc464-cgu.0.rcgu.o
libhello_world-bdf2887e2bd4c34b.hello_world.7socix02-cgu.0.rcgu.o
libhello_world-bdf2887e2bd4c34b.hello_world.7socix02-cgu.1.rcgu.o
libhello_world-bdf2887e2bd4c34b.libc.5av1bc7k-cgu.0.rcgu.o
libhello_world-bdf2887e2bd4c34b.panic_unwind.354k3km2-cgu.0.rcgu.o
libhello_world-bdf2887e2bd4c34b.rustc_demangle.cq0g9zqx-cgu.0.rcgu.o
libhello_world-bdf2887e2bd4c34b.rustc_std_workspace_alloc.eehx1vq7-cgu.0.rcgu.o
libhello_world-bdf2887e2bd4c34b.rustc_std_workspace_core.bloqy3w9-cgu.0.rcgu.o
libhello_world-bdf2887e2bd4c34b.std.7g3y6w1q-cgu.0.rcgu.o
libhello_world-bdf2887e2bd4c34b.unwind.5mw5tvii-cgu.0.rcgu.o
libbitflags-9e9e338798b0f64c.bitflags.e90hrf8b-cgu.0.rcgu.o    <------- symbols going to conflict with the above
...

[michaelwoerister]

Thanks a lot for the bug report, @nikhilm! And great debugging too!

This does look problematic indeed. It might be easy to fix by making the compiler behave consistently when -o is passed.

[michaelwoerister]

Nominating for prioritization.

[pnkfelix]

triage: P-high, unnominating. Assigning to @michaelwoerister with intention that they choose if they'd prefer to fix it themselves, or to mentor someone else in authoring a fix.

[pnkfelix]

nominating for discussion at T-compiler meeting, in hopes of finding someone to address this.

[michaelwoerister]

I'll look into this now.

[pnkfelix]

great, I'll remove the nomination label.