cdylib artifact size boost

[king6cong]

Hello, I find there is a huge boost in artifact size of cdylib library files in recent versions of rust. In our case, the lib doubles its size. I also find similar situations in other projects.

Here are the steps to reproduce (OS: 64bit Mac OS X 10.12.4 16E195):

git clone https://github.com/king6cong/sysinfo
cd sysinfo

on nightly-2017-06-30

rustup default nightly-2017-06-30
cargo clean
cargo build --features="c-interface"
ls -lht target/debug/libsysinfo.dylib
nm -g target/debug/libsysinfo.dylib|less

latest nightly (rustc 1.23.0-nightly (269cf50 2017-10-28))

rustup default nightly
cargo clean
cargo build --features="c-interface"
ls -lht target/debug/libsysinfo.dylib
nm -g target/debug/libsysinfo.dylib|less

Comparing the output:

# nightly-2017-06-30
751K Oct 29 17:59 target/debug/libsysinfo.dylib
# latest nightly
2.2M Oct 29 17:52 target/debug/libsysinfo.dylib

and there are many rust specific symbols in the latest nightly generated lib.

Library size matters, especially on mobile devices such as iOS and Android, definitely eager to figure out the reason :)

[nagisa]

target/debug

This is a debug build. I’m not surprised at all if it began putting more symbols in the files.

[nagisa]

cc @michaelwoerister is this fixed by one of your #[inline] instantiation fixes?

[king6cong]

@nagisa similar result with release version:

on nightly-2017-06-30

rustup default nightly-2017-06-30
cargo clean
cargo build --release --features="c-interface"
ls -lht target/release/libsysinfo.dylib
nm -g target/release/libsysinfo.dylib|less

latest nightly (rustc 1.23.0-nightly (269cf50 2017-10-28))

rustup default nightly
cargo clean
cargo build --release --features="c-interface"
ls -lht target/release/libsysinfo.dylib
nm -g target/release/libsysinfo.dylib|less

Comparing the release output:

260K Oct 29 17:59 target/release/libsysinfo.dylib
1.7M Oct 29 17:52 target/release/libsysinfo.dylib

still, many rust specific symbols in the latest nightly generated lib, while nightly-2017-06-30 works as expected.

[Mark-Simulacrum]

Nominating. Would be good to track this down, to at least diagnose the cause and add to the release notes. Probably not too high priority...

[michaelwoerister]

Thanks for the report, @king6cong! This looks indeed like a bug. We haven't changed anything in our compilation strategy the would require such a change.

is this fixed by one of your #[inline] instantiation fixes?

That might have a positive effect here, but since inline instantiation seems to have regressed already at the beginning of this year, something else must be the problem.

cc @alexcrichton

[alexcrichton]

I believe this is all debug information. On Linux tonight's nightly is 3.0MB and nightly-2017-06-30 is 2.4MB. After strip -g both are 1.4MB.

[king6cong]

@alexcrichton

I tried on Linux (x86_64 Linux 4.12.12-1-ARCH) and reproduced the problem:

on nightly-2017-06-30

rustup default nightly-2017-06-30
cargo clean
cargo build --release --features="c-interface"
ls -lht target/release/libsysinfo.so
strip -g target/release/libsysinfo.so
ls -lht target/release/libsysinfo.so
nm -g target/release/libsysinfo.so|less

on latest nightly

rustup default nightly
cargo clean
cargo build --release --features="c-interface"
ls -lht target/release/libsysinfo.so
strip -g target/release/libsysinfo.so
ls -lht target/release/libsysinfo.so
nm -g target/release/libsysinfo.so|less

Comparing the output on Linux:

# nightly-2017-06-30
# before strip
2.1M Oct 31 13:52 target/release/libsysinfo.so
# after strip
403K Oct 31 13:52 target/release/libsysinfo.so

# latest nightly
# before strip
3.0M Oct 31 13:54 target/release/libsysinfo.so
# after strip
1.4M Oct 31 13:54 target/release/libsysinfo.so

similar result for strip -x on macOS:

# nightly-2017-06-30
260K # before strip
152K # after strip
# latest nightly
1.7M # before strip
1.1M # after strip

And we can see many rust specific symbols in the striped lastest nightly version.

[alexcrichton]

Huh that's odd, I'm not sure what happened before... Now I can reproduce the results you're getting!

Somehow this is related to --crate-type rlib --crate-type cdylib (both at the same time). When I drop the rlib crate type then current nightly is as small than the older nightly.

[alexcrichton]

Bisection via nightlies points to ae98ebf...15aa15b (rust-nightly-2017-07-21 vs rust-nightly-2017-07-22) of which the most likely candidate is #43183.

@michaelwoerister I think the change here was that the set of exported symbols doesn't take the crate type as input, so we pessimistically export all symbols as if we were making an rlib, but apparently before we did a better job of internalizing things via the linker I guess?

@king6cong is it problematic to split the project into a rlib crate and a cdylib crate? (vs having both in one). Failing that we can try to fix this in rustc.

[king6cong]

@alexcrichton Great, that works!

# macOS
256K/148K # older nightly w/o strip
305K/194K # latest nightly w/o strip

# Linux
2.1M/403K # older nightly w/o strip
2.0M/393K # latest nightly w/o strip

For our project, the dylib size dropped from 23M to 9M, a huge difference!

[michaelwoerister]

This looks kind of wrong:

rust/src/librustc_trans/back/linker.rs

Lines 747 to 755 in 6713736

	fn exported_symbols(tcx: TyCtxt, crate_type: CrateType) -> Vec<String> {
	let mut symbols = Vec::new();
	let export_threshold = symbol_export::threshold(tcx);
	for &(ref name, _, level) in tcx.exported_symbols(LOCAL_CRATE).iter() {
	if level.is_below_threshold(export_threshold) {
	symbols.push(name.clone());
	}
	}

It does not take the crate type into account when computing the export threshold.