Tracking issue for linker performance

[bjorn3]

cg_llvm

simple-raytracer $ for i in 0 1 2 3 4 5 6 7 8 9; do rm -r target/debug/{deps/libraytracer-*,incremental}; cargo rustc --bin main -- -Ztime-passes 2>/dev/null | grep linking; done
  time: 0.792; rss: 202MB       linking
  time: 0.789; rss: 202MB       linking
  time: 0.793; rss: 202MB       linking
  time: 0.789; rss: 203MB       linking
  time: 0.795; rss: 202MB       linking
  time: 0.788; rss: 203MB       linking
  time: 0.786; rss: 206MB       linking
  time: 0.796; rss: 204MB       linking
  time: 0.790; rss: 201MB       linking
  time: 0.795; rss: 203MB       linking

cg_clif

simple-raytracer $ for i in 0 1 2 3 4 5 6 7 8 9; do rm -r target/x86_64-unknown-linux-gnu/debug/{deps/libraytracer-*,incremental}; CHANNEL=release ../cargo.sh rustc --bin main -- -Ztime-passes 2>/dev/null | grep linking; done
  time: 0.923; rss: 150MB       linking
  time: 0.924; rss: 150MB       linking
  time: 0.915; rss: 151MB       linking
  time: 0.925; rss: 151MB       linking
  time: 0.915; rss: 152MB       linking
  time: 0.920; rss: 151MB       linking
  time: 0.923; rss: 150MB       linking
  time: 0.913; rss: 150MB       linking
  time: 0.922; rss: 150MB       linking
  time: 0.922; rss: 151MB       linking

The linker is responsible for more than half of the compilation time of simple-raytracer's lib and bin combined when using cg_clif.

[philipc]

Can you compare with the object::write backend? It doesn't use function sections so linking may be faster.

[bjorn3]

On macOS I got the following results:

	cg_llvm	cg_clif (faerie)	cg_clif (object + faerie sysroot)	cg_clif (object + object sysroot)
link time	~0.4s	~3.7s	~2.2s	~1.2s
bin size	1.8MB	5.9MB	23MB	29MB
dep dir size	65MB	52MB	68MB	75MB

Conclusion: on macOS using object::write gives faster link times, but it comes at the cost of extra disk usage.

It doesn't use function sections so linking may be faster.

That prevents --gc-sections (which is used by cg_llvm) from functioning, right?

Edit: made best results for measured values bold.

[philipc]

That prevents --gc-sections (which is used by cg_llvm) from functioning, right?

Mach-O doesn't have function sections. However, I think object::write isn't setting SUBSECTIONS_VIA_SYMBOLS either. We can probably safely set that without any other changes.

For ELF, lack of function sections will prevent --gc-sections. I haven't looked at what cg_llvm does, but I wasn't expecting function sections to be used for debug builds. BTW object::write does have support for function sections, but cranelift-object isn't using it. I want to make it a configuration option.

[bjorn3]

I haven't looked at what cg_llvm does, but I wasn't expecting function sections to be used for debug builds.

It does.

$ echo "pub fn a() {} pub fn b() { loop {} }" | rustc --target x86_64-unknown-linux-gnu --crate-type lib - -g
$ ar x librust_out.rlib
$ bingrep rust_out.*.rcgu.o
[...]
Syms(12):
               Addr   Bind       Type        Symbol                                Size    Section                                       Other  
                 0    LOCAL      NOTYPE                                            0x0                                                   0x0    
                 0    LOCAL      FILE        rust_out.3a1fbbbh-cgu.0               0x0     ABS                                           0x0    
                 0    LOCAL      SECTION                                           0x0     .text._ZN8rust_out1a17h4d379d7d965ecdb5E(3)   0x0    
                 0    LOCAL      SECTION                                           0x0     .text._ZN8rust_out1b17hebd538271c44b443E(4)   0x0    
                 0    LOCAL      SECTION                                           0x0     .debug_str(6)                                 0x0    
                 0    LOCAL      SECTION                                           0x0     .debug_abbrev(7)                              0x0    
                 0    LOCAL      SECTION                                           0x0     .debug_info(8)                                0x0    
                 0    LOCAL      SECTION                                           0x0     .debug_ranges(10)                             0x0    
                 0    LOCAL      SECTION                                           0x0     .debug_line(20)                               0x0    
                 0    GLOBAL     FUNC        _ZN8rust_out1a17h4d379d7d965ecdb5E    0x1     .text._ZN8rust_out1a17h4d379d7d965ecdb5E(3)   0x0    
                 0    GLOBAL     FUNC        _ZN8rust_out1b17hebd538271c44b443E    0x4     .text._ZN8rust_out1b17hebd538271c44b443E(4)   0x0    
                 0    WEAK       OBJECT      __rustc_debug_gdb_scripts_section__   0x22    .debug_gdb_scripts(5)
[...]

I want to make it a configuration option.

You could put SUBSECTIONS_VIA_SYMBOLS behind the same option, as they have the same purpose.

[bjorn3]

On Linux I get the following results:

	cg_llvm	cg_clif (faerie)	cg_clif (object + faerie sysroot)	cg_clif (object + object sysroot)
link time	~0.8s	~0.9s	~0.6s	~0.3s!!!
bin size	15MB	10MB	24MB	28MB
dep dir size	91MB	66MB	71MB	74MB

Conclusion: on Linux using object still causes more disk usage, but unlike on macOS it causes even faster link times than using cg_llvm.

[philipc]

object with function/data sections is similar to faerie.

Those times and sizes are misleading because cg_llvm generates proper debug info. The stripped cg_llvm binary is much smaller (1.6MB vs 4.5MB), with the major differences being in .rodata (40KB vs 1.1MB), .text (726KB vs 1.84MB), and symtab/strtab.

A lot of the .rodata seems to be trap messages. These also have long symbol names. Maybe we should omit symbols for these?

[bjorn3]

A lot of the .rodata seems to be trap messages. These also have long symbol names. Maybe we should omit symbols for these?

I could also hash the names.

[bjorn3]

https://github.com/michaeleisel/zld sounds promising as replacement linker on macOS. It is basically an optimized version of the official apple linker.

[bjorn3]

4d226e8 and 98a073c reduced the symbol sizes for trap messages and vtables by hashing cq skipping the Some(Binder( and )) part. Bin size is now 25MB and dep dir size is now 71MB.

[bjorn3]

Current status:

	cg_llvm	cg_clif	cg_clif with per function sections
link time	~0.65s	~0.4s	~0.6s
bin size	15MB	26MB	15MB
dep dir size	107MB	101MB	95MB

cg_llvm:

time: 0.670; rss: 120MB run_linker
time: 0.660; rss: 122MB run_linker
time: 0.667; rss: 122MB run_linker
time: 0.688; rss: 122MB run_linker
time: 0.658; rss: 121MB run_linker
time: 0.652; rss: 121MB run_linker
time: 0.643; rss: 121MB run_linker
time: 0.694; rss: 122MB run_linker
time: 0.661; rss: 121MB run_linker
time: 0.660; rss: 122MB run_linker
time: 0.654; rss: 122MB run_linker

cg_clif:

time: 0.384; rss: 138MB run_linker
time: 0.388; rss: 130MB run_linker
time: 0.383; rss: 132MB run_linker
time: 0.382; rss: 130MB run_linker
time: 0.383; rss: 127MB run_linker
time: 0.384; rss: 131MB run_linker
time: 0.393; rss: 132MB run_linker
time: 0.400; rss: 128MB run_linker
time: 0.383; rss: 132MB run_linker
time: 0.383; rss: 128MB run_linker
time: 0.388; rss: 130MB run_linker

cg_clif with per function sections:

time: 0.591; rss: 129MB run_linker
time: 0.580; rss: 130MB run_linker
time: 0.579; rss: 128MB run_linker
time: 0.604; rss: 128MB run_linker
time: 0.586; rss: 131MB run_linker
time: 0.633; rss: 128MB run_linker
time: 0.598; rss: 128MB run_linker
time: 0.588; rss: 128MB run_linker
time: 0.579; rss: 131MB run_linker
time: 0.604; rss: 131MB run_linker
time: 0.584; rss: 128MB run_linker

cg_clif has faster linking times both with and without per function sections. Executable sizes match between cg_llvm (which always has per function sections) and cg_llvm with per function sections. In all cases the deps dir is smaller for cg_clif than for cg_llvm.

I will leave this issue open to track any future link time improvements.

[bjorn3]

cg_clif causes worse linker times for Bevy when using lld than cg_llvm: https://hackmd.io/@bjorn3/BJCQC1eKD The difference is smaller when bevy is compiled as dylib, but there is still a difference.

[bjorn3]

Current status (f73b0b1):

	cg_llvm	cg_clif	cg_clif with per function sections
link time	~0.65s	~0.35s	~0.55s
bin size	19MB	20MB	12MB
dep dir size	173MB	187MB	152MB