Linker hangs with no output 9/10 times.

[hydra]

When building various example projects, I find that the rust-lld.exe linker will just stall instead of creating the .elf file. When it works, it takes about 0.4seconds, most of the time it fails and you have to ctrl+c the command and retry, it's obviously ridiculously frustrating, especially during a pair programming session.

To make sure there was nothing odd on my PC that might be affecting it, I did a complete fresh install of the OS, rust and the tools, yet the same thing still happens.

I was able to record detailed logs using ProcessMonitor of the failure. Attached to this issue are 3 files:

1. good.CSV, a log from when it works.

good.CSV

The output was this:

PS D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor> cargo build -j1
   Compiling stm32f4_pwm_input v0.1.0 (D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor)
    Finished dev [unoptimized + debuginfo] target(s) in 0.70s

2. bad.CSV and 3) bad-after-killing.CSV. A trace was started and saved as bad.CSV when rust-lld.exe hanged, then a new trace was started, ctrl+c was pressed and the new trace was saved as bad-after-killing.CSV so that you can see what happens before and after ctrl+c was pressed.

bad.CSV
bad-after-killing.CSV

The output, when it fails is this:

...
Compiling stm32f4_pwm_input v0.1.0 (D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor)
    Building [=========================> ] 98/99: stm32f4_pwm_input(bin)

I'm compiling for an STM32F446RE CPU.

I tried with stable and with nightly

Meta

Project source: rtic-examples.zip

Nightly:

> rustup show
Default host: x86_64-pc-windows-gnu
rustup home:  D:\Users\Hydra\.rustup

installed toolchains
--------------------

stable-x86_64-pc-windows-gnu
nightly-x86_64-pc-windows-gnu (default)

installed targets for active toolchain
--------------------------------------

thumbv7em-none-eabihf
x86_64-pc-windows-gnu

active toolchain
----------------

nightly-x86_64-pc-windows-gnu (default)
rustc 1.57.0-nightly (e30b68353 2021-09-05)

Stable:

> rustup show
Default host: x86_64-pc-windows-gnu
rustup home:  D:\Users\Hydra\.rustup

installed toolchains
--------------------

stable-x86_64-pc-windows-gnu (default)
nightly-x86_64-pc-windows-gnu

installed targets for active toolchain
--------------------------------------

thumbv7em-none-eabihf
x86_64-pc-windows-gnu

active toolchain
----------------

stable-x86_64-pc-windows-gnu (default)
rustc 1.54.0 (a178d0322 2021-07-26)

Hardware:

Processor	AMD Ryzen 9 5950X 16-Core Processor               3.40 GHz
Installed RAM	64.0 GB

OS:

Edition	Windows 10 Pro
Version	21H1
Installed on	‎05/‎09/‎2021
OS build	19043.928
Experience	Windows Feature Experience Pack 120.2212.551.0

If you need more information/logs/etc, just tell me what commands you want me to run and what output/files to provide.

[mati865]

Could you obtain stacktraces for the hanged process? I though we fixed all hangs like that in LLD 13 but maybe Rust's ancient GCC is even more broken than more recent versions.

[mati865]

@rustbot modify labels: +O-windows-gnu

[hydra]

@mati865 Sure, but how do I obtain a stacktrace for the hung process?

[mati865]

The easiest way is to attach debugger to the process (this sometimes fixes the hang so you cannot get the trace) or create dump by right clicking on the process in task manager and load it into the debugger.
Anyway I'm back home so I've obtained the trace myself:

[0x0]   ntdll!NtWaitForSingleObject + 0x14   
[0x1]   KERNELBASE!WaitForSingleObjectEx + 0x8e   
[0x2]   libwinpthread_1!pthread_cond_init + 0x22c   
[0x3]   libwinpthread_1!pthread_cond_init + 0x36e   
[0x4]   libwinpthread_1!pthread_cond_signal + 0xcf   
[0x5]   rust_lld + 0x282fcd9   
[0x6]   rust_lld + 0x1f6f14c   
[0x7]   rust_lld + 0xdda465   
[0x8]   rust_lld + 0xcc9958   
[0x9]   rust_lld + 0xdb497c   
[0xa]   rust_lld + 0xdc70ab   
[0xb]   rust_lld + 0xdc059a   
[0xc]   rust_lld + 0xe32940   
[0xd]   rust_lld + 0x3cc319   
[0xe]   rust_lld + 0x2b22910   
[0xf]   rust_lld + 0x13f8   
[0x10]   rust_lld + 0x151b   
[0x11]   kernel32!BaseThreadInitThunk + 0x14   
[0x12]   ntdll!RtlUserThreadStart + 0x21   

Indeed it looks like bug in old GCC or mingw-w64 used by Rust to build this target.

[hydra]

@mati865 Awesome! Thanks ever so much for digging into it, was glad you were able to replicate the issue. Keep us all posted. I and my colleagues will be so happy when this is fixed.

Is it possible to upgrade the tools used by rust to workaround this issue?

[mati865]

Unfortunately there is no easy fix, last time I tried to upgrade the tools it couldn't make it on the CI: #51989
Mingw-builds haven't provided any new build since then, I'll try to provide more up to date toolchain but cannot give any ETA.

I don't know how rust-lld is called here but as a workaround you could try using ld.lld from official LLVM builds or MSYS2.

[hydra]

rust-lld is called via cargo -> rustc -> rust-lld IIRC.

Can you reach out the mingw developers to see if they can provide an updated build?

How would I go about using ld.lld ? overwrite the rust-lld.exe with a different .exe or?

[hydra]

I found the documentation for changing the linker:

https://doc.rust-lang.org/rustc/codegen-options/index.html#linker-flavor

I will give that a try as time permits and report back.

[hydra]

This is still an issue, I changed my linker to the gnu-ld linker for other reasons (map file generation, printing memory usage stats, etc) and forgot about this issue. Today when working on a new project I encountered the error again. I'm using a much newer version of rust since I opened this issue.

rustup show reports the following:

Default host: x86_64-pc-windows-gnu
rustup home:  D:\Users\Hydra\.rustup

installed toolchains

--------------------

stable-x86_64-pc-windows-gnu
nightly-2019-07-01-x86_64-pc-windows-gnu
nightly-2021-08-23-x86_64-pc-windows-gnu
nightly-2021-09-07-x86_64-pc-windows-gnu
nightly-2022-03-27-x86_64-pc-windows-gnu (default)
nightly-2022-04-24-x86_64-pc-windows-gnu
nightly-x86_64-pc-windows-gnu

installed targets for active toolchain
--------------------------------------

thumbv7em-none-eabihf
x86_64-pc-windows-gnu

active toolchain
----------------

nightly-2022-03-27-x86_64-pc-windows-gnu (default)
rustc 1.61.0-nightly (1d9c262ee 2022-03-26)

Has there been any update to the tools? what's the current status of this? are there any other workarounds?

[hydra]

Any update on this? I find it a bit crazy that such a fundamental issue, i.e. it randomly fails 9/10 times, hasn't been prioritized over other changes.

[hydra]

This still happens on a very recent nightly:

Default host: x86_64-pc-windows-gnu
rustup home:  D:\Users\Hydra\.rustup

installed toolchains
--------------------

nightly-2019-07-01-x86_64-pc-windows-gnu
nightly-2021-08-23-x86_64-pc-windows-gnu
nightly-2021-09-07-x86_64-pc-windows-gnu
nightly-2022-03-27-x86_64-pc-windows-gnu
nightly-2022-04-24-x86_64-pc-windows-gnu
nightly-2022-05-31-x86_64-pc-windows-gnu
nightly-2022-06-17-x86_64-pc-windows-gnu
nightly-2022-07-13-x86_64-pc-windows-gnu (default)

installed targets for active toolchain
--------------------------------------

thumbv7em-none-eabihf
thumbv7m-none-eabi
x86_64-pc-windows-gnu

active toolchain
----------------

nightly-2022-07-13-x86_64-pc-windows-gnu (default)
rustc 1.64.0-nightly (1c7b36d4d 2022-07-12)

example, frustrating, user experience:

D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor>cargo build
   Compiling stm32f4_pwm_input v0.1.0 (D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor)
^C  Building [=========================> ] 97/98: stm32f4_pwm_input(bin)
D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor>cargo build
   Compiling stm32f4_pwm_input v0.1.0 (D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor)
^C  Building [=========================> ] 97/98: stm32f4_pwm_input(bin)
D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor>cargo build
   Compiling stm32f4_pwm_input v0.1.0 (D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor)
^C  Building [=========================> ] 97/98: stm32f4_pwm_input(bin)
D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor>cargo build
   Compiling stm32f4_pwm_input v0.1.0 (D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor)
    Finished dev [unoptimized + debuginfo] target(s) in 0.45s