Bevy dynamic-linking makes the program crash

[GglLfr]

Minimal reproducible example: https://github.com/GlennFolker/cranedrop

The dynamic_linking feature flag, which converts bevy and std into a dylib for compile-time speedup, completely fails when using the Cranelift codegen backend.

System information:

• OS and arch: Windows 11, x64.
• Rustc version: rustc 1.83.0-nightly (0609062a9 2024-09-13).
• Contents of ~/.cargo/Config.toml:

  [unstable]
  codegen-backend = true
  
  [profile.dev]
  codegen-backend = "cranelift"
  incremental = true
  lto = "off"
  
  [profile.dev.package."*"]
  opt-level = 3
  
  [profile.release]
  codegen-units = 1
  incremental = false
  lto = "on"
  
  [target.x86_64-pc-windows-msvc]
  linker = "rust-lld.exe"
  rustflags = [
      "-Z", "share-generics=no",
      "-Z", "threads=0",
  ]
  

cargo run with multi_threading disabled made the program encounter a segfault:

error: process didn't exit successfully: `target\x86_64-pc-windows-msvc\debug\cranedrop.exe` (exit code: 0xc0000005, STATUS_ACCESS_VIOLATION)

cargo run with multi_threading enabled made the program panic:

thread 'main' panicked at C:\Users\glenn\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_tasks-0.14.2\src\task_pool.rs:186:22:
Failed to spawn thread.: Os { code: 87, kind: InvalidInput, message: "The parameter is incorrect." }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: process didn't exit successfully: `target\x86_64-pc-windows-msvc\debug\cranedrop.exe` (exit code: 0xe06d7363)

[bjorn3]

cargo run with multi_threading disabled made the program encounter a segfault:

I don't use Windows, so I'm not sure I will be able to figure out what is going wrong. In any case could you please post a backtrace of the segfault?

cargo run with multi_threading enabled made the program panic:

This panic happens when trying to spawn a new thread. The error is not really informative about which parameter of the thread spawn function is incorrect.

[GglLfr]

I tried yesterday, but I'm not sure how to GDB a Rust dynamically linked program since it complains about libbevy_dylib_<hash> and libstd_dylib_<hash> missing from the library path, and I just couldn't find them.

I'll try later today.

[GglLfr]

Well, I couldn't figure out a way to debug the builds, but apparently x86_64-pc-windows-gnu works just fine. Error only occurs on x86_64-pc-windows-msvc.

[bjorn3]

I tried yesterday, but I'm not sure how to GDB a Rust dynamically linked program since it complains about libbevy_dylib_ and libstd_dylib_ missing from the library path, and I just couldn't find them.

You need to add both the full path to the target/x86_64-pc-windows-msvc/debug/deps dir and the path shown by rustc-clif --print target-libdir to your PATH before running the executable. Cargo and rustup together normally do this for you if you use cargo run. Also for MSVC you have to use windbg or similar as debugger. Gdb doesn't support reading debuginfo in the format MSVC uses (.pdb). Only the format unix systems use (DWARF).

[bjorn3]

Is this still an issue?

[GglLfr]

My apologies for disappearing- yes, it still happens. Forgive my ignorance, but I (still) have no idea how to debug with WinDBG, and all sources that I've found has no straight forward way to give a small gist for what we need (run a program, wait until crash, show backtrace). The minimum reproducible example is still up!

[GglLfr]

Hi, after some thorough search using both Google and ChatGPT, I finally came up with some debugging information:

• I installed a non-UWP version of WinDBG through Windows Software Development Kit (which I can then run via command-line using windbg.exe).
• I use this script to load Rust natvis files:

  @echo off
  setlocal
  
  for /f "delims=" %%i in ('rustc --print=sysroot') do set rustc_sysroot=%%i
  
  set rust_etc=%rustc_sysroot%\lib\rustlib\etc
  
  windbg.exe -c ".nvload %rust_etc%\intrinsic.natvis; .nvload %rust_etc%\liballoc.natvis; .nvload %rust_etc%\libcore.natvis;" %*

• Ran cargo build and manually copied std and bevy_lib's DLLs and PDBs to target\x86_64-pc-windows-msvc\debug.
• Ran the executable via the script, and invoked the commands g and k, from which I got this output:

************* Preparing the environment for Debugger Extensions Gallery repositories **************
   ExtensionRepository : Implicit
   UseExperimentalFeatureForNugetShare : true
   AllowNugetExeUpdate : true
   NonInteractiveNuget : true
   AllowNugetMSCredentialProviderInstall : true
   AllowParallelInitializationOfLocalRepositories : true

   EnableRedirectToV8JsProvider : false

   -- Configuring repositories
      ----> Repository : LocalInstalled, Enabled: true
      ----> Repository : UserExtensions, Enabled: true

>>>>>>>>>>>>> Preparing the environment for Debugger Extensions Gallery repositories completed, duration 0.032 seconds

************* Waiting for Debugger Extensions Gallery to Initialize **************

>>>>>>>>>>>>> Waiting for Debugger Extensions Gallery to Initialize completed, duration 0.015 seconds
   ----> Repository : UserExtensions, Enabled: true, Packages count: 0
   ----> Repository : LocalInstalled, Enabled: true, Packages count: 29

Microsoft (R) Windows Debugger Version 10.0.26100.2454 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

CommandLine: target\x86_64-pc-windows-msvc\debug\cranedrop.exe

************* Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       SRV*C:\symsrv*http://msdl.microsoft.com/download/symbols

************* Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       SRV*C:\symsrv*http://msdl.microsoft.com/download/symbols
Symbol search path is: SRV*C:\symsrv*http://msdl.microsoft.com/download/symbols
Executable search path is: 

+------------------------------------------------------------------------+
| This target supports Hardware-enforced Stack Protection. A HW based    |
| "Shadow Stack" may be available to assist in debugging and analysis.   |
| See aka.ms/userhsp for more info.                                      |
|                                                                        |
| dps @ssp                                                               |
|                                                                        |
+------------------------------------------------------------------------+

ModLoad: 00007ff7`46c70000 00007ff7`46c89000   cranedrop.exe
ModLoad: 00007ffb`b3920000 00007ffb`b3b83000   ntdll.dll
ModLoad: 00007ffb`b29f0000 00007ffb`b2ab7000   C:\WINDOWS\System32\KERNEL32.DLL
ModLoad: 00007ffb`b1070000 00007ffb`b1439000   C:\WINDOWS\System32\KERNELBASE.dll
ModLoad: 00007ffb`b0f20000 00007ffb`b106c000   C:\WINDOWS\System32\ucrtbase.dll
ModLoad: 00007ffb`8e9d0000 00007ffb`8e9ee000   C:\WINDOWS\SYSTEM32\VCRUNTIME140.dll
ModLoad: 00007ffb`39c90000 00007ffb`3a605000   C:\Users\glenn\cranedrop\target\x86_64-pc-windows-msvc\debug\std-c8733523eab5dd8b.dll
ModLoad: 00007ffb`01ea0000 00007ffb`08d55000   C:\Users\glenn\cranedrop\target\x86_64-pc-windows-msvc\debug\bevy_dylib-741ea37351629613.dll
ModLoad: 00007ffb`b35c0000 00007ffb`b35ea000   C:\WINDOWS\System32\GDI32.dll
ModLoad: 00007ffb`b35f0000 00007ffb`b3664000   C:\WINDOWS\System32\WS2_32.dll
ModLoad: 00007ffb`b3670000 00007ffb`b3786000   C:\WINDOWS\System32\RPCRT4.dll
ModLoad: 00007ffb`b1440000 00007ffb`b1467000   C:\WINDOWS\System32\win32u.dll
ModLoad: 00007ffb`b16a0000 00007ffb`b1739000   C:\WINDOWS\System32\bcryptprimitives.dll
ModLoad: 00007ffb`affb0000 00007ffb`affdb000   C:\WINDOWS\SYSTEM32\USERENV.dll
ModLoad: 00007ffb`b0c60000 00007ffb`b0d91000   C:\WINDOWS\System32\gdi32full.dll
ModLoad: 00007ffb`b0bb0000 00007ffb`b0c53000   C:\WINDOWS\System32\msvcp_win.dll
ModLoad: 00007ffb`b1790000 00007ffb`b195a000   C:\WINDOWS\System32\USER32.dll
ModLoad: 00007ffb`b2c70000 00007ffb`b2d46000   C:\WINDOWS\System32\oleaut32.dll
ModLoad: 00007ffb`b2660000 00007ffb`b29e2000   C:\WINDOWS\System32\combase.dll
ModLoad: 00007ffb`539a0000 00007ffb`53aac000   C:\WINDOWS\SYSTEM32\OPENGL32.dll
ModLoad: 00007ffb`aa550000 00007ffb`aa9cf000   C:\WINDOWS\SYSTEM32\d3dcompiler_47.dll
ModLoad: 00007ffb`b2b80000 00007ffb`b2c29000   C:\WINDOWS\System32\msvcrt.dll
ModLoad: 00007ffb`b2d70000 00007ffb`b2f05000   C:\WINDOWS\System32\ole32.dll
ModLoad: 00007ffb`b02c0000 00007ffb`b02dc000   C:\WINDOWS\SYSTEM32\cryptsp.dll
ModLoad: 0000023c`a4050000 0000023c`a4102000   C:\WINDOWS\System32\advapi32.dll
ModLoad: 00007ffb`b2ac0000 00007ffb`b2b72000   C:\WINDOWS\System32\ADVAPI32.dll
ModLoad: 00007ffb`b32c0000 00007ffb`b3366000   C:\WINDOWS\System32\sechost.dll
ModLoad: 00007ffb`ace60000 00007ffb`acf5c000   C:\WINDOWS\SYSTEM32\propsys.dll
ModLoad: 00007ffb`b0450000 00007ffb`b0476000   C:\WINDOWS\SYSTEM32\bcrypt.dll
ModLoad: 00007ffb`73a80000 00007ffb`73aad000   C:\WINDOWS\SYSTEM32\GLU32.dll
ModLoad: 00007ffb`adcb0000 00007ffb`adcf7000   C:\WINDOWS\SYSTEM32\dxcore.dll
ModLoad: 00007ffb`b0140000 00007ffb`b014c000   C:\WINDOWS\SYSTEM32\CRYPTBASE.DLL
(485c.121b0): Break instruction exception - code 80000003 (first chance)
Processing initial command '.nvload C:\Users\glenn\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\etc\intrinsic.natvis; .nvload C:\Users\glenn\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\etc\liballoc.natvis; .nvload C:\Users\glenn\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\etc\libcore.natvis;'
ntdll!LdrpDoDebuggerBreak+0x35:
00007ffb`b3a414a9 cc              int     3
0:000> .nvload C:\Users\glenn\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\etc\intrinsic.natvis; .nvload C:\Users\glenn\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\etc\liballoc.natvis; .nvload C:\Users\glenn\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\etc\libcore.natvis;
Successfully loaded visualizers in "C:\Users\glenn\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\etc\intrinsic.natvis"
Successfully loaded visualizers in "C:\Users\glenn\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\etc\liballoc.natvis"
Successfully loaded visualizers in "C:\Users\glenn\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\etc\libcore.natvis"
0:000> g
ModLoad: 00007ffb`b2c30000 00007ffb`b2c5f000   C:\WINDOWS\System32\IMM32.DLL
(485c.121b0): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
*** WARNING: Unable to verify checksum for C:\Users\glenn\cranedrop\target\x86_64-pc-windows-msvc\debug\std-c8733523eab5dd8b.dll
std_c8733523eab5dd8b!core::sync::atomic::atomic_compare_exchange+0x4 [inlined in std_c8733523eab5dd8b!std::io::stdio::impl$26::write_all+0x9f]:
00007ffb`39caa2ef f0410fb04e0c    lock cmpxchg byte ptr [r14+0Ch],cl ds:00007ffb`06e3f7fc=ff
0:000> k
 # Child-SP          RetAddr               Call Site
00 (Inline Function) --------`--------     std_c8733523eab5dd8b!core::sync::atomic::atomic_compare_exchange+0x4 [/rustc/a2e63569fd6702ac5dd027a80a9fdaadce73adae/library\core\src\sync\atomic.rs @ 3810] 
01 (Inline Function) --------`--------     std_c8733523eab5dd8b!core::sync::atomic::AtomicU8::compare_exchange+0x4 [/rustc/a2e63569fd6702ac5dd027a80a9fdaadce73adae/library\core\src\sync\atomic.rs @ 2881] 
02 (Inline Function) --------`--------     std_c8733523eab5dd8b!std::sys::sync::mutex::futex::Mutex::lock+0x4 [/rustc/a2e63569fd6702ac5dd027a80a9fdaadce73adae/library\std\src\sys\sync\mutex\futex.rs @ 28] 
03 (Inline Function) --------`--------     std_c8733523eab5dd8b!std::sync::reentrant_lock::ReentrantLock::lock+0x82 [/rustc/a2e63569fd6702ac5dd027a80a9fdaadce73adae/library\std\src\sync\reentrant_lock.rs @ 289] 
04 (Inline Function) --------`--------     std_c8733523eab5dd8b!std::io::stdio::Stderr::lock+0x85 [/rustc/a2e63569fd6702ac5dd027a80a9fdaadce73adae/library\std\src\io\stdio.rs @ 1001] 
05 0000008b`20d3d160 00007ffb`39ca9bc7     std_c8733523eab5dd8b!std::io::stdio::impl$26::write_all+0x9f [/rustc/a2e63569fd6702ac5dd027a80a9fdaadce73adae/library\std\src\io\stdio.rs @ 1060] 
*** WARNING: Unable to verify checksum for C:\Users\glenn\cranedrop\target\x86_64-pc-windows-msvc\debug\bevy_dylib-741ea37351629613.dll
06 0000008b`20d3d1c0 00007ffb`01efcd02     std_c8733523eab5dd8b!std::io::stdio::impl$25::write_all+0x17 [/rustc/a2e63569fd6702ac5dd027a80a9fdaadce73adae/library\std\src\io\stdio.rs @ 1035] 
07 0000008b`20d3d200 0000008b`20d3d2c0     bevy_dylib_741ea37351629613!ZN103_$LT$alloc..vec..into_iter..IntoIter$LT$T$C$A$GT$$u20$as$u20$core..iter..traits..iterator..Iterator$GT$8try_fold17h0028cb97fbea0ddeE+0x722
08 0000008b`20d3d208 0000008b`20d3d2d0     0x0000008b`20d3d2c0
09 0000008b`20d3d210 00007ffb`082e69b0     0x0000008b`20d3d2d0
0a 0000008b`20d3d218 0000008b`20d3d2d0     bevy_dylib_741ea37351629613!ZN106_$LT$bevy_app..terminal_ctrl_c_handler..TerminalCtrlCHandlerPlugin$u20$as$u20$bevy_app..plugin..Plugin$GT$5build10__CALLSITE4META17h6d21c383157d1477E
0b 0000008b`20d3d220 0000023c`a3e0d680     0x0000008b`20d3d2d0
0c 0000008b`20d3d228 0000023c`a3e0d678     0x0000023c`a3e0d680
0d 0000008b`20d3d230 0000023c`a3e0d680     0x0000023c`a3e0d678
0e 0000008b`20d3d238 0000023c`a3e0d678     0x0000023c`a3e0d680
0f 0000008b`20d3d240 0000008b`20d3d2c0     0x0000023c`a3e0d678
10 0000008b`20d3d248 0000008b`20d3d2d0     0x0000008b`20d3d2c0
11 0000008b`20d3d250 5a02ffff`ffff22ff     0x0000008b`20d3d2d0
12 0000008b`20d3d258 00000000`00000001     0x5a02ffff`ffff22ff
13 0000008b`20d3d260 0000023c`a3e0d680     0x1
14 0000008b`20d3d268 0000008b`20d3d2c0     0x0000023c`a3e0d680
15 0000008b`20d3d270 0000023c`a3e0d680     0x0000008b`20d3d2c0
16 0000008b`20d3d278 0000023c`a3e0d678     0x0000023c`a3e0d680
17 0000008b`20d3d280 0000008b`20d3d310     0x0000023c`a3e0d678
18 0000008b`20d3d288 00007ffb`01eb3101     0x0000008b`20d3d310
19 0000008b`20d3d290 00000000`00000000     bevy_dylib_741ea37351629613!ZN113_$LT$tracing_subscriber..layer..layered..Layered$LT$L$C$S$GT$$u20$as$u20$tracing_core..subscriber..Subscriber$GT$17register_callsite17hcd61da19f710b800E+0x57

Relevant system information:

• Windows: Version 24H2 (OS Build 26100.3476)
• Rustup: 1.28.1 (f9edccde0 2025-03-05)
• Rust: 1.87.0-nightly (a2e63569f 2025-03-26)
• link: 14.43.34809.0
• lld-link (installed through Visual Studio's Clang package): 19.1.1
• Contents of .cargo/config.toml:

  [unstable]
  codegen-backend = true
  package-workspace = true
  
  [build]
  target = "x86_64-pc-windows-msvc"
  
  [profile.dev]
  codegen-backend = "cranelift"
  incremental = true
  lto = "off"
  debug = 2
  
  [profile.dev.package."*"]
  opt-level = 3

[BigBadE]

This is still occurring on Bevy 1.15.3. Is there anything else that can be done to help with a fix?

[bjorn3]

Does this also occur with a version of cg_clif downloaded from https://github.com/rust-lang/rustc_codegen_cranelift/releases/tag/dev rather than from rustup? If not, there is a fair chance that this is an ABI compatibility issue between Cranelift and LLVM: #1525

[GglLfr]

cargo-clif build (also temporarily removed .cargo/config.toml for sanitation) produces the exact same error, unfortunately...

[bjorn3]

While it could still be an ABI issue with respect to C, it is a lot less likely that this is the cause as C FFI tends to cause less issues and the crash here happens within rust code seemingly without any C code in between. I think I will need to get access to a Windows machine at some point and see if I can either poke around with lldb or if lldb doesn't work, learn windbg. Or maybe if this reproduces with mingw, I could enable DWARF debuginfo on mingw targets and poke around with gdb. (Currently there is exactly zero debuginfo support on Windows as PDB doesn't have any official documentation other than a header dump microsoft did a couple years back and even LLVM needed to reverse-engineer it)

[GglLfr]

I believe I've reproduced this issue on a much smaller case with zero dependency footprints, by mimicking bevy_ecs's method of interning values in this repository.

I also believe it's widely reproducible, as indicated by this instance of GitHub Actions runner.