Tracking issue for RFC 2603, "Rust Symbol Mangling (v0)"

[Centril]

This is a tracking issue for the RFC "Rust Symbol Mangling (v0)" (rust-lang/rfcs#2603).

Current status:

Since #90128, you can control the mangling scheme with -C symbol-mangling-version, which can be:

• legacy: the older mangling version, still the default currently
  • explicitly specifying this is unstable-only and also requires -Z unstable-options
    (to allow for eventual removal after v0 becomes the default)
• v0: the new RFC mangling version, as implemented by Introduce Rust symbol mangling scheme. #57967

(Before #90128, this flag was the nightly-only -Z symbol-mangling-version)

To test the new mangling, set RUSTFLAGS=-Csymbol-mangling-version=v0 (or change rustflags in .cargo/config.toml). Please note that only symbols from crates built with that flag will use the new mangling, and that tool support (e.g. debuggers) will be limited initially, until everything is upstreamed. However, RUST_BACKTRACE and rustfilt should work out of the box with either mangling version.

Steps:

• Implement the RFC (Introduce Rust symbol mangling scheme. #57967 + Support demangling the new Rust mangling scheme (v0). rustc-demangle#23)
• Upstream C implementation of the demangler to:
  • binutils/gdb (GNU libiberty)
    • [PATCH] Move rust_{is_mangled,demangle_sym} to a private libiberty header.
      committed as gcc-mirror/gcc@979526c
    • [PATCH] Simplify and generalize rust-demangle's unescaping logic.
      committed as gcc-mirror/gcc@42bf58b
    • [PATCH] Remove some restrictions from rust-demangle.
      committed as gcc-mirror/gcc@e1cb00d
    • [PATCH] Refactor rust-demangle to be independent of C++ demangling.
      (original submission) committed as gcc-mirror/gcc@32fc371
    • [PATCH] Support the new ("v0") mangling scheme in rust-demangle.
      (original submission) committed as gcc-mirror/gcc@8409649
  • Linux perf (through binutils 2.36 and/or libiberty 11.0, or later versions - may vary between distros)
  • valgrind
• Implement demangling support in LLVM, including lldb, lld, llvm-objdump, llvm-nm, llvm-symbolizer, llvm-cxxfilt
• Resolve issue around rustc generating invalid symbol names (v0 mangled symbol doesn't match specification nor can it be demangled with rustfilt or c++filt #83611)
• Adjust documentation (see instructions on rustc-guide)
• Stabilization PR (see instructions on rustc-guide)

Unresolved questions:

• Punycode vs UTF-8, some prior discussion in Ident mangling and unicode. #7539
• Encoding parameter types for function symbols

Desired availability of tooling:

Linux:

• Tools: binutils, gdb, lldb, perf, valgrind

Distro	Has versions of all tools with support?
Debian (latest stable)	?
Arch	?
Ubuntu (latest release)	?
Ubuntu (latest LTS)	?
Fedora (latest release)	?
Alpine (latest release)	?

Windows:

Windows does not have support for demangling either legacy or v0 Rust symbols and requires debuginfo to load the appropriate function name. As such, no special support is required.

macOS:

More investigation is needed to determine to what extent macOS system tools already support Rust v0 mangling.

[eddyb]

cc @rust-lang/compiler

I'd add this as an unresolved question: rust-lang/rfcs#2603 (comment)

I'd rather change the grammar now, if ever, before merging #57967.

[michaelwoerister]

I think the Punycode vs UTF-8 and Encoding parameters types for function symbols questions are actually resolved. The RFC text contains a recommendation on what the resolution should look like (since before the motion to merge) and nobody objected.

[eddyb]

I tried the "type/value namespace" shorthands, and this is what I got, stats-wise:
(apologies for having two tables, but it was hard to combine them further)

986807 symbols: sizes and compression ratios (▶)

v0 without shorthands (types use Nt and values use Nv)

	legacy	mw	▶	mw + compress	v0	▶	v0 + compress
min	30	25	0.97x	25	25	0.96x	25
avg	81	313	1.46x	193	318	1.47x	194
max	688	13409	14.83x	1115	13759	15.24x	1105
total	76.6MiB	295.4MiB	1.62x	182.0MiB	300.2MiB	1.64x	183.5MiB

v0 with shorthands (types use Z and values use V)

	legacy	mw	▶	mw + compress	v0	▶	v0 + compress
min	30	25	0.97x	25	24	0.96x	24
avg	81	313	1.46x	193	303	1.48x	184
max	688	13409	14.83x	1115	12925	14.99x	1047
total	76.6MiB	295.4MiB	1.62x	182.0MiB	285.5MiB	1.64x	173.9MiB

So that's around 5% reduction, kinda nice, but not that much.

[robinmoussu]

I just opened a bug in compiler-explorer because I noticed collision in the demangled name of different monomorphisation of the same function. As far as I understand, the current mangling scheme of rust still use the v1 witch doesn't contains the required information, while this proposed v2 would solved it? Did I correctly understood what the current situation is?