Work around custom section flag regression in LLVM

[nikic]

LLVM 13 has removed support for setting custom section flags in https://reviews.llvm.org/D100944. LLVM 13 carries a revert of the problematic change (https://reviews.llvm.org/D107216). However, LLVM 14 will no longer carry it, and we need to use alternative ways to set custom section flags.

There are currently two places affected by this. The .llvmbc bitcode section created in

rust/compiler/rustc_codegen_llvm/src/back/write.rs

Line 926 in ffba430

unsafe fn embed_bitcode(

and the .rustc metadata section created in

rust/compiler/rustc_codegen_llvm/src/base.rs

Line 41 in 612356a

pub fn write_compressed_metadata<'tcx>(

. In both cases, section flags are used to ensure that the section is not linked into the final binary or loaded into memory.

There are two ways to address this. The first one is to define the section entirely in module-level assembly. Rather than only emitting .section, we should also emit .ascii "ENCODED_SECTION_CONTENTS", as well as the symbol name and any other necessary directives (e.g. for visibility). I'm not sure exactly what is needed on different platforms here.

The second one is to use the object crate to create the object file containing the section. The cranelift backend already implements this for the .rustc section in

rust/compiler/rustc_codegen_cranelift/src/metadata.rs

Line 10 in ffba430

pub(crate) fn new_metadata_object(

. A nice side benefit of this is that the handling would be codegen-backend agnostic and could be shared between the llvm/cranelift/gcc backends.

The suggested course of action is to use the object crate for .rustc and inline assembly for .llvmbc. The reason for the latter is that the section is part of a larger object file emitted by LLVM, and the object crate doesn't seem great for modifying existing object files (as opposed to creating one from scratch). Though if we can use the object crate for everything, that would of course be great.

[durin42]

Quoting one of my LLVM experts: "the .llvmbc issue is an issue for Clang as well (e.g. using clang -fembed-bitcode on ELF has this problem). Clearly that ought to be solved in some manner other than inline-asm hacks or post-processing object files." So there's some appetite for a more systematic solution upstream in LLVM.

[jyknight]

(Jumping in from LLVM side -- I was pointed to this bug.)

clang -fembed-bitcode also causes the bitcode to end up in the binary on ELF platforms, just like Rust would without the section-flag workaround. That's perhaps not entirely useful, but the -fembed-bitcode flag is almost unused everywhere except on Apple platforms (where it's required in order to upload to appstore in some cases) -- so basically nobody really cares.

Rust seems to want to go to some trouble to avoid this outcome -- but it's not clear to me why you want the embedded bitcode in the first place -- other than on iOS/apple? Perhaps it'd be nice to fix this someday, but is it more important to Rust, than it is to Clang?

[nikic]

@jyknight Under some circumstances, Rust uses fat object files that contain both machine code and bitcode. The MC version is used without LTO, the BC version used with LTO.

Changing the flags that LLVM automatically applies to .llvmbc would solve (the more ugly) half of the problem, but I wasn't sure whether unconditionally using SHF_EXCLUDE/LNK_REMOVE would be fine for other use cases.

[jyknight]

LTO input wasn't really the intended use-case of the embedded bitcode feature. I don't think that even works at all with clang+lld, does it? The original purpose was an effectively apple-specific use-case: to allow Apple to rerun the codegen phase of compilation separately for each original object file, and then relink the binary. This would be used either in order to workaround CPU bugs with a new version of the toolchain, or to recompile an app to a new specially-designed-to-be-ABI-compatible architecture (bitcode for watchOS on ARM32 can be rebuilt for watchos on AArch64 (the later they call "arm64_32", since it uses a 32bit ABI)).

For LTO, it's better to just emit the correct object file for the linking mode/modes you're actually using. That's why it surprises me that this is something rust is running into.

The overhead of doing a fat object file (both size of the extraneous output and time to generate it) can be substantial, and additionally, the pass pipelines are different for generating pre-LTO bitcode vs pre-object-emission bitcode. (The semantics of llvmbc used by -fembed-bitcode is that it contains the pre-object-emission bitcode, not LTO bitcode).

Simply SHF_EXCLUDE'ing .llvmbc isn't a solution by itself. The primary purpose of -fembed-bitcode is intended to be that you can get the bitcode output in the final binary when you ask for it. On Mach-O platforms, this is done with special linker support: if the linker is passed "-bitcode_bundle", it takes the bitcode sections of the objects, puts them into a weird container format ("XAR"), then puts that into a section in the output binary. If that linker option isn't set, it ignores the bitcode sections in the input object files.

This doesn't really work on non-MachO platforms at the moment -- both because bitcode gets included into the output without a flag, and because when it is emitted, it's just concatenated, rather than put into a useful form.

We could potentially add proper support, but again, I don't think this is actually something Rust should be using at all...

[bjorn3]

For LTO, it's better to just emit the correct object file for the linking mode/modes you're actually using. That's why it surprises me that this is something rust is running into.

We have to support libraries that are capable of LTO, but don't require it. The same standard library is used for LTO and non-LTO usage and as such needs to contain LTO bitcode, but also machine code at the same time.

[bjorn3]

Simply SHF_EXCLUDE'ing .llvmbc isn't a solution by itself. The primary purpose of -fembed-bitcode is intended to be that you can get the bitcode output in the final binary when you ask for it.

Would you prefer the section to be called .rust_llvm_lto_bitcode or something like that? We don't use this section for the same purpose as -fembed-bitcode in clang it seems.