referenced symbols in discarded section using sancov-module with inline-8bit-counters

[inglorion]

Building a simple fuzz target using --cfg fuzzing -Cpasses=sancov-module -Cllvm-args=-sanitizer-coverage-level=4 -Cllvm-args=-sanitizer-coverage-inline-8bit-counters results in a linker error: {symbol} referenced in section {section} at {rcgu}: defined in discarded section {other section} of {other rcgu}.

One possible workaround is to set -Ccodegen-units=1 (any higher value triggers the error).

Alternatively, -Znew-llvm-pass-manager=no and -Cpasses=sancov (instead of -Cpasses=sancov-module) will also result in a successful build.

Code

Repro steps:

$ cat <<EOT > Cargo.toml
[package]
name = "fuzz-test"
version = "0.1.0"
edition = "2021"

[dependencies]
libfuzzer-sys = "0.4"

[[bin]]
name = "fuzz_test"
path = "fuzz_test.rs"
EOT

$ cat <<EOT >fuzz_test.rs
#![no_main]
use libfuzzer_sys::fuzz_target;

fuzz_target!(|_data: &[u8]| { });
EOT

$ cargo rustc --release -- \
  --cfg fuzzing \
  -Ccodegen-units=2 \
  -Cpasses=sancov-module \
  -Cllvm-args=-sanitizer-coverage-level=4 \
  -Cllvm-args=-sanitizer-coverage-inline-8bit-counters \
  -Clink-arg="-Wl,--no-gc-sections"

Meta

rustc --version --verbose:

rustc 1.63.0-nightly (e71440575 2022-06-02)
binary: rustc
commit-hash: e71440575c930dcecac288b7c3536410d688b351
commit-date: 2022-06-02
host: x86_64-unknown-linux-gnu
release: 1.63.0-nightly
LLVM version: 14.0.4

Note: The same error also occurs with rustc 1.61.0 (stable channel).

ld --version:

GNU ld (GNU Binutils for Debian) 2.35.2
Copyright (C) 2020 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.

Note: GNU ld is the default linker on this system. On another system where the linker is LLD, a similar error occurs, the message is just worded a bit differently.

Error output

error: linking with `cc` failed: exit status: 1
  |
  = note: "cc" "-m64" "/tmp/rustcburZ4c/symbols.o" "/home/inglorion/fuzz-test/target/release/deps/fuzz_test-81808fa57bf1cda3.fuzz_test.e322ed36-cgu.0.rcgu.o" "/home/inglorion/fuzz-test/target/release/deps/fuzz_"
  = note: `.text.sancov.module_ctor_8bit_counters.19' referenced in section `.init_array.2[sancov.module_ctor_8bit_counters.19]' of /home/inglorion/fuzz-test/target/release/deps/fuzz_test-81808fa57bf1cda3.fuzz_o
          collect2: error: ld returned 1 exit status


error: could not compile `fuzz-test` due to previous error

[inglorion]

We somehow end up running sancov-module on an input that already has @sancov.module_ctor_8bit_counters defined. For the example IR below, the following happens:

1. We generate a new @sancov.module_ctor_8bit_counters.NN, which has a numeric suffix (".19" in the error message above). This is part of the $sancov.module_ctor_8bit_counters (no suffix) comdat.
2. The suffixed @sancov.module_ctor_8bit_counters.NN is added to @llvm.global_ctors, which already mentions @f and the original @sancov.module_ctor_8bit_counters (no suffix).
3. The .init_array.2 that is generated from @llvm.global_ctors is mentioned in two comdat groups: f and sancov.module_ctor_8bit_counters.

If we subsequently link this object with another object which also has a sancov.module_ctor_8bit_counters comdat group, the linker will preserve only one of the sancov.module_ctor_8bit_counters comdat groups. The error message shown earlier happens when the linker discards the comdat group that contains @sancov.module_ctor_8bit_counters.NN, but keeps the .init_array.2 that references that symbol. It can do this because that .init_array.2 is also mentioned in comdat group f.

LLVM IR before sancov-module:

$f = comdat any

@llvm.global_ctors = appending global [2 x { i32, void ()*, i8* }] [{ i32, void ()*, i8* } { i32 2, void ()* @sancov.module_ctor_8bit_counters, i8* bitcast (void ()* @sancov.module_ctor_8bit_counters to i8*) },{ i32, void ()*, i8* } { i32 2, void ()* @f, i8* bitcast (void ()* @f to i8*) }]                                                                                                                                   

define internal void @f() comdat {
  ret void
}

define internal void @sancov.module_ctor_8bit_counters() {
  ret void
}

Some relevant sections of the IR after opt -passes=sancov-module -sanitizer-coverage-level=1 -sanitizer-inline-8bit-counters=1:

$f = comdat any

$sancov.module_ctor_8bit_counters = comdat any

@__sancov_gen_ = private global [1 x i8] zeroinitializer, section "__sancov_cntrs", comdat($f), align 1
@llvm.global_ctors = appending global [3 x { i32, void ()*, i8* }] [{ i32, void ()*, i8* } { i32 2, void ()* @sancov.module_ctor_8bit_counters, i8* bitcast (void ()* @sancov.module_ctor_8bit_counters to i8*) },\
 { i32, void ()*, i8* } { i32 2, void ()* @f, i8* bitcast (void ()* @f to i8*) }, { i32, void ()*, i8* } { i32 2, void ()* @sancov.module_ctor_8bit_counters.1, i8* bitcast (void ()* @sancov.module_ctor_8bit_cou\
nters.1 to i8*) }]
@llvm.used = appending global [1 x i8*] [i8* bitcast (void ()* @sancov.module_ctor_8bit_counters.1 to i8*)], section "llvm.metadata"
@llvm.compiler.used = appending global [1 x i8*] [i8* getelementptr inbounds ([1 x i8], [1 x i8]* @__sancov_gen_, i32 0, i32 0)], section "llvm.metadata"

define internal void @sancov.module_ctor_8bit_counters() {
  ret void
}

; Function Attrs: nounwind
define internal void @sancov.module_ctor_8bit_counters.1() #0 comdat($sancov.module_ctor_8bit_counters) {
  call void @__sanitizer_cov_8bit_counters_init(i8* @__start___sancov_cntrs, i8* @__stop___sancov_cntrs)
  ret void
}

attributes #0 = { nounwind }

After machine code generation, llvm-readelf --section-groups:

COMDAT group section [    3] `.group' [f] contains 5 sections:
   [Index]    Name
   [    4]   .text.f
   [    5]   .rela.text.f
   [    9]   __sancov_cntrs
   [   12]   .init_array.2
   [   13]   .rela.init_array.2

COMDAT group section [    6] `.group' [sancov.module_ctor_8bit_counters] contains 4 sections:
   [Index]    Name
   [    7]   .text.sancov.module_ctor_8bit_counters.1
   [    8]   .rela.text.sancov.module_ctor_8bit_counters.1
   [   10]   .init_array.2
   [   11]   .rela.init_array.2

COMDAT group section [   14] `.group' [sancov.module_ctor_8bit_counters.1] contains 2 sections:
   [Index]    Name
   [   15]   .init_array.2
   [   16]   .rela.init_array.2

There is an assert in SanitizerCoverage.cpp CreateInitCallsForSections:

  std::tie(CtorFunc, std::ignore) = createSanitizerCtorAndInitFunctions(                                                                                                                                          
      M, CtorName, InitFunctionName, {PtrTy, PtrTy}, {SecStart, SecEnd});                                                                                                                                         
  assert(CtorFunc->getName() == CtorName);                                                                                                                                                                        

This assert will trigger when @sancov.module_ctor_8bit_counters.1 is created. Of course, that only happens in LLVM builds with asserts enabled.

What I don't yet know is why the sancov-module pass is being run on a module it has already been run for. I can see the multiple runs in the output if I pass in -Cllvm-args=-print-before-all, one of which creates the suffixed function, but I do not know what causes the pass to be run multiple times on the same module.

[tmiasko]

Passes configured with -Cpasses should probably be included only in pre-link optimization stage in:

rust/compiler/rustc_codegen_llvm/src/back/write.rs

Line 470 in f19ccc2

let extra_passes = config.passes.join(",");

Similarly to sanitizer passes configured a few lines earlier:

rust/compiler/rustc_codegen_llvm/src/back/write.rs

Lines 444 to 446 in f19ccc2

	let is_lto = opt_stage == llvm::OptStage::ThinLTO || opt_stage == llvm::OptStage::FatLTO;
	// Sanitizer instrumentation is only inserted during the pre-link optimization stage.
	let sanitizer_options = if !is_lto {

Otherwise they will be executed multiple times with LTO (a thin local LTO is enabled by default when using multiple codegen units).

[inglorion]

Thank you. I was thinking along the same lines. I have a patch that does this, which I will send for review once I've tested it. I am wondering if there is value in also being able to specify passes specifically for the LTO stage; perhaps with a new -Zlink-passes flag. What do you think?