[LLVM][rtsan] Add module pass to initialize rtsan

[cjappl]

This allows shared libraries instrumented with RTSan to be initialized.

This approach directly mirrors the approach in Tsan, Asan and many of the other sanitizers:

llvm-project/llvm/lib/Transforms/Instrumentation/ThreadSanitizer.cpp

Line 175 in c74e223

M, kTsanModuleCtorName, kTsanInitName, /*InitArgTypes=*/{},

[llvmbot]

@llvm/pr-subscribers-llvm-transforms
@llvm/pr-subscribers-compiler-rt-sanitizer
@llvm/pr-subscribers-clang-codegen

@llvm/pr-subscribers-clang

Author: Chris Apple (cjappl)

Changes

This allows shared libraries instrumented with RTSan to be initialized.

This approach directly mirrors the approach in Tsan, Asan and many of the other sanitizers:

llvm-project/llvm/lib/Transforms/Instrumentation/ThreadSanitizer.cpp

Line 175 in c74e223

M, kTsanModuleCtorName, kTsanInitName, /*InitArgTypes=*/{},

---

Full diff: https://github.com/llvm/llvm-project/pull/118989.diff

5 Files Affected:

• (modified) clang/lib/CodeGen/BackendUtil.cpp (+3-1)
• (modified) llvm/include/llvm/Transforms/Instrumentation/RealtimeSanitizer.h (+7)
• (modified) llvm/lib/Passes/PassRegistry.def (+1)
• (modified) llvm/lib/Transforms/Instrumentation/RealtimeSanitizer.cpp (+15)
• (modified) llvm/test/Instrumentation/RealtimeSanitizer/rtsan.ll (+5-1)

diff --git a/clang/lib/CodeGen/BackendUtil.cpp b/clang/lib/CodeGen/BackendUtil.cpp
index bf9b04f02e9f44..fbb3fb6e5ea423 100644
--- a/clang/lib/CodeGen/BackendUtil.cpp
+++ b/clang/lib/CodeGen/BackendUtil.cpp
@@ -1024,12 +1024,14 @@ void EmitAssemblyHelper::RunOptimizationPipeline(
             FPM.addPass(BoundsCheckingPass());
           });
 
-    if (LangOpts.Sanitize.has(SanitizerKind::Realtime))
+    if (LangOpts.Sanitize.has(SanitizerKind::Realtime)) {
       PB.registerScalarOptimizerLateEPCallback(
           [](FunctionPassManager &FPM, OptimizationLevel Level) {
             RealtimeSanitizerOptions Opts;
             FPM.addPass(RealtimeSanitizerPass(Opts));
           });
+      MPM.addPass(ModuleRealtimeSanitizerPass());
+    }
 
     // Don't add sanitizers if we are here from ThinLTO PostLink. That already
     // done on PreLink stage.
diff --git a/llvm/include/llvm/Transforms/Instrumentation/RealtimeSanitizer.h b/llvm/include/llvm/Transforms/Instrumentation/RealtimeSanitizer.h
index f2ce1636551ce2..151e70e87b294d 100644
--- a/llvm/include/llvm/Transforms/Instrumentation/RealtimeSanitizer.h
+++ b/llvm/include/llvm/Transforms/Instrumentation/RealtimeSanitizer.h
@@ -33,6 +33,13 @@ class RealtimeSanitizerPass : public PassInfoMixin<RealtimeSanitizerPass> {
   static bool isRequired() { return true; }
 };
 
+/// Create ctor and init functions.
+struct ModuleRealtimeSanitizerPass
+    : public PassInfoMixin<ModuleRealtimeSanitizerPass> {
+  PreservedAnalyses run(Module &M, ModuleAnalysisManager &AM);
+  static bool isRequired() { return true; }
+};
+
 } // namespace llvm
 
 #endif // LLVM_TRANSFORMS_INSTRUMENTATION_REALTIMESANITIZER_H
diff --git a/llvm/lib/Passes/PassRegistry.def b/llvm/lib/Passes/PassRegistry.def
index 772ec5fd10e633..2ddebb07017c2a 100644
--- a/llvm/lib/Passes/PassRegistry.def
+++ b/llvm/lib/Passes/PassRegistry.def
@@ -138,6 +138,7 @@ MODULE_PASS("rel-lookup-table-converter", RelLookupTableConverterPass())
 MODULE_PASS("rewrite-statepoints-for-gc", RewriteStatepointsForGC())
 MODULE_PASS("rewrite-symbols", RewriteSymbolPass())
 MODULE_PASS("rpo-function-attrs", ReversePostOrderFunctionAttrsPass())
+MODULE_PASS("rtsan-module", ModuleRealtimeSanitizerPass())
 MODULE_PASS("sample-profile", SampleProfileLoaderPass())
 MODULE_PASS("sancov-module", SanitizerCoveragePass())
 MODULE_PASS("sanmd-module", SanitizerBinaryMetadataPass())
diff --git a/llvm/lib/Transforms/Instrumentation/RealtimeSanitizer.cpp b/llvm/lib/Transforms/Instrumentation/RealtimeSanitizer.cpp
index 88cb04695217d5..890c79ae96718a 100644
--- a/llvm/lib/Transforms/Instrumentation/RealtimeSanitizer.cpp
+++ b/llvm/lib/Transforms/Instrumentation/RealtimeSanitizer.cpp
@@ -17,12 +17,16 @@
 #include "llvm/IR/IRBuilder.h"
 #include "llvm/IR/InstIterator.h"
 #include "llvm/IR/Module.h"
+#include "llvm/Transforms/Utils/ModuleUtils.h"
 
 #include "llvm/Demangle/Demangle.h"
 #include "llvm/Transforms/Instrumentation/RealtimeSanitizer.h"
 
 using namespace llvm;
 
+const char kRtsanModuleCtorName[] = "rtsan.module_ctor";
+const char kRtsanInitName[] = "__rtsan_ensure_initialized";
+
 static SmallVector<Type *> getArgTypes(ArrayRef<Value *> FunctionArgs) {
   SmallVector<Type *> Types;
   for (Value *Arg : FunctionArgs)
@@ -89,3 +93,14 @@ PreservedAnalyses RealtimeSanitizerPass::run(Function &Fn,
 
   return PreservedAnalyses::all();
 }
+
+PreservedAnalyses ModuleRealtimeSanitizerPass::run(Module &M,
+                                                   ModuleAnalysisManager &MAM) {
+  getOrCreateSanitizerCtorAndInitFunctions(
+      M, kRtsanModuleCtorName, kRtsanInitName, /*InitArgTypes=*/{},
+      /*InitArgs=*/{},
+      // This callback is invoked when the functions are created the first
+      // time. Hook them into the global ctors list in that case:
+      [&](Function *Ctor, FunctionCallee) { appendToGlobalCtors(M, Ctor, 0); });
+  return PreservedAnalyses::none();
+}
diff --git a/llvm/test/Instrumentation/RealtimeSanitizer/rtsan.ll b/llvm/test/Instrumentation/RealtimeSanitizer/rtsan.ll
index a0bc4aef2cc319..41b610d040e51c 100644
--- a/llvm/test/Instrumentation/RealtimeSanitizer/rtsan.ll
+++ b/llvm/test/Instrumentation/RealtimeSanitizer/rtsan.ll
@@ -1,4 +1,4 @@
-; RUN: opt < %s -passes=rtsan -S | FileCheck %s
+; RUN: opt < %s -passes='function(rtsan),module(rtsan-module)' -S | FileCheck %s
 
 define void @violation() #0 {
   %1 = alloca ptr, align 8
@@ -25,3 +25,7 @@ attributes #0 = { mustprogress noinline sanitize_realtime optnone ssp uwtable(sy
 ; RealtimeSanitizer pass should insert __rtsan_realtime_exit right before function return
 ; CHECK: call{{.*}}@__rtsan_realtime_exit
 ; CHECK-NEXT: ret{{.*}}void
+
+; RealtimeSanitizer pass should insert call to initialize the runtime
+; CHECK: define internal void @rtsan.module_ctor()
+; CHECK: call void @__rtsan_ensure_initialized()

[vitalybuka]

@cjappl
@why do we use this location and not addSanitizers? In particular registerOptimizerLastEPCallback?

[cjappl]

I'll investigate if moving it is possible! Off the top of my head I don't know if there is a reason.

I'll report back with PRs on either/both of these soon.

[vitalybuka]

You need to consider performance vs quality.

For other sanitizers it's essentially:

1. If it's early - better precision - in sense we detect issues in C/C++ program which can be optimized out. (can Realtime related issue be optimized out, and do you care about them at all)
2. If it's late - better performance, but we may miss bugs
   I believe performance is ~2x difference, so we can't afford that e.g. in Asan.

[vitalybuka]

Also we converted all but tsan from function+module to just module pass. Easier to maintain.

[cjappl]

Sounds good, I'll work on swapping rtsan over.