[ASan] Ensure Symbolize Flag setting on Windows through __asan_default_options() is maintained throughout runtime

[39otsu]

As a consequence of the ASAN DLL's initialization process on Windows, some flags defined by the user through overriding the __asan_default_options() method will not be honored. More information here: #117925

This PR aims to alleviate this for the symbolize flag in relation to this user's concern here.

1. Declared Symbolizer::UpdateSymbolizerTools().
2. Defined Symbolizer::UpdateSymbolizerTools(). Upon invocation of the weak function callback of __asan_default_options(), Symbolizer::tools_ will be cleared if the user specifies symbolize=0.
3. Added tests.

[github-actions]

Thank you for submitting a Pull Request (PR) to the LLVM Project!

This PR will be automatically labeled and the relevant teams will be notified.

If you wish to, you can add reviewers by using the "Reviewers" section on this page.

If this is not working for you, it is probably because you do not have write permissions for the repository. In which case you can instead tag reviewers by name in a comment by using @ followed by their GitHub username.

If you have received no comments on your PR for a week, you can request a review by "ping"ing the PR by adding a comment “Ping”. The common courtesy "ping" rate is once a week. Please remember that you are asking for valuable time from other developers.

If you have further questions, they may be answered by the LLVM GitHub User Guide.

You can also ask questions in a comment on this PR, on the LLVM Discord or on the forums.

[llvmbot]

@llvm/pr-subscribers-compiler-rt-sanitizer

Author: MacGyver Codilla (39otsu)

Changes

As a consequence of the ASAN DLL's initialization process on Windows, some flags defined by the user through overriding the __asan_default_options() method will not be honored. More information here: #117925

This PR aims to alleviate this for the symbolize flag in relation to this user's concern here.

1. Declared Symbolizer::UpdateSymbolizerTools().
2. Defined Symbolizer::UpdateSymbolizerTools(). Upon invocation of the weak function callback of __asan_default_options(), Symbolizer::tools_ will be cleared if the user specifies symbolize=0.
3. Added tests.

---

Full diff: https://github.com/llvm/llvm-project/pull/132811.diff

4 Files Affected:

• (modified) compiler-rt/lib/asan/asan_flags.cpp (+2-1)
• (modified) compiler-rt/lib/sanitizer_common/sanitizer_symbolizer.h (+3)
• (modified) compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_libcdep.cpp (+13)
• (added) compiler-rt/test/asan/TestCases/Windows/symbolize.cpp (+38)

diff --git a/compiler-rt/lib/asan/asan_flags.cpp b/compiler-rt/lib/asan/asan_flags.cpp
index 9cfb70bd00c78..955560d77a683 100644
--- a/compiler-rt/lib/asan/asan_flags.cpp
+++ b/compiler-rt/lib/asan/asan_flags.cpp
@@ -247,7 +247,8 @@ void InitializeFlags() {
         // See GH issue 'https://github.com/llvm/llvm-project/issues/117925' for
         // details.
         SetAllocatorMayReturnNull(common_flags()->allocator_may_return_null);
-      });
+        Symbolizer::UpdateSymbolizerTools();
+    });
 
 #  if CAN_SANITIZE_UB
   AddRegisterWeakFunctionCallback(
diff --git a/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer.h b/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer.h
index bd89dc4e302fc..604f143513193 100644
--- a/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer.h
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer.h
@@ -136,6 +136,9 @@ class Symbolizer final {
   /// (if it wasn't already initialized).
   static Symbolizer *GetOrInit();
   static void LateInitialize();
+#if SANITIZER_WINDOWS
+  static void UpdateSymbolizerTools();
+#endif
   // Returns a list of symbolized frames for a given address (containing
   // all inlined functions, if necessary).
   SymbolizedStack *SymbolizePC(uptr address);
diff --git a/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_libcdep.cpp b/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_libcdep.cpp
index 74458028ae8f5..320f92746df0c 100644
--- a/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_libcdep.cpp
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_libcdep.cpp
@@ -26,6 +26,19 @@ Symbolizer *Symbolizer::GetOrInit() {
   return symbolizer_;
 }
 
+#if SANITIZER_WINDOWS
+// If the 'symbolize' flag is set to 0, it clears the tools
+// associated with the symbolizer to prevent unnecessary symbolization and
+// resource usage. This is necessary because of the late binding of the
+// overridden method, __asan_default_options().
+void Symbolizer::UpdateSymbolizerTools() {
+  SpinMutexLock l(&init_mu_);
+  if (!common_flags()->symbolize) {
+    symbolizer_->tools_.clear();
+  }
+}
+#endif
+
 // See sanitizer_symbolizer_markup.cpp.
 #if !SANITIZER_SYMBOLIZER_MARKUP
 
diff --git a/compiler-rt/test/asan/TestCases/Windows/symbolize.cpp b/compiler-rt/test/asan/TestCases/Windows/symbolize.cpp
new file mode 100644
index 0000000000000..7f8cd5aea633b
--- /dev/null
+++ b/compiler-rt/test/asan/TestCases/Windows/symbolize.cpp
@@ -0,0 +1,38 @@
+// RUN: %clangxx_asan -O0 %s -o %t
+// RUN: %env_asan_opts=symbolize=0 not %run %t 2>&1 | FileCheck %s --check-prefix=CHECK-SYMBOLIZE-OFF
+// RUN: %env_asan_opts=symbolize=1 not %run %t 2>&1 | FileCheck %s --check-prefix=CHECK-SYMBOLIZE-ON
+
+// RUN: %clangxx_asan -O0 %s -o %t -DUSER_FUNCTION_OFF
+// RUN: not %run %t 2>&1 | FileCheck %s --check-prefix=CHECK-SYMBOLIZE-OFF
+// RUN: %env_asan_opts=symbolize=0 not %run %t 2>&1 | FileCheck %s --check-prefix=CHECK-SYMBOLIZE-OFF
+// RUN: %env_asan_opts=symbolize=1 not %run %t 2>&1 | FileCheck %s --check-prefix=CHECK-SYMBOLIZE-ON
+
+// RUN: %clangxx_asan -O0 %s -o %t -DUSER_FUNCTION_ON
+// RUN: not %run %t 2>&1 | FileCheck %s --check-prefix=CHECK-SYMBOLIZE-ON
+// RUN: %env_asan_opts=symbolize=0 not %run %t 2>&1 | FileCheck %s --check-prefix=CHECK-SYMBOLIZE-OFF
+// RUN: %env_asan_opts=symbolize=1 not %run %t 2>&1 | FileCheck %s --check-prefix=CHECK-SYMBOLIZE-ON
+#if USER_FUNCTION_OFF
+
+extern "C" __declspec(dllexport) extern const char *__asan_default_options() {
+  return "symbolize=0";
+}
+
+#endif
+
+#if USER_FUNCTION_ON
+
+extern "C" __declspec(dllexport) extern const char *__asan_default_options() {
+  return "symbolize=1";
+}
+
+#endif
+
+#include <cstdio>
+#include <cstdlib>
+
+volatile static int heapBufferOverflowValue = 10;
+int main() {
+  int *array = new int[10];
+  heapBufferOverflowValue = array[10]; // CHECK-SYMBOLIZE-ON: symbolize.cpp:36
+  return 0; // CHECK-SYMBOLIZE-OFF: symbolize.cpp.tmp+0x
+}
\ No newline at end of file