[libc++] Fix the behavior of throwing operator new under -fno-exceptions (#69498)

In D144319, Clang tried to land a change that would cause some functions
that are not supposed to return nullptr to optimize better. As reported
in https://reviews.llvm.org/D144319#4203982, libc++ started seeing
failures in its CI shortly after this change was landed.

As explained in D146379, the reason for these failures is that libc++'s
throwing `operator new` can in fact return nullptr when compiled with
exceptions disabled. However, this contradicts the Standard, which
clearly says that the throwing version of `operator new(size_t)` should
never return nullptr. This is actually a long standing issue. I've
previously seen a case where LTO would optimize incorrectly based on the
assumption that `operator new` doesn't return nullptr, an assumption
that was violated in that case because libc++.dylib was compiled with
-fno-exceptions.

Unfortunately, fixing this is kind of tricky. The Standard has a few
requirements for the allocation functions, some of which are impossible
to satisfy under -fno-exceptions:
1. `operator new(size_t)` must never return nullptr
2. `operator new(size_t, nothrow_t)` must call the throwing version and
return nullptr on failure to allocate
3. We can't throw exceptions when compiled with -fno-exceptions

In the case where exceptions are enabled, things work nicely.
`new(size_t)` throws and `new(size_t, nothrow_t)` uses a try-catch to
return nullptr. However, when compiling the library with
-fno-exceptions, we can't throw an exception from `new(size_t)`, and we
can't catch anything from `new(size_t, nothrow_t)`. The only thing we
can do from `new(size_t)` is actually abort the program, which does not
make it possible for `new(size_t, nothrow_t)` to catch something and
return nullptr.

This patch makes the following changes:
1. When compiled with -fno-exceptions, the throwing version of `operator
new` will now abort on failure instead of returning nullptr on failure.
This resolves the issue that the compiler could mis-compile based on the
assumption that nullptr is never returned. This constitutes an API and
ABI breaking change for folks compiling the library with -fno-exceptions
(which is not the general public, who merely uses libc++ headers but use
a shared library that has already been compiled). This should mostly
impact vendors and other folks who compile libc++.dylib themselves.

2. When the library is compiled with -fexceptions, the nothrow version
of `operator new` has no change. When the library is compiled with
-fno-exceptions, the nothrow version of `operator new` will now check
whether the throwing version of `operator new` has been overridden. If
it has not been overridden, then it will use an implementation
equivalent to that of the throwing `operator new`, except it will return
nullptr on failure to allocate (instead of terminating). However, if the
throwing `operator new` has been overridden, it is now an error NOT to
also override the nothrow `operator new`. Indeed, there is no way for us
to implement a valid nothrow `operator new` without knowing the exact
implementation of the throwing version.

In summary, this change will impact people who fall into the following
intersection of conditions:
- They use the libc++ shared/static library built with `-fno-exceptions`
- They do not override `operator new(..., std::nothrow_t)`
- They override `operator new(...)` (the throwing version)
- They use `operator new(..., std::nothrow_t)`

We believe this represents a small number of people.

Fixes #60129
rdar://103958777

Differential Revision: https://reviews.llvm.org/D150610

Author: ldionne

libcxx/docs/ReleaseNotes/18.rst

@@ -237,6 +237,29 @@ LLVM 20
 ABI Affecting Changes
 ---------------------
 
+- When the shared/static library is built with ``-fno-exceptions``, the behavior of ``operator new`` was changed
+  to make it standards-conforming. In LLVM 17 and before, the throwing versions of ``operator new`` would return
+  ``nullptr`` upon failure to allocate, when the shared/static library was built with exceptions disabled. This
+  was non-conforming, since the throwing versions of ``operator new`` are never expected to return ``nullptr``, and
+  this non-conformance could actually lead to miscompiles in subtle cases.
+
+  Starting in LLVM 18, the throwing versions of ``operator new`` will abort the program when they fail to allocate
+  if the shared/static library has been built with ``-fno-exceptions``. This is consistent with the behavior of all
+  other potentially-throwing functions in the library, which abort the program instead of throwing when ``-fno-exceptions``
+  is used.
+
+  Furthermore, when the shared/static library is built with ``-fno-exceptions``, users who override the throwing
+  version of ``operator new`` will now need to also override the ``std::nothrow_t`` version of ``operator new`` if
+  they want to use it. Indeed, this is because there is no way to implement a conforming ``operator new(nothrow)``
+  from a conforming potentially-throwing ``operator new`` when compiled with ``-fno-exceptions``. In that case, using
+  ``operator new(nothrow)`` without overriding it explicitly but after overriding the throwing ``operator new`` will
+  result in an error.
+
+  Note that this change only impacts vendors/users that build the shared/static library themselves and pass
+  ``-DLIBCXX_ENABLE_EXCEPTIONS=OFF``, which is not the default configuration. If you are using the default
+  configuration of the library, the libc++ shared/static library will be built with exceptions enabled, and
+  there is no change between LLVM 17 and LLVM 18, even for users who build their own code using ``-fno-exceptions``.
+
 - The symbol of a non-visible function part of ``std::system_error`` was removed.
   This is not a breaking change as the private function ``__init`` was never referenced internally outside of the dylib.
 

libcxx/src/include/overridable_function.h

@@ -0,0 +1,119 @@
+// -*- C++ -*-
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef _LIBCPP_SRC_INCLUDE_OVERRIDABLE_FUNCTION_H
+#define _LIBCPP_SRC_INCLUDE_OVERRIDABLE_FUNCTION_H
+
+#include <__config>
+#include <cstdint>
+
+#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
+#  pragma GCC system_header
+#endif
+
+//
+// This file provides the std::__is_function_overridden utility, which allows checking
+// whether an overridable function (typically a weak symbol) like `operator new`
+// has been overridden by a user or not.
+//
+// This is a low-level utility which does not work on all platforms, since it needs
+// to make assumptions about the object file format in use. Furthermore, it requires
+// the "base definition" of the function (the one we want to check whether it has been
+// overridden) to be annotated with the _LIBCPP_MAKE_OVERRIDABLE_FUNCTION_DETECTABLE macro.
+//
+// This currently works with Mach-O files (used on Darwin) and with ELF files (used on Linux
+// and others). On platforms where we know how to implement this detection, the macro
+// _LIBCPP_CAN_DETECT_OVERRIDDEN_FUNCTION is defined to 1, and it is defined to 0 on
+// other platforms. The _LIBCPP_MAKE_OVERRIDABLE_FUNCTION_DETECTABLE macro is defined to
+// nothing on unsupported platforms so that it can be used to decorate functions regardless
+// of whether detection is actually supported.
+//
+// How does this work?
+// -------------------
+//
+// Let's say we want to check whether a weak function `f` has been overridden by the user.
+// The general mechanism works by placing `f`'s definition (in the libc++ built library)
+// inside a special section, which we do using the `__section__` attribute via the
+// _LIBCPP_MAKE_OVERRIDABLE_FUNCTION_DETECTABLE macro.
+//
+// Then, when comes the time to check whether the function has been overridden, we take
+// the address of the function and we check whether it falls inside the special function
+// we created. This can be done by finding pointers to the start and the end of the section
+// (which is done differently for ELF and Mach-O), and then checking whether `f` falls
+// within those bounds. If it falls within those bounds, then `f` is still inside the
+// special section and so it is the version we defined in the libc++ built library, i.e.
+// it was not overridden. Otherwise, it was overridden by the user because it falls
+// outside of the section.
+//
+// Important note
+// --------------
+//
+// This mechanism should never be used outside of the libc++ built library. In particular,
+// attempting to use this within the libc++ headers will not work at all because we don't
+// want to be defining special sections inside user's executables which use our headers.
+// This is provided inside libc++'s include tree solely to make it easier to share with
+// libc++abi, which needs the same mechanism.
+//
+
+#if defined(_LIBCPP_OBJECT_FORMAT_MACHO)
+
+#  define _LIBCPP_CAN_DETECT_OVERRIDDEN_FUNCTION 1
+#  define _LIBCPP_MAKE_OVERRIDABLE_FUNCTION_DETECTABLE                                                                 \
+    __attribute__((__section__("__TEXT,__lcxx_override,regular,pure_instructions")))
+
+_LIBCPP_BEGIN_NAMESPACE_STD
+template <class _Ret, class... _Args>
+_LIBCPP_HIDE_FROM_ABI bool __is_function_overridden(_Ret (*__fptr)(_Args...)) noexcept {
+  // Declare two dummy bytes and give them these special `__asm` values. These values are
+  // defined by the linker, which means that referring to `&__lcxx_override_start` will
+  // effectively refer to the address where the section starts (and same for the end).
+  extern char __lcxx_override_start __asm("section$start$__TEXT$__lcxx_override");
+  extern char __lcxx_override_end __asm("section$end$__TEXT$__lcxx_override");
+
+  // Now get a uintptr_t out of these locations, and out of the function pointer.
+  uintptr_t __start = reinterpret_cast<uintptr_t>(&__lcxx_override_start);
+  uintptr_t __end   = reinterpret_cast<uintptr_t>(&__lcxx_override_end);
+  uintptr_t __ptr   = reinterpret_cast<uintptr_t>(__fptr);
+
+  // Finally, the function was overridden if it falls outside of the section's bounds.
+  return __ptr < __start || __ptr > __end;
+}
+_LIBCPP_END_NAMESPACE_STD
+
+#elif defined(_LIBCPP_OBJECT_FORMAT_ELF)
+
+#  define _LIBCPP_CAN_DETECT_OVERRIDDEN_FUNCTION 1
+#  define _LIBCPP_MAKE_OVERRIDABLE_FUNCTION_DETECTABLE __attribute__((__section__("__lcxx_override")))
+
+// This is very similar to what we do for Mach-O above. The ELF linker will implicitly define
+// variables with those names corresponding to the start and the end of the section.
+//
+// See https://stackoverflow.com/questions/16552710/how-do-you-get-the-start-and-end-addresses-of-a-custom-elf-section
+extern char __start___lcxx_override;
+extern char __stop___lcxx_override;
+
+_LIBCPP_BEGIN_NAMESPACE_STD
+template <class _Ret, class... _Args>
+_LIBCPP_HIDE_FROM_ABI bool __is_function_overridden(_Ret (*__fptr)(_Args...)) noexcept {
+  uintptr_t __start = reinterpret_cast<uintptr_t>(&__start___lcxx_override);
+  uintptr_t __end   = reinterpret_cast<uintptr_t>(&__stop___lcxx_override);
+  uintptr_t __ptr   = reinterpret_cast<uintptr_t>(__fptr);
+
+  return __ptr < __start || __ptr > __end;
+}
+_LIBCPP_END_NAMESPACE_STD
+
+#else
+
+#  define _LIBCPP_CAN_DETECT_OVERRIDDEN_FUNCTION 0
+#  define _LIBCPP_MAKE_OVERRIDABLE_FUNCTION_DETECTABLE /* nothing */
+
+#endif
+
+#endif // _LIBCPP_SRC_INCLUDE_OVERRIDABLE_FUNCTION_H

libcxx/src/new.cpp

@@ -6,7 +6,9 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "include/overridable_function.h"
 #include <__memory/aligned_alloc.h>
+#include <cstddef>
 #include <cstdlib>
 #include <new>
 
@@ -15,6 +17,10 @@
 // The code below is copied as-is into libc++abi's libcxxabi/src/stdlib_new_delete.cpp
 // file. The version in this file is the canonical one.
 
+inline void __throw_bad_alloc_shim() { std::__throw_bad_alloc(); }
+
+#  define _LIBCPP_ASSERT_SHIM(expr, str) _LIBCPP_ASSERT(expr, str)
+
 // ------------------ BEGIN COPY ------------------
 // Implement all new and delete operators as weak definitions
 // in this shared library, so that they can be overridden by programs
@@ -36,41 +42,63 @@ static void* operator_new_impl(std::size_t size) {
   return p;
 }
 
-_LIBCPP_WEAK void* operator new(std::size_t size) _THROW_BAD_ALLOC {
+_LIBCPP_MAKE_OVERRIDABLE_FUNCTION_DETECTABLE _LIBCPP_WEAK void* operator new(std::size_t size) _THROW_BAD_ALLOC {
   void* p = operator_new_impl(size);
-#  ifndef _LIBCPP_HAS_NO_EXCEPTIONS
   if (p == nullptr)
-    throw std::bad_alloc();
-#  endif
+    __throw_bad_alloc_shim();
   return p;
 }
 
 _LIBCPP_WEAK void* operator new(size_t size, const std::nothrow_t&) noexcept {
+#  ifdef _LIBCPP_HAS_NO_EXCEPTIONS
+#    if _LIBCPP_CAN_DETECT_OVERRIDDEN_FUNCTION
+  _LIBCPP_ASSERT_SHIM(
+      !std::__is_function_overridden(static_cast<void* (*)(std::size_t)>(&operator new)),
+      "libc++ was configured with exceptions disabled and `operator new(size_t)` has been overridden, "
+      "but `operator new(size_t, nothrow_t)` has not been overridden. This is problematic because "
+      "`operator new(size_t, nothrow_t)` must call `operator new(size_t)`, which will terminate in case "
+      "it fails to allocate, making it impossible for `operator new(size_t, nothrow_t)` to fulfill its "
+      "contract (since it should return nullptr upon failure). Please make sure you override "
+      "`operator new(size_t, nothrow_t)` as well.");
+#    endif
+
+  return operator_new_impl(size);
+#  else
   void* p = nullptr;
-#  ifndef _LIBCPP_HAS_NO_EXCEPTIONS
   try {
-#  endif // _LIBCPP_HAS_NO_EXCEPTIONS
     p = ::operator new(size);
-#  ifndef _LIBCPP_HAS_NO_EXCEPTIONS
   } catch (...) {
   }
-#  endif // _LIBCPP_HAS_NO_EXCEPTIONS
   return p;
+#  endif
 }
 
-_LIBCPP_WEAK void* operator new[](size_t size) _THROW_BAD_ALLOC { return ::operator new(size); }
+_LIBCPP_MAKE_OVERRIDABLE_FUNCTION_DETECTABLE _LIBCPP_WEAK void* operator new[](size_t size) _THROW_BAD_ALLOC {
+  return ::operator new(size);
+}
 
 _LIBCPP_WEAK void* operator new[](size_t size, const std::nothrow_t&) noexcept {
+#  ifdef _LIBCPP_HAS_NO_EXCEPTIONS
+#    if _LIBCPP_CAN_DETECT_OVERRIDDEN_FUNCTION
+  _LIBCPP_ASSERT_SHIM(
+      !std::__is_function_overridden(static_cast<void* (*)(std::size_t)>(&operator new[])),
+      "libc++ was configured with exceptions disabled and `operator new[](size_t)` has been overridden, "
+      "but `operator new[](size_t, nothrow_t)` has not been overridden. This is problematic because "
+      "`operator new[](size_t, nothrow_t)` must call `operator new[](size_t)`, which will terminate in case "
+      "it fails to allocate, making it impossible for `operator new[](size_t, nothrow_t)` to fulfill its "
+      "contract (since it should return nullptr upon failure). Please make sure you override "
+      "`operator new[](size_t, nothrow_t)` as well.");
+#    endif
+
+  return operator_new_impl(size);
+#  else
   void* p = nullptr;
-#  ifndef _LIBCPP_HAS_NO_EXCEPTIONS
   try {
-#  endif // _LIBCPP_HAS_NO_EXCEPTIONS
     p = ::operator new[](size);
-#  ifndef _LIBCPP_HAS_NO_EXCEPTIONS
   } catch (...) {
   }
-#  endif // _LIBCPP_HAS_NO_EXCEPTIONS
   return p;
+#  endif
 }
 
 _LIBCPP_WEAK void operator delete(void* ptr) noexcept { std::free(ptr); }
@@ -107,43 +135,66 @@ static void* operator_new_aligned_impl(std::size_t size, std::align_val_t alignm
   return p;
 }
 
-_LIBCPP_WEAK void* operator new(std::size_t size, std::align_val_t alignment) _THROW_BAD_ALLOC {
+_LIBCPP_MAKE_OVERRIDABLE_FUNCTION_DETECTABLE _LIBCPP_WEAK void*
+operator new(std::size_t size, std::align_val_t alignment) _THROW_BAD_ALLOC {
   void* p = operator_new_aligned_impl(size, alignment);
-#    ifndef _LIBCPP_HAS_NO_EXCEPTIONS
   if (p == nullptr)
-    throw std::bad_alloc();
-#    endif
+    __throw_bad_alloc_shim();
   return p;
 }
 
 _LIBCPP_WEAK void* operator new(size_t size, std::align_val_t alignment, const std::nothrow_t&) noexcept {
+#    ifdef _LIBCPP_HAS_NO_EXCEPTIONS
+#      if _LIBCPP_CAN_DETECT_OVERRIDDEN_FUNCTION
+  _LIBCPP_ASSERT_SHIM(
+      !std::__is_function_overridden(static_cast<void* (*)(std::size_t, std::align_val_t)>(&operator new)),
+      "libc++ was configured with exceptions disabled and `operator new(size_t, align_val_t)` has been overridden, "
+      "but `operator new(size_t, align_val_t, nothrow_t)` has not been overridden. This is problematic because "
+      "`operator new(size_t, align_val_t, nothrow_t)` must call `operator new(size_t, align_val_t)`, which will "
+      "terminate in case it fails to allocate, making it impossible for `operator new(size_t, align_val_t, nothrow_t)` "
+      "to fulfill its contract (since it should return nullptr upon failure). Please make sure you override "
+      "`operator new(size_t, align_val_t, nothrow_t)` as well.");
+#      endif
+
+  return operator_new_aligned_impl(size, alignment);
+#    else
   void* p = nullptr;
-#    ifndef _LIBCPP_HAS_NO_EXCEPTIONS
   try {
-#    endif // _LIBCPP_HAS_NO_EXCEPTIONS
     p = ::operator new(size, alignment);
-#    ifndef _LIBCPP_HAS_NO_EXCEPTIONS
   } catch (...) {
   }
-#    endif // _LIBCPP_HAS_NO_EXCEPTIONS
   return p;
+#    endif
 }
 
-_LIBCPP_WEAK void* operator new[](size_t size, std::align_val_t alignment) _THROW_BAD_ALLOC {
+_LIBCPP_MAKE_OVERRIDABLE_FUNCTION_DETECTABLE _LIBCPP_WEAK void*
+operator new[](size_t size, std::align_val_t alignment) _THROW_BAD_ALLOC {
   return ::operator new(size, alignment);
 }
 
 _LIBCPP_WEAK void* operator new[](size_t size, std::align_val_t alignment, const std::nothrow_t&) noexcept {
+#    ifdef _LIBCPP_HAS_NO_EXCEPTIONS
+#      if _LIBCPP_CAN_DETECT_OVERRIDDEN_FUNCTION
+  _LIBCPP_ASSERT_SHIM(
+      !std::__is_function_overridden(static_cast<void* (*)(std::size_t, std::align_val_t)>(&operator new[])),
+      "libc++ was configured with exceptions disabled and `operator new[](size_t, align_val_t)` has been overridden, "
+      "but `operator new[](size_t, align_val_t, nothrow_t)` has not been overridden. This is problematic because "
+      "`operator new[](size_t, align_val_t, nothrow_t)` must call `operator new[](size_t, align_val_t)`, which will "
+      "terminate in case it fails to allocate, making it impossible for `operator new[](size_t, align_val_t, "
+      "nothrow_t)` to fulfill its contract (since it should return nullptr upon failure). Please make sure you "
+      "override "
+      "`operator new[](size_t, align_val_t, nothrow_t)` as well.");
+#      endif
+
+  return operator_new_aligned_impl(size, alignment);
+#    else
   void* p = nullptr;
-#    ifndef _LIBCPP_HAS_NO_EXCEPTIONS
   try {
-#    endif // _LIBCPP_HAS_NO_EXCEPTIONS
     p = ::operator new[](size, alignment);
-#    ifndef _LIBCPP_HAS_NO_EXCEPTIONS
   } catch (...) {
   }
-#    endif // _LIBCPP_HAS_NO_EXCEPTIONS
   return p;
+#    endif
 }
 
 _LIBCPP_WEAK void operator delete(void* ptr, std::align_val_t) noexcept { std::__libcpp_aligned_free(ptr); }

libcxx/test/libcxx/language.support/support.dynamic/assert.nothrow_new_not_overridden_fno_exceptions.pass.cpp

@@ -0,0 +1,58 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// void* operator new(std::size_t, const std::nothrow_t&);
+// void* operator new(std::size_t, std::align_val_t, const std::nothrow_t&);
+// void* operator new[](std::size_t, const std::nothrow_t&);
+// void* operator new[](std::size_t, std::align_val_t, const std::nothrow_t&);
+
+// This test ensures that we catch the case where `new` has been overridden but `new(nothrow)`
+// has not been overridden, and the library is compiled with -fno-exceptions.
+//
+// In that case, it is impossible for libc++ to provide a Standards conforming implementation
+// of `new(nothrow)`, so the only viable option is to terminate the program.
+
+// REQUIRES: has-unix-headers
+// UNSUPPORTED: c++03
+
+// We only know how to diagnose this on platforms that use the ELF or Mach-O object file formats.
+// XFAIL: target={{.+}}-windows-{{.+}}
+
+// TODO: We currently don't have a way to express that the built library was
+//       compiled with -fno-exceptions, so if the library was built with support
+//       for exceptions but we run the test suite without exceptions, this will
+//       spuriously fail.
+// REQUIRES: no-exceptions
+
+#include <cstddef>
+#include <new>
+
+#include "check_assertion.h"
+
+// Override the throwing versions of operator new, but not the nothrow versions.
+alignas(32) char DummyData[32 * 3];
+void* operator new(std::size_t) { return DummyData; }
+void* operator new(std::size_t, std::align_val_t) { return DummyData; }
+void* operator new[](std::size_t) { return DummyData; }
+void* operator new[](std::size_t, std::align_val_t) { return DummyData; }
+
+void operator delete(void*) noexcept {}
+void operator delete(void*, std::align_val_t) noexcept {}
+void operator delete[](void*) noexcept {}
+void operator delete[](void*, std::align_val_t) noexcept {}
+
+int main(int, char**) {
+  std::size_t size       = 3;
+  std::align_val_t align = static_cast<std::align_val_t>(32);
+  EXPECT_ANY_DEATH((void)operator new(size, std::nothrow));
+  EXPECT_ANY_DEATH((void)operator new(size, align, std::nothrow));
+  EXPECT_ANY_DEATH((void)operator new[](size, std::nothrow));
+  EXPECT_ANY_DEATH((void)operator new[](size, align, std::nothrow));
+
+  return 0;
+}