UBSAN crashes when calling a JIT compiled function

[kobalicek]

Clang 17+ (and the current master) crashes with UBSAN when a JIT compiled function is being called. For some reason it crashes on address, which is funcPtr - 8. Note that both ASAN and valgrind pass without any issues. Clang's 16 UBSAN and older work, GCC's UBSAN works as well.

It seems to me that something is wrong with clang's 17+ UBSAN, but I have no idea why funcPtr - 8 is the problem - such arithmetic is not used in the code; it seems injected by UBSAN.

I have been able to make this reproducible with asmjit library, without the need to modify anything:

git clone https://github.com/asmjit/asmjit.git
cd asmjit/tools
CC=clang CXX=clang++ ./configure-sanitizers.sh
cd ..
cd build/Release_UBSAN
make
./asmjit_test_emitters

When ./asmjit_test_emitters is executed, the output is the following:

AsmJit Emitters Test-Suite v1.10.0

Using x86::Assembler:
  mov rax, rdi
  mov rcx, rsi
  movdqu xmm0, [rcx]
  movdqu xmm1, [rdx]
  paddd xmm0, xmm1
  movdqu [rax], xmm0
  ret
UndefinedBehaviorSanitizer:DEADLYSIGNAL
==309866==ERROR: UndefinedBehaviorSanitizer: SEGV on unknown address 0x7f03e8aacff8 (pc 0x555d011597e7 bp 0x7fff4f1e86e0 sp 0x7fff4f1e82e0 T309866)
==309866==The signal is caused by a READ memory access.
    #0 0x555d011597e7  (/home/petr/workspace/asmjit/build/Release_UBSAN/asmjit_test_emitters+0x327e7)
    #1 0x555d01158ba3  (/home/petr/workspace/asmjit/build/Release_UBSAN/asmjit_test_emitters+0x31ba3)
    #2 0x7f03e86456c9  (/lib/x86_64-linux-gnu/libc.so.6+0x276c9) (BuildId: 072feb34c63e054d60d94cbc68d92e4caad25d72)
    #3 0x7f03e8645784  (/lib/x86_64-linux-gnu/libc.so.6+0x27784) (BuildId: 072feb34c63e054d60d94cbc68d92e4caad25d72)
    #4 0x555d0112e5a0  (/home/petr/workspace/asmjit/build/Release_UBSAN/asmjit_test_emitters+0x75a0)

UndefinedBehaviorSanitizer can not provide additional info.
SUMMARY: UndefinedBehaviorSanitizer: SEGV (/home/petr/workspace/asmjit/build/Release_UBSAN/asmjit_test_emitters+0x327e7) 
==309866==ABORTING

The logger is okay - it successfully assembled the function, however, it then fails when trying to execute it. When I have debugged this, I have found out that the pointer to the function is actually 0x7f03e8aad000, but UBSAN fails with 0x7f03e8aacff8 address, which is 8 bytes less and doesn't exist.

To add a bit more to the context, here is a snippet of the code from the test:

  // Add the code generated to the runtime.
  SumIntsFunc fn;
  err = rt.add(&fn, &code);

  if (err) {
    printf("** FAILURE: JitRuntime::add() failed (%s) **\n", DebugUtils::errorAsString(err));
    return 1;
  }

  // Execute the generated function.
  int inA[4] = { 4, 3, 2, 1 };
  int inB[4] = { 1, 5, 2, 8 };
  int out[4];

  // <---- IT FAILS HERE, WHEN CALLING `fn`
  fn(out, inA, inB);

  // Should print {5 8 4 9}.
  printf("Result = { %d %d %d %d }\n\n", out[0], out[1], out[2], out[3]);

  rt.release(fn);
  return !(out[0] == 5 && out[1] == 8 && out[2] == 4 && out[3] == 9);

It fails exactly when calling fn via a pointer to function. UBSAN for some reason accesses fn - 8 (probably to get some metadata, not sure) and then crashes. AsmJit's virtual memory allocator uses mmap() or VirtualAlloc() to allocate executable memory, which is used only for executable code (there is nothing else stored to maintain the pages), and the first generated function is exactly at the beginning of the allocated page. I would say with 99.9999% confidence that UBSAN is trying to read from a memory that is not mapped in this case. And I personally consider this a bug, because I'm not aware of any material which would state that a function cannot start at the beginning of a page.

OS: Debian Linux
Architecture: x86_64
Clang Version: 17+, master

[llvmbot]

@llvm/issue-subscribers-orcjit

[kobalicek]

This is the commit that introduces the regression:

• ad31a2d

It states that "It is extremely unlikely for two words preceding a function entry to be inaccessible. One way to achieve this is by ensuring that a function is aligned at a page boundary and making the preceding page unmapped or unreadable. This is not reasonable for application or library code."

However, this is exactly what JIT compilers are doing to take advantage of every byte of executable memory allocated. Many JIT compilers place the function entry at the beginning of the allocated page, and I don't see anything wrong on this - why to waste a cache line?

BTW this is not just about asmjit - UBSAN crashing on JIT function calls would make its use problematic in many other projects that use JIT compilation either directly or indirectly - from asmjit, xbyak, sljit, to projects that use sljit like pcre2. There is dozens of projects that use asmjit and its virtual memory allocator.

[kobalicek]

Please don't release clang 17 with this regression.

[lhames]

Removing the orcjit tag -- this doesn't seem to be ORC related.

[lhames]

@MaskRay -- you're probably best placed to comment on this?

Not sure if this helps, but two potential workarounds that leap to mind are: (1) add 8 bytes of padding to the start of your code allocations when running under an UBSan'd JIT, or (2) add a non-UBSan'd trampoline to bounce to JIT'd code.

[kobalicek]

I think, that UBSAN should help us with finding bugs in software and not force us to write software to workaround bugs in UBSAN.

If this hits an internal implementation detail of Clang's UBSAN, which assumes something that is not safe, then I think there is only one place where this should be fixed.

This is a regression, something that has worked until clang-16 is now broken.

[lhames]

This is a regression, something that has worked until clang-16 is now broken.

As a general observation "this worked for me yesterday" is not a strong argument. (E.g. we've had people insist that we maintain a particular result for operations with undefined behavior between compiler versions, but that's not a reasonable thing to expect).

I'm not sure what invariants (if any) are promised for the interaction between ubsan'd an non-ubsan'd code. We'll need a UBSan expert to weigh in on that. They may argue that this is an implementation choice though, not a bug. At that point we'd need to weigh up the cost of reverting the UBSan change (regressing UBSan + BTI / IBT interop in the process) vs clients implementing workarounds.

[lhames]

I've pinged #sanitizers on the LLVM discord to see if we can get someone with a definitive answer to this.

Also (belatedly) -- thank you for tracking down the offending commit and analyzing the issue. That's a huge help, whatever the answer turns out to be.