[BOLT] Instrumentation clobbers used stack slot

[nikic]

When instrumenting a PGO-optimized libLLVM.so, the instrumented binary binary crashes in _ZNK4llvm4Loop18isLoopSimplifyFormEv. See https://gist.github.com/nikic/e695bd62d1dd40e506f6365b935d417c for the instrumented and uninstrumented assembly of the function.

The problem is that the value spilled by mov %r8,-0x10(%rsp) is later clobbered by the pushes in the instrumentation sequence:

   0x00000000075873d9 <+975>:	push   %rax
   0x00000000075873da <+976>:	mov    $0x0,%eax
   0x00000000075873df <+981>:	lahf   
   0x00000000075873e0 <+982>:	push   %rax
   0x00000000075873e1 <+983>:	mov    $0x0,%eax
   0x00000000075873e6 <+988>:	seto   %al
   0x00000000075873e9 <+991>:	lock incq 0x57fa3d7(%rip)        # 0xcd817c8
   0x00000000075873f1 <+999>:	add    $0x7f,%al
   0x00000000075873f4 <+1002>:	pop    %rax
   0x00000000075873f5 <+1003>:	sahf   
   0x00000000075873f6 <+1004>:	pop    %rax

I've uploaded the libLLVM.so in question here: https://drive.google.com/file/d/1lAbqukTx7b1aPasrR7rP-p2nsoRQSgUa/view?usp=sharing

I've also included an opt binary and a test case which make it possible to test the instrumented/uninstrumented libraries as follows:

# Runs fine
LD_LIBRARY_PATH=$PWD valgrind ./opt -S -passes=simple-loop-unswitch < test.ll

# Instrument
mv libLLVM-16-rust-1.69.0-nightly.so libLLVM-16-rust-1.69.0-nightly.so.orig
llvm-bolt -instrument libLLVM-16-rust-1.69.0-nightly.so.orig -o libLLVM-16-rust-1.69.0-nightly.so --instrumentation-file-append-pid

# Produces "Conditional jump or move depends on uninitialised value(s)"
LD_LIBRARY_PATH=$PWD valgrind ./opt -S -passes=simple-loop-unswitch < test.ll

[llvmbot]

@llvm/issue-subscribers-bolt

[aaupov]

Reproduced with

$ llvm-bolt -instrument libLLVM-16-rust-1.69.0-nightly.so.orig -o libLLVM-16-rust-1.69.0-nightly.so -funcs=_ZNK4llvm4Loop18isLoopSimplifyFormEv
$ LD_LIBRARY_PATH=$PWD valgrind ./opt -S -passes=simple-loop-unswitch < test.ll
...
==13120== Conditional jump or move depends on uninitialised value(s)          
==13120==    at 0xA400109: ??? (in /home/aaupov/llvm-build-debug/bolt_repro/libLLVM-16-rust-1.69.0-nightly.so)                                               
==13120==    by 0x1FFEFFD3B7: ???                                                                                                                            
...

[aaupov]

BOLT recognizes the function as non-leaf. We assume we can clobber red zone if the function is non-leaf. But it looks like in this case the function has no calls in its body except a tail call at the end, so it can clobber its red zone. https://reviews.llvm.org/D145202 fixes this issue and I think is an appropriate solution.

[nikic]

Thanks! I've verified that this patch fixes the original build (I applied it on top of release/16.x).

[aaupov]

Thank you for a reproducer and a test case!