[BOLT][AArch64] Assertion `CurrentState == State::Empty && "cannot ignore non-empty functions in current mode"' failed.

[sig-loha-huang]

BOLT-INFO: shared object or position-independent executable detected
BOLT-INFO: Target architecture: aarch64
BOLT-INFO: BOLT version: 87a4215
BOLT-INFO: first alloc address is 0x0
BOLT-INFO: creating new program header table at address 0xc20000, offset 0xc20000
BOLT-INFO: enabling relocation mode
BOLT-INFO: pre-processing profile using branch profile reader
llvm-bolt: /home/llvm/llvm-bolt/llvm-project/bolt/lib/Core/BinaryFunction.cpp:3220: void llvm::bolt::BinaryFunction::setIgnored(): Assertion `CurrentState == State::Empty && "cannot ignore non-empty functions in current mode"' failed.
#0 0x00000064f4e30d10 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (.localalias) (/usr/bin/llvm-bolt+0xd00d10)
#1 0x00000064f4e2eb20 llvm::sys::RunSignalHandlers() (.localalias) (/usr/bin/llvm-bolt+0xcfeb20)
#2 0x00000064f4e2eca0 SignalHandler(int, siginfo_t*, void*) (/usr/bin/llvm-bolt+0xcfeca0)
#3 0x00000072b41cf848 (linux-vdso.so.1+0x848)
#4 0x00000072b3d48e6c raise ./signal/../sysdeps/unix/sysv/linux/raise.c:51:1
#5 0x00000072b3d35a60 abort ./stdlib/abort.c:81:7
#6 0x00000072b3d42438 __assert_fail_base ./assert/assert.c:89:7
#7 0x00000072b3d4249c (/lib/aarch64-linux-gnu/libc.so.6+0x2d49c)
#8 0x00000064f55f6ad0 llvm::bolt::BinaryFunction::postProcessEntryPoints() (/usr/bin/llvm-bolt+0x14c6ad0)
#9 0x00000064f54f11e8 llvm::bolt::MarkRAStates::runOnFunction(llvm::bolt::BinaryFunction&) (.localalias) (/usr/bin/llvm-bolt+0x13c11e8)
#10 0x00000064f565f684 std::_Function_handler<void (), std::_Bind<llvm::bolt::ParallelUtilities::runOnEachFunction(llvm::bolt::BinaryContext&, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function<void (llvm::bolt::BinaryFunction&)>, std::function<bool (llvm::bolt::BinaryFunction const&)>, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, bool, unsigned int)::'lambda'(std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction> >, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction> >) (std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction> >, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction> >)> >::_M_invoke(std::_Any_data const&) (/usr/bin/llvm-bolt+0x152f684)
#11 0x00000064f4f0efe8 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<std::function<void ()> > >, void> >::_M_invoke(std::_Any_data const&) (/usr/bin/llvm-bolt+0xddefe8)
#12 0x00000064f4f0f8dc std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) (/usr/bin/llvm-bolt+0xddf8dc)
#13 0x00000072b417e3ec __pthread_once_slow ./nptl/pthread_once.c:118:7
#14 0x00000064f4f0f780 std::__future_base::_Deferred_state<std::thread::_Invoker<std::tuple<std::function<void ()> > >, void>::_M_complete_async() (/usr/bin/llvm-bolt+0xddf780)
#15 0x00000064f4f13b70 std::_Function_handler<void (), std::shared_future llvm::ThreadPoolInterface::asyncImpl(std::function<void ()>, llvm::ThreadPoolTaskGroup*)::'lambda'()>::_M_invoke(std::_Any_data const&) (/usr/bin/llvm-bolt+0xde3b70)
#16 0x00000064f64263d8 llvm::StdThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) (.localalias) (/usr/bin/llvm-bolt+0x22f63d8)
#17 0x00000064f6426fa4 void* llvm::thread::ThreadProxy<std::tuplellvm::StdThreadPool::grow(int)::'lambda'() >(void*) (/usr/bin/llvm-bolt+0x22f6fa4)
#18 0x00000072b4175648 start_thread ./nptl/pthread_create.c:477:8
#19 0x00000072b3de6c9c ./misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:81:0
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
./bolt_perf2bolt.sh: line 114: 2370 Aborted llvm-bolt "$sym_path" -o "$bolt_output" -data="$input_fdata" --reorder-blocks=ext-tsp --reorder-functions=cdsort --split-functions --split-strategy=profile2 --split-eh --dyno-stats --no-huge-pages --align-text=0x4000 --use-old-text

• Result: FAILED (llvm-bolt returned error)

[llvmbot]

@llvm/issue-subscribers-bolt

Author: sig-loha-lh (sig-loha-huang)

BOLT-INFO: shared object or position-independent executable detected BOLT-INFO: Target architecture: aarch64 BOLT-INFO: BOLT version: 87a4215 BOLT-INFO: first alloc address is 0x0 BOLT-INFO: creating new program header table at address 0xc20000, offset 0xc20000 BOLT-INFO: enabling relocation mode BOLT-INFO: pre-processing profile using branch profile reader llvm-bolt: /home/llvm/llvm-bolt/llvm-project/bolt/lib/Core/BinaryFunction.cpp:3220: void llvm::bolt::BinaryFunction::setIgnored(): Assertion `CurrentState == State::Empty && "cannot ignore non-empty functions in current mode"' failed. #0 0x00000064f4e30d10 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (.localalias) (/usr/bin/llvm-bolt+0xd00d10) #1 0x00000064f4e2eb20 llvm::sys::RunSignalHandlers() (.localalias) (/usr/bin/llvm-bolt+0xcfeb20) #2 0x00000064f4e2eca0 SignalHandler(int, siginfo_t*, void*) (/usr/bin/llvm-bolt+0xcfeca0) #3 0x00000072b41cf848 (linux-vdso.so.1+0x848) #4 0x00000072b3d48e6c raise ./signal/../sysdeps/unix/sysv/linux/raise.c:51:1 #5 0x00000072b3d35a60 abort ./stdlib/abort.c:81:7 #6 0x00000072b3d42438 __assert_fail_base ./assert/assert.c:89:7 #7 0x00000072b3d4249c (/lib/aarch64-linux-gnu/libc.so.6+0x2d49c) #8 0x00000064f55f6ad0 llvm::bolt::BinaryFunction::postProcessEntryPoints() (/usr/bin/llvm-bolt+0x14c6ad0) #9 0x00000064f54f11e8 llvm::bolt::MarkRAStates::runOnFunction(llvm::bolt::BinaryFunction&) (.localalias) (/usr/bin/llvm-bolt+0x13c11e8) #10 0x00000064f565f684 std::_Function_handler<void (), std::_Bind<llvm::bolt::ParallelUtilities::runOnEachFunction(llvm::bolt::BinaryContext&, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function<void (llvm::bolt::BinaryFunction&)>, std::function<bool (llvm::bolt::BinaryFunction const&)>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, unsigned int)::'lambda'(std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction> >, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction> >) (std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction> >, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction> >)> >::_M_invoke(std::_Any_data const&) (/usr/bin/llvm-bolt+0x152f684) #11 0x00000064f4f0efe8 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<std::function<void ()> > >, void> >::_M_invoke(std::_Any_data const&) (/usr/bin/llvm-bolt+0xddefe8) #12 0x00000064f4f0f8dc std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) (/usr/bin/llvm-bolt+0xddf8dc) #13 0x00000072b417e3ec __pthread_once_slow ./nptl/pthread_once.c:118:7 #14 0x00000064f4f0f780 std::__future_base::_Deferred_state<std::thread::_Invoker<std::tuple<std::function<void ()> > >, void>::_M_complete_async() (/usr/bin/llvm-bolt+0xddf780) #15 0x00000064f4f13b70 std::_Function_handler<void (), std::shared_future<void> llvm::ThreadPoolInterface::asyncImpl<void>(std::function<void ()>, llvm::ThreadPoolTaskGroup*)::'lambda'()>::_M_invoke(std::_Any_data const&) (/usr/bin/llvm-bolt+0xde3b70) #16 0x00000064f64263d8 llvm::StdThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) (.localalias) (/usr/bin/llvm-bolt+0x22f63d8) #17 0x00000064f6426fa4 void* llvm::thread::ThreadProxy<std::tuple<llvm::StdThreadPool::grow(int)::'lambda'()> >(void*) (/usr/bin/llvm-bolt+0x22f6fa4) #18 0x00000072b4175648 start_thread ./nptl/pthread_create.c:477:8 #19 0x00000072b3de6c9c ./misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:81:0 PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace. ./bolt_perf2bolt.sh: line 114: 2370 Aborted llvm-bolt "$sym_path" -o "$bolt_output" -data="$input_fdata" --reorder-blocks=ext-tsp --reorder-functions=cdsort --split-functions --split-strategy=profile2 --split-eh --dyno-stats --no-huge-pages --align-text=0x4000 --use-old-text - Result: FAILED (llvm-bolt returned error)

[paschalis-mpeis]

Hey @sig-loha-huang,

Is there a particular reason you are using --use-old-text? Also what version of BOLT you are using?

At a quick glance, here's what I think is hapenning:

• The stack trace shows MarkRAStates, so I assume you use some version based on:
  • [BOLT][AArch64] Handle OpNegateRAState to enable optimizing binaries with pac-ret hardening #120064
  • Probably not the latest version, as a flag was introduced recently
  • That PR ignores some functions it finds to have an incompatible NegateRA state
• You are also using --use-old-text, which forces all functions to be processed

As a result, a function ignored by #120064, is now forced with -use-old-text, triggering an assertion. Given these circumstances, this is expected.

[paschalis-mpeis]

Assuming the above is correct, there are two options:

• if you’re using use-old-text to reduce code size, try --lite mode. It's not fully complete yet, but if you don’t see any assertion failures, it’ll most likely work fine
• if you don’t need pointer authentication, disable it when compiling your code.

---

Note to @bgergely0:
As an enhancement, we might need to exit with more gracefully with options like --use-old-text, --strict, etc.

Also, it could be nice to verify whether an input function was ignored because the compiler misused CFIs, or for another reason. If so, you may need input from @sig-loha-huang.

[sig-loha-huang]

hi @paschalis-mpeis

thanks for your reply

1.Yes, pointer authentication has already disabled.

2. When I remove --use-old-text, the Assert problem does not occur . this is the result log
   BOLT-INFO: shared object or position-independent executable detected
   BOLT-INFO: Target architecture: aarch64
   BOLT-INFO: BOLT version: 87a4215
   BOLT-INFO: first alloc address is 0x0
   BOLT-INFO: creating new program header table at address 0xa20000, offset 0xa20000
   BOLT-INFO: enabling relocation mode
   BOLT-INFO: pre-processing profile using branch profile reader
   BOLT-INFO: number of removed linker-inserted veneers: 0
   BOLT-INFO: 4341 out of 17018 functions in the binary (25.5%) have non-empty execution profile
   BOLT-INFO: 95 functions with profile could not be optimized
   BOLT-WARNING: 32 (0.7% of all profiled) functions have invalid (possibly stale) profile. Use -report-stale to see the list.
   BOLT-WARNING: 4409596 out of 39272729 samples in the binary (11.2%) belong to functions with invalid (possibly stale) profile.
   BOLT-INFO: 0 stale functions (0.0% of all stale) have matching block count.
   BOLT-INFO: 0 stale blocks (nan% of all stale) have matching icount.
   BOLT-INFO: profile for 1 objects was ignored
   BOLT-INFO: profile quality metrics for the hottest 1000 functions (reporting top 5% values): function CFG discontinuity 93.51%; call graph flow conservation gap 100.00%; CFG flow conservation gap 100.00% (weighted) 100.00% (worst)
   BOLT-INFO: basic block reordering modified layout of 3046 functions (70.17% of profiled, 16.96% of total)
   BOLT-INFO: 3 Functions were reordered by LoopInversionPass
   BOLT-INFO: program-wide dynostats after all optimizations before SCTC and FOP:

        14255214 : executed forward branches
         7004317 : taken forward branches
        10309943 : executed backward branches
         9503891 : taken backward branches
         2924904 : executed unconditional branches
         6552891 : all function calls
         2338013 : indirect calls
          433336 : PLT calls
       224468054 : executed instructions
        38175162 : executed load instructions
        33369890 : executed store instructions
               0 : taken jump table branches
               0 : taken unknown indirect branches
        27490061 : total branches
        19433112 : taken branches
         8056949 : non-taken conditional branches
        16508208 : taken conditional branches
        24565157 : all conditional branches
               0 : linker-inserted veneer calls
   
        12927145 : executed forward branches (-9.3%)
          555514 : taken forward branches (-92.1%)
        11638012 : executed backward branches (+12.9%)
         8110547 : taken backward branches (-14.7%)
          919039 : executed unconditional branches (-68.6%)
         6552891 : all function calls (=)
         2338013 : indirect calls (=)
          433336 : PLT calls (=)
       232660188 : executed instructions (+3.6%)
        38175162 : executed load instructions (=)
        33369890 : executed store instructions (=)
               0 : taken jump table branches (=)
               0 : taken unknown indirect branches (=)
        25484196 : total branches (-7.3%)
         9585100 : taken branches (-50.7%)
        15899096 : non-taken conditional branches (+97.3%)
         8666061 : taken conditional branches (-47.5%)
        24565157 : all conditional branches (=)
               0 : linker-inserted veneer calls (=)
   

BOLT-INFO: Starting stub-insertion pass
BOLT-INFO: Inserted 8259 stubs in the hot area and 4227 stubs in the cold area. Shared 0 times, iterated 6 times.
BOLT-INFO: patched build-id (flipped last bit)
BOLT-INFO: setting __hot_start to 0xa24000
BOLT-INFO: setting __hot_end to 0xb39fe4

3. When I use --lite mode. Another .so has an exception
   BOLT-INFO: shared object or position-independent executable detected
   BOLT-INFO: Target architecture: aarch64
   BOLT-INFO: BOLT version: 87a4215
   BOLT-INFO: first alloc address is 0x0
   BOLT-INFO: creating new program header table at address 0x110000, offset 0x110000
   BOLT-INFO: enabling relocation mode
   BOLT-INFO: enabling lite mode
   BOLT-INFO: pre-processing profile using branch profile reader
   llvm-bolt: /home/llvm/llvm-bolt/llvm-project/bolt/lib/Target/AArch64/AArch64MCPlusBuilder.cpp:2313: virtual std::optionalllvm::bolt::Relocation {anonymous}::AArch64MCPlusBuilder::createRelocation(const llvm::MCFixup&, const llvm::MCAsmBackend&) const: Assertion `FKI.TargetOffset == 0 && "0-bit relocation offset expected"' failed.
   #0 0x000000624c850d10 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (.localalias) (/usr/bin/llvm-bolt+0xd00d10)
   Fixing Rust build #1 0x000000624c84eb20 llvm::sys::RunSignalHandlers() (.localalias) (/usr/bin/llvm-bolt+0xcfeb20)
   Fix a typo #2 0x000000624c84eca0 SignalHandler(int, siginfo_t*, void*) (/usr/bin/llvm-bolt+0xcfeca0)
   Fix a compilation error on gcc8 #3 0x0000007efa5a3848 (linux-vdso.so.1+0x848)
   Fix a compilation error on gcc8 #4 0x0000007efa11ce6c raise ./signal/../sysdeps/unix/sysv/linux/raise.c:51:1
   Seedarg/member/wip #5 0x0000007efa109a60 abort ./stdlib/abort.c:81:7
   Add check for matching HeaderFilter before emitting Diagnostic #6 0x0000007efa116438 __assert_fail_base ./assert/assert.c:89:7
   Random engine swap #7 0x0000007efa11649c (/lib/aarch64-linux-gnu/libc.so.6+0x2d49c)
   Update rwlock init #8 0x000000624cfb6b58 (anonymous namespace)::AArch64MCPlusBuilder::createRelocation(llvm::MCFixup const&, llvm::MCAsmBackend const&) const (/usr/bin/llvm-bolt+0x1466b58)
   Next80 merge test #9 0x000000624d02bb68 llvm::bolt::BinaryFunction::scanExternalRefs() (/usr/bin/llvm-bolt+0x14dbb68)
   Fix typo: 'soruce' -> 'source' #10 0x000000624c8f766c llvm::bolt::RewriteInstance::disassembleFunctions() (.localalias) (/usr/bin/llvm-bolt+0xda766c)
   fix for gcc 8 #11 0x000000624c928584 llvm::bolt::RewriteInstance::run() (/usr/bin/llvm-bolt+0xdd8584)
   Fix "Allman" style example #12 0x000000624be30ca8 main (/usr/bin/llvm-bolt+0x2e0ca8)
   Defer diag #13 0x0000007efa109dd8 __libc_start_main ./csu/../csu/libc-start.c:342:3
   WIP lifetime categories #14 0x000000624beabc38 _start (/usr/bin/llvm-bolt+0x35bc38)
   PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
   Stack dump:
4. Program arguments: llvm-bolt /home/bolt/symble/libc.so -o bolt_output/libc.so.bolt -data=fdata_output/libc.so.fdata --reorder-blocks=ext-tsp --reorder-functions=cdsort --split-functions --split-strategy=profile2 --split-eh --dyno-stats --no-huge-pages --align-text=0x4000 --lite

Br

[paschalis-mpeis]

Hey @sig-loha-huang,

Thanks for the info.

Regarding (1), just to confirm: your input binary wasn't compiled with PAuth, right?

Also, were you using llvm-bolt version based on #120064? That hasn't merged yet.
What happens if you try with latest trunk instead?

Would it be okay not to use -use-old-text, or do you specifically need it for code size reductin?

Regarding lite mode:
I recommended --lite as I thought you wanted smaller binaries. This is the issue you are seeing:

• [BOLT][AArch64] 0-bit relocation offset expected #138264

On latest trunk a workaround has landed today #140669. Can you try that and report back? Proper support will come at some point later by #139565.