[BOLT] Optimized binary segfaults when using --use-old-text

[Zentrik]

I'm using BOLT built from LLVM 18.1.4, I also see a segfault on 18.1.3 and 16.

I run bolt libjulia-internal.so.1.12.0.original -o libjulia-internal.so.1.12.0 --use-old-text -no-huge-pages (-no-huge-pages is needed otherwise it doesn't use the old text). Here's the log

BOLT-INFO: shared object or position-independent executable detected
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: e6c3289804a67ea0bb6a86fadbe454dd93b8d855
BOLT-INFO: first alloc address is 0x0
BOLT-INFO: creating new program header table at address 0x61e000, offset 0x61e000
BOLT-WARNING: debug info will be stripped from the binary. Use -update-debug-sections to keep it.
BOLT-INFO: enabling relocation mode
BOLT-INFO: enabling -align-macro-fusion=all since no profile was specified
BOLT-WARNING: Failed to analyze 22 relocations
BOLT-INFO: forcing -jump-tables=move as PIC jump table was detected in function scm_to_julia_/1(*2)
BOLT-INFO: 0 out of 5651 functions in the binary (0.0%) have non-empty execution profile
BOLT-INFO: the input contains 597 (dynamic count : 0) opportunities for macro-fusion optimization that are going to be fixed
BOLT-INFO: 36848 instructions were shortened
BOLT-INFO: removed 12783 empty blocks
BOLT-INFO: SCTC: patched 77 tail calls (71 forward) tail calls (6 backward) from a total of 77 while removing 0 double jumps and removing 52 basic blocks totalling 260 bytes of code. CTCs total execution count is 0 and the number of times CTCs are taken is 0
BOLT-INFO: using original .text for new code with 0x1000 alignment
BOLT-INFO: setting _end to 0x66669c
BOLT-INFO: patched build-id (flipped last bit)

I compiled with gcc with -fno-reorder-blocks-and-partition and -Wl,--emit-relocs.

Binaries are here if you want them (this was with the BOLT from LLVM 16).

If you want to reproduce the segfault on x86_64 linux you can download julia from the artifacts here https://buildkite.com/julialang/julia-master/builds/34942#018e6d3a-e24c-4bad-8406-6fbb670d9309. You can then BOLT the original binary or use the one I've already BOLTED here and replace libjulia-internal.so.1.12.0 in lib/julia with it. Then just run bin/julia from the root directory.

[aaupov]

Can you please check which function contains the illegal instruction, and dump the function throughout BOLT's optimizations with --print-only=<func> --print-all?

[Zentrik]

Doing bolt libjulia-internal.so.1.12.0.original -o libjulia-internal.so.1.12.0 --use-old-text -no-huge-pages --print-only=fl_file --print-all doesn't print anything extra, if I remove the print-only I see lots of assembly.

The segfault is

rag@pop-os:~/Documents/Code/julia/contrib/bolt$ bolt.build/julia

[506995] signal 11 (1): Segmentation fault
in expression starting at none:0
fl_file at /home/rag/Documents/Code/julia/contrib/bolt/bolt.build/usr/bin/../lib/libjulia-internal.so.1.12 (unknown line)
unknown function (ip: (nil))
Allocations: 1 (Pool: 1; Big: 0); GC: 0

[506995] signal 11 (1): Segmentation fault
in expression starting at none:0
fl_file at /home/rag/Documents/Code/julia/contrib/bolt/bolt.build/usr/bin/../lib/libjulia-internal.so.1.12 (unknown line)
unknown function (ip: (nil))
Allocations: 1 (Pool: 1; Big: 0); GC: 0
Segmentation fault (core dumped)

greping the optimized binary I get

rag@pop-os:~/Documents/Code/julia/contrib/bolt$ objdump -d bolt.build/usr/lib/libjulia-internal.so | grep fl_file
000000000011aec0 <fl_file>:
  11aed7:       0f 84 cb 01 00 00       je     11b0a8 <fl_file+0x1e8>
  11aee3:       0f 8e 3f 01 00 00       jle    11b028 <fl_file+0x168>
  11af10:       eb 40                   jmp    11af52 <fl_file+0x92>
  11af19:       74 47                   je     11af62 <fl_file+0xa2>
  11af22:       74 4c                   je     11af70 <fl_file+0xb0>
  11af2b:       0f 84 e6 00 00 00       je     11b017 <fl_file+0x157>
  11af50:       74 2b                   je     11af7d <fl_file+0xbd>
  11af58:       75 b8                   jne    11af12 <fl_file+0x52>
  11af60:       eb da                   jmp    11af3c <fl_file+0x7c>
  11af6e:       eb cc                   jmp    11af3c <fl_file+0x7c>
  11af7b:       eb bf                   jmp    11af3c <fl_file+0x7c>
  11af84:       0f 84 9e 00 00 00       je     11b028 <fl_file+0x168>
  11aff5:       0f 84 92 00 00 00       je     11b08d <fl_file+0x1cd>
  11affe:       74 05                   je     11b005 <fl_file+0x145>
  11b023:       e9 14 ff ff ff          jmp    11af3c <fl_file+0x7c>
  11b087:       0f 85 78 ff ff ff       jne    11b005 <fl_file+0x145>

The non optimized binary gives

rag@pop-os:~/Documents/Code/julia/contrib/bolt$ objdump -d bolt.build/usr/lib/libjulia-internal.so.1.12.0.original | grep fl_file
00000000001277a0 <fl_file>:
  1277b7:       0f 84 03 02 00 00       je     1279c0 <fl_file+0x220>
  1277c3:       0f 8e 77 01 00 00       jle    127940 <fl_file+0x1a0>
  1277f0:       eb 54                   jmp    127846 <fl_file+0xa6>
  127807:       74 57                   je     127860 <fl_file+0xc0>
  127810:       74 5e                   je     127870 <fl_file+0xd0>
  127819:       0f 84 01 01 00 00       je     127920 <fl_file+0x180>
  127844:       74 3a                   je     127880 <fl_file+0xe0>
  12784c:       75 b2                   jne    127800 <fl_file+0x60>
  127854:       eb da                   jmp    127830 <fl_file+0x90>
  12786c:       eb c2                   jmp    127830 <fl_file+0x90>
  12787b:       eb b3                   jmp    127830 <fl_file+0x90>
  127887:       0f 84 b3 00 00 00       je     127940 <fl_file+0x1a0>
  1278f8:       0f 84 a7 00 00 00       je     1279a5 <fl_file+0x205>
  127901:       74 05                   je     127908 <fl_file+0x168>
  12792c:       e9 ff fe ff ff          jmp    127830 <fl_file+0x90>
  12799f:       0f 85 63 ff ff ff       jne    127908 <fl_file+0x168>

Let me know if you need any more info.

[Zentrik]

Here's the function from the original binary

00000000001277a0 <fl_file>:
  1277a0:       f3 0f 1e fa             endbr64 
  1277a4:       41 57                   push   %r15
  1277a6:       41 56                   push   %r14
  1277a8:       41 55                   push   %r13
  1277aa:       41 54                   push   %r12
  1277ac:       55                      push   %rbp
  1277ad:       53                      push   %rbx
  1277ae:       48 89 fd                mov    %rdi,%rbp
  1277b1:       48 83 ec 18             sub    $0x18,%rsp
  1277b5:       85 d2                   test   %edx,%edx
  1277b7:       0f 84 03 02 00 00       je     1279c0 <fl_file+0x220>
  1277bd:       48 89 f1                mov    %rsi,%rcx
  1277c0:       83 fa 01                cmp    $0x1,%edx
  1277c3:       0f 8e 77 01 00 00       jle    127940 <fl_file+0x1a0>
  1277c9:       83 ea 02                sub    $0x2,%edx
  1277cc:       4c 8b 87 28 07 00 00    mov    0x728(%rdi),%r8
  1277d3:       48 8d 46 08             lea    0x8(%rsi),%rax
  1277d7:       45 31 f6                xor    %r14d,%r14d
  1277da:       48 8d 7c d6 10          lea    0x10(%rsi,%rdx,8),%rdi
  1277df:       45 31 c9                xor    %r9d,%r9d
  1277e2:       31 db                   xor    %ebx,%ebx
  1277e4:       45 31 e4                xor    %r12d,%r12d
  1277e7:       45 31 ed                xor    %r13d,%r13d
  1277ea:       41 ba 01 00 00 00       mov    $0x1,%r10d
  1277f0:       eb 54                   jmp    127846 <fl_file+0xa6>
  1277f2:       66 66 2e 0f 1f 84 00    data16 cs nopw 0x0(%rax,%rax,1)
  1277f9:       00 00 00 00 
  1277fd:       0f 1f 00                nopl   (%rax)
  127800:       48 3b 95 30 07 00 00    cmp    0x730(%rbp),%rdx
  127807:       74 57                   je     127860 <fl_file+0xc0>
  127809:       48 3b 95 38 07 00 00    cmp    0x738(%rbp),%rdx
  127810:       74 5e                   je     127870 <fl_file+0xd0>
  127812:       48 3b 95 40 07 00 00    cmp    0x740(%rbp),%rdx
  127819:       0f 84 01 01 00 00       je     127920 <fl_file+0x180>
  12781f:       48 3b 95 20 07 00 00    cmp    0x720(%rbp),%rdx
  127826:       45 0f 44 ea             cmove  %r10d,%r13d
  12782a:       66 0f 1f 44 00 00       nopw   0x0(%rax,%rax,1)
  127830:       44 89 ea                mov    %r13d,%edx
  127833:       89 de                   mov    %ebx,%esi
  127835:       48 83 c0 08             add    $0x8,%rax
  127839:       44 09 e2                or     %r12d,%edx
  12783c:       44 09 ce                or     %r9d,%esi
  12783f:       09 f2                   or     %esi,%edx
  127841:       48 39 c7                cmp    %rax,%rdi
  127844:       74 3a                   je     127880 <fl_file+0xe0>
  127846:       48 8b 10                mov    (%rax),%rdx
  127849:       4c 39 c2                cmp    %r8,%rdx
  12784c:       75 b2                   jne    127800 <fl_file+0x60>
  12784e:       41 bc 01 00 00 00       mov    $0x1,%r12d
  127854:       eb da                   jmp    127830 <fl_file+0x90>
  127856:       66 2e 0f 1f 84 00 00    cs nopw 0x0(%rax,%rax,1)
  12785d:       00 00 00 
  127860:       41 be 01 00 00 00       mov    $0x1,%r14d
  127866:       41 bc 01 00 00 00       mov    $0x1,%r12d
  12786c:       eb c2                   jmp    127830 <fl_file+0x90>
  12786e:       66 90                   xchg   %ax,%ax
  127870:       bb 01 00 00 00          mov    $0x1,%ebx
  127875:       41 bc 01 00 00 00       mov    $0x1,%r12d
  12787b:       eb b3                   jmp    127830 <fl_file+0x90>
  12787d:       0f 1f 00                nopl   (%rax)
  127880:       44 09 f2                or     %r14d,%edx
  127883:       44 89 0c 24             mov    %r9d,(%rsp)
  127887:       0f 84 b3 00 00 00       je     127940 <fl_file+0x1a0>
  12788d:       48 8b b5 58 07 00 00    mov    0x758(%rbp),%rsi
  127894:       ba b8 00 00 00          mov    $0xb8,%edx
  127899:       48 89 ef                mov    %rbp,%rdi
  12789c:       48 89 4c 24 08          mov    %rcx,0x8(%rsp)
  1278a1:       e8 0a 76 fe ff          call   10eeb0 <cvalue>
  1278a6:       48 8b 4c 24 08          mov    0x8(%rsp),%rcx
  1278ab:       48 8d 15 4f 76 17 00    lea    0x17764f(%rip),%rdx        # 29ef01 <.LC145>
  1278b2:       48 89 ef                mov    %rbp,%rdi
  1278b5:       49 89 c7                mov    %rax,%r15
  1278b8:       48 8b 31                mov    (%rcx),%rsi
  1278bb:       e8 00 91 fe ff          call   1109c0 <tostring>
  1278c0:       44 8b 0c 24             mov    (%rsp),%r9d
  1278c4:       41 89 d8                mov    %ebx,%r8d
  1278c7:       44 89 e1                mov    %r12d,%ecx
  1278ca:       49 89 c2                mov    %rax,%r10
  1278cd:       4c 89 f8                mov    %r15,%rax
  1278d0:       44 89 ea                mov    %r13d,%edx
  1278d3:       48 83 e0 f8             and    $0xfffffffffffffff8,%rax
  1278d7:       4c 89 d6                mov    %r10,%rsi
  1278da:       4c 89 54 24 08          mov    %r10,0x8(%rsp)
  1278df:       48 8b 78 08             mov    0x8(%rax),%rdi
  1278e3:       48 89 3c 24             mov    %rdi,(%rsp)
  1278e7:       e8 04 80 00 00          call   12f8f0 <ios_file>
  1278ec:       48 8b 3c 24             mov    (%rsp),%rdi
  1278f0:       48 85 c0                test   %rax,%rax
  1278f3:       4c 8b 54 24 08          mov    0x8(%rsp),%r10
  1278f8:       0f 84 a7 00 00 00       je     1279a5 <fl_file+0x205>
  1278fe:       45 85 f6                test   %r14d,%r14d
  127901:       74 05                   je     127908 <fl_file+0x168>
  127903:       e8 38 76 00 00          call   12ef40 <ios_seek_end>
  127908:       48 83 c4 18             add    $0x18,%rsp
  12790c:       4c 89 f8                mov    %r15,%rax
  12790f:       5b                      pop    %rbx
  127910:       5d                      pop    %rbp
  127911:       41 5c                   pop    %r12
  127913:       41 5d                   pop    %r13
  127915:       41 5e                   pop    %r14
  127917:       41 5f                   pop    %r15
  127919:       c3                      ret    
  12791a:       66 0f 1f 44 00 00       nopw   0x0(%rax,%rax,1)
  127920:       41 b9 01 00 00 00       mov    $0x1,%r9d
  127926:       41 bc 01 00 00 00       mov    $0x1,%r12d
  12792c:       e9 ff fe ff ff          jmp    127830 <fl_file+0x90>
  127931:       66 66 2e 0f 1f 84 00    data16 cs nopw 0x0(%rax,%rax,1)
  127938:       00 00 00 00 
  12793c:       0f 1f 40 00             nopl   0x0(%rax)
  127940:       48 8b b5 58 07 00 00    mov    0x758(%rbp),%rsi
  127947:       ba b8 00 00 00          mov    $0xb8,%edx
  12794c:       48 89 ef                mov    %rbp,%rdi
  12794f:       48 89 0c 24             mov    %rcx,(%rsp)
  127953:       e8 58 75 fe ff          call   10eeb0 <cvalue>
  127958:       48 8b 0c 24             mov    (%rsp),%rcx
  12795c:       48 8d 15 9e 75 17 00    lea    0x17759e(%rip),%rdx        # 29ef01 <.LC145>
  127963:       48 89 ef                mov    %rbp,%rdi
  127966:       49 89 c7                mov    %rax,%r15
  127969:       48 8b 31                mov    (%rcx),%rsi
  12796c:       e8 4f 90 fe ff          call   1109c0 <tostring>
  127971:       45 31 c9                xor    %r9d,%r9d
  127974:       45 31 c0                xor    %r8d,%r8d
  127977:       31 c9                   xor    %ecx,%ecx
  127979:       49 89 c2                mov    %rax,%r10
  12797c:       4c 89 f8                mov    %r15,%rax
  12797f:       ba 01 00 00 00          mov    $0x1,%edx
  127984:       48 83 e0 f8             and    $0xfffffffffffffff8,%rax
  127988:       4c 89 d6                mov    %r10,%rsi
  12798b:       4c 89 14 24             mov    %r10,(%rsp)
  12798f:       48 8b 78 08             mov    0x8(%rax),%rdi
  127993:       e8 58 7f 00 00          call   12f8f0 <ios_file>
  127998:       4c 8b 14 24             mov    (%rsp),%r10
  12799c:       48 85 c0                test   %rax,%rax
  12799f:       0f 85 63 ff ff ff       jne    127908 <fl_file+0x168>
  1279a5:       48 8b b5 a0 07 01 00    mov    0x107a0(%rbp),%rsi
  1279ac:       4c 89 d1                mov    %r10,%rcx
  1279af:       48 8d 15 83 89 17 00    lea    0x178983(%rip),%rdx        # 2a0339 <.LC4>
  1279b6:       48 89 ef                mov    %rbp,%rdi
  1279b9:       31 c0                   xor    %eax,%eax
  1279bb:       e8 30 e2 fe ff          call   115bf0 <lerrorf>
  1279c0:       48 8b b7 b8 07 01 00    mov    0x107b8(%rdi),%rsi
  1279c7:       4c 8d 05 2d f9 10 00    lea    0x10f92d(%rip),%r8        # 2372fb <.LC0>
  1279ce:       48 8d 0d 2c 75 17 00    lea    0x17752c(%rip),%rcx        # 29ef01 <.LC145>
  1279d5:       48 8d 15 36 f9 10 00    lea    0x10f936(%rip),%rdx        # 237312 <.LC10>
  1279dc:       31 c0                   xor    %eax,%eax
  1279de:       e8 0d e2 fe ff          call   115bf0 <lerrorf>
  1279e3:       66 66 2e 0f 1f 84 00    data16 cs nopw 0x0(%rax,%rax,1)
  1279ea:       00 00 00 00 
  1279ee:       66 90                   xchg   %ax,%ax

Here's the optimized one

000000000011aec0 <fl_file>:
  11aec0:       f3 0f 1e fa             endbr64 
  11aec4:       41 57                   push   %r15
  11aec6:       41 56                   push   %r14
  11aec8:       41 55                   push   %r13
  11aeca:       41 54                   push   %r12
  11aecc:       55                      push   %rbp
  11aecd:       53                      push   %rbx
  11aece:       48 89 fd                mov    %rdi,%rbp
  11aed1:       48 83 ec 18             sub    $0x18,%rsp
  11aed5:       85 d2                   test   %edx,%edx
  11aed7:       0f 84 cb 01 00 00       je     11b0a8 <fl_file+0x1e8>
  11aedd:       48 89 f1                mov    %rsi,%rcx
  11aee0:       83 fa 01                cmp    $0x1,%edx
  11aee3:       0f 8e 3f 01 00 00       jle    11b028 <fl_file+0x168>
  11aee9:       83 ea 02                sub    $0x2,%edx
  11aeec:       4c 8b 87 28 07 00 00    mov    0x728(%rdi),%r8
  11aef3:       48 8d 46 08             lea    0x8(%rsi),%rax
  11aef7:       45 31 f6                xor    %r14d,%r14d
  11aefa:       48 8d 7c d6 10          lea    0x10(%rsi,%rdx,8),%rdi
  11aeff:       45 31 c9                xor    %r9d,%r9d
  11af02:       31 db                   xor    %ebx,%ebx
  11af04:       45 31 e4                xor    %r12d,%r12d
  11af07:       45 31 ed                xor    %r13d,%r13d
  11af0a:       41 ba 01 00 00 00       mov    $0x1,%r10d
  11af10:       eb 40                   jmp    11af52 <fl_file+0x92>
  11af12:       48 3b 95 30 07 00 00    cmp    0x730(%rbp),%rdx
  11af19:       74 47                   je     11af62 <fl_file+0xa2>
  11af1b:       48 3b 95 38 07 00 00    cmp    0x738(%rbp),%rdx
  11af22:       74 4c                   je     11af70 <fl_file+0xb0>
  11af24:       48 3b 95 40 07 00 00    cmp    0x740(%rbp),%rdx
  11af2b:       0f 84 e6 00 00 00       je     11b017 <fl_file+0x157>
  11af31:       48 3b 95 20 07 00 00    cmp    0x720(%rbp),%rdx
  11af38:       45 0f 44 ea             cmove  %r10d,%r13d
  11af3c:       44 89 ea                mov    %r13d,%edx
  11af3f:       89 de                   mov    %ebx,%esi
  11af41:       48 83 c0 08             add    $0x8,%rax
  11af45:       44 09 e2                or     %r12d,%edx
  11af48:       44 09 ce                or     %r9d,%esi
  11af4b:       09 f2                   or     %esi,%edx
  11af4d:       48 39 c7                cmp    %rax,%rdi
  11af50:       74 2b                   je     11af7d <fl_file+0xbd>
  11af52:       48 8b 10                mov    (%rax),%rdx
  11af55:       4c 39 c2                cmp    %r8,%rdx
  11af58:       75 b8                   jne    11af12 <fl_file+0x52>
  11af5a:       41 bc 01 00 00 00       mov    $0x1,%r12d
  11af60:       eb da                   jmp    11af3c <fl_file+0x7c>
  11af62:       41 be 01 00 00 00       mov    $0x1,%r14d
  11af68:       41 bc 01 00 00 00       mov    $0x1,%r12d
  11af6e:       eb cc                   jmp    11af3c <fl_file+0x7c>
  11af70:       bb 01 00 00 00          mov    $0x1,%ebx
  11af75:       41 bc 01 00 00 00       mov    $0x1,%r12d
  11af7b:       eb bf                   jmp    11af3c <fl_file+0x7c>
  11af7d:       44 09 f2                or     %r14d,%edx
  11af80:       44 89 0c 24             mov    %r9d,(%rsp)
  11af84:       0f 84 9e 00 00 00       je     11b028 <fl_file+0x168>
  11af8a:       48 8b b5 58 07 00 00    mov    0x758(%rbp),%rsi
  11af91:       ba b8 00 00 00          mov    $0xb8,%edx
  11af96:       48 89 ef                mov    %rbp,%rdi
  11af99:       48 89 4c 24 08          mov    %rcx,0x8(%rsp)
  11af9e:       e8 e7 91 fe ff          call   10418a <cvalue>
  11afa3:       48 8b 4c 24 08          mov    0x8(%rsp),%rcx
  11afa8:       48 8d 15 52 3f 18 00    lea    0x183f52(%rip),%rdx        # 29ef01 <.LC145>
  11afaf:       48 89 ef                mov    %rbp,%rdi
  11afb2:       49 89 c7                mov    %rax,%r15
  11afb5:       48 8b 31                mov    (%rcx),%rsi
  11afb8:       e8 c3 aa fe ff          call   105a80 <tostring>
  11afbd:       44 8b 0c 24             mov    (%rsp),%r9d
  11afc1:       41 89 d8                mov    %ebx,%r8d
  11afc4:       44 89 e1                mov    %r12d,%ecx
  11afc7:       49 89 c2                mov    %rax,%r10
  11afca:       4c 89 f8                mov    %r15,%rax
  11afcd:       44 89 ea                mov    %r13d,%edx
  11afd0:       48 83 e0 f8             and    $0xfffffffffffffff8,%rax
  11afd4:       4c 89 d6                mov    %r10,%rsi
  11afd7:       4c 89 54 24 08          mov    %r10,0x8(%rsp)
  11afdc:       48 8b 78 08             mov    0x8(%rax),%rdi
  11afe0:       48 89 3c 24             mov    %rdi,(%rsp)
  11afe4:       e8 6b 73 00 00          call   122354 <ios_file>
  11afe9:       48 8b 3c 24             mov    (%rsp),%rdi
  11afed:       48 85 c0                test   %rax,%rax
  11aff0:       4c 8b 54 24 08          mov    0x8(%rsp),%r10
  11aff5:       0f 84 92 00 00 00       je     11b08d <fl_file+0x1cd>
  11affb:       45 85 f6                test   %r14d,%r14d
  11affe:       74 05                   je     11b005 <fl_file+0x145>
  11b000:       e8 7b 6a 00 00          call   121a80 <ios_seek_end>
  11b005:       48 83 c4 18             add    $0x18,%rsp
  11b009:       4c 89 f8                mov    %r15,%rax
  11b00c:       5b                      pop    %rbx
  11b00d:       5d                      pop    %rbp
  11b00e:       41 5c                   pop    %r12
  11b010:       41 5d                   pop    %r13
  11b012:       41 5e                   pop    %r14
  11b014:       41 5f                   pop    %r15
  11b016:       c3                      ret    
  11b017:       41 b9 01 00 00 00       mov    $0x1,%r9d
  11b01d:       41 bc 01 00 00 00       mov    $0x1,%r12d
  11b023:       e9 14 ff ff ff          jmp    11af3c <fl_file+0x7c>
  11b028:       48 8b b5 58 07 00 00    mov    0x758(%rbp),%rsi
  11b02f:       ba b8 00 00 00          mov    $0xb8,%edx
  11b034:       48 89 ef                mov    %rbp,%rdi
  11b037:       48 89 0c 24             mov    %rcx,(%rsp)
  11b03b:       e8 4a 91 fe ff          call   10418a <cvalue>
  11b040:       48 8b 0c 24             mov    (%rsp),%rcx
  11b044:       48 8d 15 b6 3e 18 00    lea    0x183eb6(%rip),%rdx        # 29ef01 <.LC145>
  11b04b:       48 89 ef                mov    %rbp,%rdi
  11b04e:       49 89 c7                mov    %rax,%r15
  11b051:       48 8b 31                mov    (%rcx),%rsi
  11b054:       e8 27 aa fe ff          call   105a80 <tostring>
  11b059:       45 31 c9                xor    %r9d,%r9d
  11b05c:       45 31 c0                xor    %r8d,%r8d
  11b05f:       31 c9                   xor    %ecx,%ecx
  11b061:       49 89 c2                mov    %rax,%r10
  11b064:       4c 89 f8                mov    %r15,%rax
  11b067:       ba 01 00 00 00          mov    $0x1,%edx
  11b06c:       48 83 e0 f8             and    $0xfffffffffffffff8,%rax
  11b070:       4c 89 d6                mov    %r10,%rsi
  11b073:       4c 89 14 24             mov    %r10,(%rsp)
  11b077:       48 8b 78 08             mov    0x8(%rax),%rdi
  11b07b:       e8 d4 72 00 00          call   122354 <ios_file>
  11b080:       4c 8b 14 24             mov    (%rsp),%r10
  11b084:       48 85 c0                test   %rax,%rax
  11b087:       0f 85 78 ff ff ff       jne    11b005 <fl_file+0x145>
  11b08d:       48 8b b5 a0 07 01 00    mov    0x107a0(%rbp),%rsi
  11b094:       4c 89 d1                mov    %r10,%rcx
  11b097:       48 8d 15 9b 52 18 00    lea    0x18529b(%rip),%rdx        # 2a0339 <.LC4>
  11b09e:       48 89 ef                mov    %rbp,%rdi
  11b0a1:       31 c0                   xor    %eax,%eax
  11b0a3:       e8 2c f4 fe ff          call   10a4d4 <lerrorf>
  11b0a8:       48 8b b7 b8 07 01 00    mov    0x107b8(%rdi),%rsi
  11b0af:       4c 8d 05 45 c2 11 00    lea    0x11c245(%rip),%r8        # 2372fb <.LC0>
  11b0b6:       48 8d 0d 44 3e 18 00    lea    0x183e44(%rip),%rcx        # 29ef01 <.LC145>
  11b0bd:       48 8d 15 4e c2 11 00    lea    0x11c24e(%rip),%rdx        # 237312 <.LC10>
  11b0c4:       31 c0                   xor    %eax,%eax
  11b0c6:       e8 09 f4 fe ff          call   10a4d4 <lerrorf>
  11b0cb:       90                      nop

Here's the optimized function without --use-old-text:

rag@pop-os:~/Documents/Code/julia/contrib/bolt$ objdump -d bolt.build/usr/lib/libjulia-internal.so.1.12.0 | grep "0 <fl_file>" -A 136
0000000000707ec0 <fl_file>:
  707ec0:       f3 0f 1e fa             endbr64 
  707ec4:       41 57                   push   %r15
  707ec6:       41 56                   push   %r14
  707ec8:       41 55                   push   %r13
  707eca:       41 54                   push   %r12
  707ecc:       55                      push   %rbp
  707ecd:       53                      push   %rbx
  707ece:       48 89 fd                mov    %rdi,%rbp
  707ed1:       48 83 ec 18             sub    $0x18,%rsp
  707ed5:       85 d2                   test   %edx,%edx
  707ed7:       0f 84 cb 01 00 00       je     7080a8 <fl_file+0x1e8>
  707edd:       48 89 f1                mov    %rsi,%rcx
  707ee0:       83 fa 01                cmp    $0x1,%edx
  707ee3:       0f 8e 3f 01 00 00       jle    708028 <fl_file+0x168>
  707ee9:       83 ea 02                sub    $0x2,%edx
  707eec:       4c 8b 87 28 07 00 00    mov    0x728(%rdi),%r8
  707ef3:       48 8d 46 08             lea    0x8(%rsi),%rax
  707ef7:       45 31 f6                xor    %r14d,%r14d
  707efa:       48 8d 7c d6 10          lea    0x10(%rsi,%rdx,8),%rdi
  707eff:       45 31 c9                xor    %r9d,%r9d
  707f02:       31 db                   xor    %ebx,%ebx
  707f04:       45 31 e4                xor    %r12d,%r12d
  707f07:       45 31 ed                xor    %r13d,%r13d
  707f0a:       41 ba 01 00 00 00       mov    $0x1,%r10d
  707f10:       eb 40                   jmp    707f52 <fl_file+0x92>
  707f12:       48 3b 95 30 07 00 00    cmp    0x730(%rbp),%rdx
  707f19:       74 47                   je     707f62 <fl_file+0xa2>
  707f1b:       48 3b 95 38 07 00 00    cmp    0x738(%rbp),%rdx
  707f22:       74 4c                   je     707f70 <fl_file+0xb0>
  707f24:       48 3b 95 40 07 00 00    cmp    0x740(%rbp),%rdx
  707f2b:       0f 84 e6 00 00 00       je     708017 <fl_file+0x157>
  707f31:       48 3b 95 20 07 00 00    cmp    0x720(%rbp),%rdx
  707f38:       45 0f 44 ea             cmove  %r10d,%r13d
  707f3c:       44 89 ea                mov    %r13d,%edx
  707f3f:       89 de                   mov    %ebx,%esi
  707f41:       48 83 c0 08             add    $0x8,%rax
  707f45:       44 09 e2                or     %r12d,%edx
  707f48:       44 09 ce                or     %r9d,%esi
  707f4b:       09 f2                   or     %esi,%edx
  707f4d:       48 39 c7                cmp    %rax,%rdi
  707f50:       74 2b                   je     707f7d <fl_file+0xbd>
  707f52:       48 8b 10                mov    (%rax),%rdx
  707f55:       4c 39 c2                cmp    %r8,%rdx
  707f58:       75 b8                   jne    707f12 <fl_file+0x52>
  707f5a:       41 bc 01 00 00 00       mov    $0x1,%r12d
  707f60:       eb da                   jmp    707f3c <fl_file+0x7c>
  707f62:       41 be 01 00 00 00       mov    $0x1,%r14d
  707f68:       41 bc 01 00 00 00       mov    $0x1,%r12d
  707f6e:       eb cc                   jmp    707f3c <fl_file+0x7c>
  707f70:       bb 01 00 00 00          mov    $0x1,%ebx
  707f75:       41 bc 01 00 00 00       mov    $0x1,%r12d
  707f7b:       eb bf                   jmp    707f3c <fl_file+0x7c>
  707f7d:       44 09 f2                or     %r14d,%edx
  707f80:       44 89 0c 24             mov    %r9d,(%rsp)
  707f84:       0f 84 9e 00 00 00       je     708028 <fl_file+0x168>
  707f8a:       48 8b b5 58 07 00 00    mov    0x758(%rbp),%rsi
  707f91:       ba b8 00 00 00          mov    $0xb8,%edx
  707f96:       48 89 ef                mov    %rbp,%rdi
  707f99:       48 89 4c 24 08          mov    %rcx,0x8(%rsp)
  707f9e:       e8 e7 91 fe ff          call   6f118a <cvalue>
  707fa3:       48 8b 4c 24 08          mov    0x8(%rsp),%rcx
  707fa8:       48 8d 15 52 6f b9 ff    lea    -0x4690ae(%rip),%rdx        # 29ef01 <.LC145>
  707faf:       48 89 ef                mov    %rbp,%rdi
  707fb2:       49 89 c7                mov    %rax,%r15
  707fb5:       48 8b 31                mov    (%rcx),%rsi
  707fb8:       e8 c3 aa fe ff          call   6f2a80 <tostring>
  707fbd:       44 8b 0c 24             mov    (%rsp),%r9d
  707fc1:       41 89 d8                mov    %ebx,%r8d
  707fc4:       44 89 e1                mov    %r12d,%ecx
  707fc7:       49 89 c2                mov    %rax,%r10
  707fca:       4c 89 f8                mov    %r15,%rax
  707fcd:       44 89 ea                mov    %r13d,%edx
  707fd0:       48 83 e0 f8             and    $0xfffffffffffffff8,%rax
  707fd4:       4c 89 d6                mov    %r10,%rsi
  707fd7:       4c 89 54 24 08          mov    %r10,0x8(%rsp)
  707fdc:       48 8b 78 08             mov    0x8(%rax),%rdi
  707fe0:       48 89 3c 24             mov    %rdi,(%rsp)
  707fe4:       e8 6b 73 00 00          call   70f354 <ios_file>
  707fe9:       48 8b 3c 24             mov    (%rsp),%rdi
  707fed:       48 85 c0                test   %rax,%rax
  707ff0:       4c 8b 54 24 08          mov    0x8(%rsp),%r10
  707ff5:       0f 84 92 00 00 00       je     70808d <fl_file+0x1cd>
  707ffb:       45 85 f6                test   %r14d,%r14d
  707ffe:       74 05                   je     708005 <fl_file+0x145>
  708000:       e8 7b 6a 00 00          call   70ea80 <ios_seek_end>
  708005:       48 83 c4 18             add    $0x18,%rsp
  708009:       4c 89 f8                mov    %r15,%rax
  70800c:       5b                      pop    %rbx
  70800d:       5d                      pop    %rbp
  70800e:       41 5c                   pop    %r12
  708010:       41 5d                   pop    %r13
  708012:       41 5e                   pop    %r14
  708014:       41 5f                   pop    %r15
  708016:       c3                      ret    
  708017:       41 b9 01 00 00 00       mov    $0x1,%r9d
  70801d:       41 bc 01 00 00 00       mov    $0x1,%r12d
  708023:       e9 14 ff ff ff          jmp    707f3c <fl_file+0x7c>
  708028:       48 8b b5 58 07 00 00    mov    0x758(%rbp),%rsi
  70802f:       ba b8 00 00 00          mov    $0xb8,%edx
  708034:       48 89 ef                mov    %rbp,%rdi
  708037:       48 89 0c 24             mov    %rcx,(%rsp)
  70803b:       e8 4a 91 fe ff          call   6f118a <cvalue>
  708040:       48 8b 0c 24             mov    (%rsp),%rcx
  708044:       48 8d 15 b6 6e b9 ff    lea    -0x46914a(%rip),%rdx        # 29ef01 <.LC145>
  70804b:       48 89 ef                mov    %rbp,%rdi
  70804e:       49 89 c7                mov    %rax,%r15
  708051:       48 8b 31                mov    (%rcx),%rsi
  708054:       e8 27 aa fe ff          call   6f2a80 <tostring>
  708059:       45 31 c9                xor    %r9d,%r9d
  70805c:       45 31 c0                xor    %r8d,%r8d
  70805f:       31 c9                   xor    %ecx,%ecx
  708061:       49 89 c2                mov    %rax,%r10
  708064:       4c 89 f8                mov    %r15,%rax
  708067:       ba 01 00 00 00          mov    $0x1,%edx
  70806c:       48 83 e0 f8             and    $0xfffffffffffffff8,%rax
  708070:       4c 89 d6                mov    %r10,%rsi
  708073:       4c 89 14 24             mov    %r10,(%rsp)
  708077:       48 8b 78 08             mov    0x8(%rax),%rdi
  70807b:       e8 d4 72 00 00          call   70f354 <ios_file>
  708080:       4c 8b 14 24             mov    (%rsp),%r10
  708084:       48 85 c0                test   %rax,%rax
  708087:       0f 85 78 ff ff ff       jne    708005 <fl_file+0x145>
  70808d:       48 8b b5 a0 07 01 00    mov    0x107a0(%rbp),%rsi
  708094:       4c 89 d1                mov    %r10,%rcx
  708097:       48 8d 15 9b 82 b9 ff    lea    -0x467d65(%rip),%rdx        # 2a0339 <.LC4>
  70809e:       48 89 ef                mov    %rbp,%rdi
  7080a1:       31 c0                   xor    %eax,%eax
  7080a3:       e8 2c f4 fe ff          call   6f74d4 <lerrorf>
  7080a8:       48 8b b7 b8 07 01 00    mov    0x107b8(%rdi),%rsi
  7080af:       4c 8d 05 45 f2 b2 ff    lea    -0x4d0dbb(%rip),%r8        # 2372fb <.LC0>
  7080b6:       48 8d 0d 44 6e b9 ff    lea    -0x4691bc(%rip),%rcx        # 29ef01 <.LC145>
  7080bd:       48 8d 15 4e f2 b2 ff    lea    -0x4d0db2(%rip),%rdx        # 237312 <.LC10>
  7080c4:       31 c0                   xor    %eax,%eax
  7080c6:       e8 09 f4 fe ff          call   6f74d4 <lerrorf>
  7080cb:       90                      nop

[Zentrik]

Ok I had to use --print-only=fl_file/1
Here's the output: https://gist.github.com/Zentrik/d901bfc144af6348942f00a9558b8bd8

[Zentrik]

Btw the segfault occurs at 11af52: 48 8b 10 mov (%rax),%rdx.

[Zentrik]

As far as I can tell in the apply_cl function we have jmp *(%r14, %rax, 8). Without --use-old-text this doesn't jump to fl_file but it does with --use-old-text. %rax is 66 in both binaries.

[Zentrik]

This replicates on 0dcabba. It seems in apply_cl we use computed go to's which is what that jump is.

[Zentrik]

Ok so the issue seems to be that BOLT doesn't update the values in https://github.com/JuliaLang/julia/blob/7f8635f11cae5f3f592afcc7b55c8e0e23589c3d/src/flisp/opcodes.h#L39 so we end up jumping to the wrong place. Similarly it doesn't look like vm_apply_labels gets updated.

[aaupov]

Thank you for finding the root cause. Please skip the function containing computed goto jump table with -skip-funcs=<func>. We don’t fully support them, and there’s a similar issue with CPython (I had a WIP diff to support it but it’s low priority for me at the moment).

[Zentrik]

Thanks. If I don't use the old text flag is it fine to not skip it or might it still cause problems?

[Zentrik]

Just to note, neither -skip-funcs=apply_cl or -skip-funcs=apply_cl* seemed to fix it

[maksfb]

Even with --skip-funcs, listed functions are still rewritten with --use-old-text. We need a proper support for computed goto's with PIC. Meanwhile, you have to drop --use-old-text.