Stackmaps only really make sense if there is only one compilation unit

[vext01]

Bugzilla Link	38277
Version	trunk
OS	OpenBSD
CC	@hfinkel,@jh7370,@preames

Extended Description

Hi,

I've noticed that LLVM tooling surrounding stackmaps assumes only one compilation unit.

When using the >1 CU, one set of stackmap information is generated for each compilation unit. At the link stage, these multiple stackmap tables get concatenated by the linker into a compound .llvm_stackmaps section in the resulting binary.

Minimal example:

Suppose we have a simple C program split between two C files:

$ cat main.c
#include <stdio.h>
#include <stdlib.h>

void do_print(void);

int
main(int argc, char **argv)
{
    do_print();
    return (EXIT_SUCCESS);
}


$ cat do_print.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

void
do_print(int argc, char **argv)
{
    printf("pid is %d\n", getpid());
}

Let's get these files as ll code and add stackmap calls:

$ clang -c -emit-llvm main.c
$ clang -c -emit-llvm do_print.c

$ llvm-dis main.bc
$ llvm-dis do_print.bc

$ cp main.ll main.ll.orig
$ cp do_print.ll do_print.ll.orig

$ vim main.ll # add stackmap calls
$ vim do_print.ll # add more

Suppose we edit like so:

--- main.ll.orig        2018-07-23 14:50:49.414647729 +0100
+++ main.ll     2018-07-23 14:53:22.082798103 +0100
@@ -3,15 +3,21 @@
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"

+declare void @​llvm.experimental.stackmap(i64, i32, ...)
+
 ; Function Attrs: noinline nounwind optnone uwtable
 define dso_local i32 @​main(i32, i8**) #​0 {
+  call void (i64, i32, ...) @​llvm.experimental.stackmap(i64 0, i32 8)
   %3 = alloca i32, align 4
   %4 = alloca i32, align 4
   %5 = alloca i8**, align 8
+  call void (i64, i32, ...) @​llvm.experimental.stackmap(i64 1, i32 8)
   store i32 0, i32* %3, align 4
   store i32 %0, i32* %4, align 4
   store i8** %1, i8*** %5, align 8
+  call void (i64, i32, ...) @​llvm.experimental.stackmap(i64 2, i32 8)
   call void @​do_print()
+  call void (i64, i32, ...) @​llvm.experimental.stackmap(i64 3, i32 8)
   ret i32 0
 }


$ diff -u do_print.ll.orig  do_print.ll
--- do_print.ll.orig    2018-07-23 14:50:58.754656962 +0100
+++ do_print.ll 2018-07-23 14:54:36.070870620 +0100
@@ -3,12 +3,16 @@
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"

+declare void @​llvm.experimental.stackmap(i64, i32, ...)
+
 @.str = private unnamed_addr constant [11 x i8] c"pid is %d\0A\00", align 1

 ; Function Attrs: noinline nounwind optnone uwtable
 define dso_local void @​do_print(i32, i8**) #​0 {
+  call void (i64, i32, ...) @​llvm.experimental.stackmap(i64 100, i32 8)
   %3 = alloca i32, align 4
   %4 = alloca i8**, align 8
+  call void (i64, i32, ...) @​llvm.experimental.stackmap(i64 101, i32 8)
   store i32 %0, i32* %3, align 4
   store i8** %1, i8*** %4, align 8
   %5 = call i32 @​getpid() #​3

So main.ll has stackmaps with id 0-3 and do_print.ll has 100 and 101.

Let's now make objects and verify they have stackmaps:

$ clang -c main.ll
$ clang -c do_print.ll

$ llvm-readelf -stackmap main.o                                                                                                                                                                   
LLVM StackMap Version: 2
Num Functions: 1
  Function address: 0, stack size: 56, callsite record count: 4
Num Constants: 0
Num Records: 4
  Record ID: 0, instruction offset: 15
    0 locations:
    0 live-outs: [ ]
  Record ID: 1, instruction offset: 23
    0 locations:
    0 live-outs: [ ]
  Record ID: 2, instruction offset: 44
    0 locations:
    0 live-outs: [ ]
  Record ID: 3, instruction offset: 57
    0 locations:
    0 live-outs: [ ]
$ llvm-readelf -stackmap do_print.o
LLVM StackMap Version: 2
Num Functions: 1
  Function address: 0, stack size: 56, callsite record count: 2
Num Constants: 0
Num Records: 2
  Record ID: 100, instruction offset: 15
    0 locations:
    0 live-outs: [ ]
  Record ID: 101, instruction offset: 23
    0 locations:
    0 live-outs: [ ]

So far so good.

Now let's link a binary and look at the stackmaps:

$ clang main.o do_print.o
$ ./a.out # check binary works
pid is 16530

$ llvm-readelf -stackmap a.out
LLVM StackMap Version: 2
Num Functions: 1
  Function address: 4195664, stack size: 56, callsite record count: 4
Num Constants: 0
Num Records: 4
  Record ID: 0, instruction offset: 15
    0 locations:
    0 live-outs: [ ]
  Record ID: 1, instruction offset: 23
    0 locations:
    0 live-outs: [ ]
  Record ID: 2, instruction offset: 44
    0 locations:
    0 live-outs: [ ]
  Record ID: 3, instruction offset: 57
    0 locations:
    0 live-outs: [ ]

What happened to stackmaps 100 and 101 from do_print.ll?

Let's link the binary in a different order:

$ clang do_print.o main.o
$ llvm-readelf -stackmap a.out
LLVM StackMap Version: 2
Num Functions: 1
  Function address: 4195664, stack size: 56, callsite record count: 2
Num Constants: 0
Num Records: 2
  Record ID: 100, instruction offset: 15
    0 locations:
    0 live-outs: [ ]
  Record ID: 101, instruction offset: 23
    0 locations:
    0 live-outs: [ ]

What happened to stackmaps 0-3 from main.ll?

In each case, the linker had concatenated both sets of stackmap data together back to back, but llvm-readelf is assuming there is only one set of stackmap data.

$ readelf --sections do_print.o                                                                                                                                                                     
There are 12 section headers, starting at offset 0x338:

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
...
  [ 7] .llvm_stackmaps   PROGBITS         0000000000000000  000000d0
       0000000000000058  0000000000000000   A       0     0     8

So do_print.ll's stackmap info is 0x58 in length (and guaranteed aligned I think). If we go to this offset in .llvm_stackmaps in the binary where we linked in do_print.ll first:

$ r2 a.out
[0x00400450]> s section..llvm_stackmaps + 0x58
[0x004006d8]> px 16
- offset -   0 1  2 3  4 5  6 7  8 9  A B  C D  E F  0123456789ABCDEF
0x004006d8  0300 0000 0100 0000 0000 0000 0400 0000  ................

This is the start of main.ll's stackmap info.

• Byte 0 is the version number of the stackmap info: 3.
• Byte C is the number of records: 4 (100-103).

I've not yet found a robust way to know how many sets of stackmap info are in a binary.

[vext01]

Byte C is the number of records: 4 (100-103).

That was supposed to say:

Byte C is the number of records: 4 (0-3).

[vext01]

I should mention which version I'm using:

$ clang -v
clang version 7.0.0 (trunk 336996) (llvm/trunk 336905)
Target: x86_64-unknown-linux-gnu

[preames]

Taking a look at the code for this, it looks like the printing code is assuming there is only a single blob of stackmap data when there is in fact two? (If I read your description correctly?) If so, isn't this just a case of needing to check for leftover section contents and restart the parser? (i.e. didn't the linker do a completely reasonable thing here?)

If all of the above is correct, then the logical thing to do is to add an abstraction for a single object file worth of contents to the StackMapParserV2 and add support for iterating through a set of object file stackmaps (constructed with simple concatenation by the linker) when parsing. Right?

One thing worth highlighting. We are not guaranteed that all stackmaps within the linked file are all the same version. We can probably just assert this at the moment (well, early exit), but it's something to keep in mind.

[vext01]

Hi Philip,

That's right, there's more than one stackmap blob, but the reference parser assumes there is one.

And, yep, you'd need some way of iterating over multiple blobs.

One suggestion I've heard is: keep parsing until you are out of the stackmap section. I actually don't like that much myself, as it seems a little ad-hoc. I think it would be better to have a way to know how many blobs to expect ahead of time.