Skip to content

Commit 9b29547

Browse files
SLTozerSLTozer
committed
Update migration doc to explain how to handle textual IR updates
1 parent 8afa6cf commit 9b29547

File tree

1 file changed

+104
-1
lines changed

1 file changed

+104
-1
lines changed

‎llvm/docs/RemoveDIsDebugInfo.md

Lines changed: 104 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,12 +24,115 @@ The debug records are not instructions, do not appear in the instruction list, a
2424

2525
# Great, what do I need to do!
2626

27-
Approximately nothing -- we've already instrumented all of LLVM to handle these new records ("`DbgRecords`") and behave identically to past LLVM behaviour. We plan on turning this on by default some time soon, with IR converted to the intrinsic form of debug info at terminals (textual IR, bitcode) for a short while, before then changing the textual IR and bitcode formats.
27+
Very little -- we've already instrumented all of LLVM to handle these new records ("`DbgRecords`") and behave identically to past LLVM behaviour. This is currently being turned on by default, so that `DbgRecords` will be used by default in memory, IR, and bitcode.
28+
29+
## API Changes
2830

2931
There are two significant changes to be aware of. Firstly, we're adding a single bit of debug relevant data to the `BasicBlock::iterator` class (it's so that we can determine whether ranges intend on including debug info at the beginning of a block or not). That means when writing passes that insert LLVM IR instructions, you need to identify positions with `BasicBlock::iterator` rather than just a bare `Instruction *`. Most of the time this means that after identifying where you intend on inserting something, you must also call `getIterator` on the instruction position -- however when inserting at the start of a block you _must_ use `getFirstInsertionPt`, `getFirstNonPHIIt` or `begin` and use that iterator to insert, rather than just fetching a pointer to the first instruction.
3032

3133
The second matter is that if you transfer sequences of instructions from one place to another manually, i.e. repeatedly using `moveBefore` where you might have used `splice`, then you should instead use the method `moveBeforePreserving`. `moveBeforePreserving` will transfer debug info records with the instruction they're attached to. This is something that happens automatically today -- if you use `moveBefore` on every element of an instruction sequence, then debug intrinsics will be moved in the normal course of your code, but we lose this behaviour with non-instruction debug info.
3234

35+
## Textual IR Changes
36+
37+
As we change from using debug intrinsics to debug records, any tools that depend on parsing IR produced by LLVM will need to handle the new format. For the most part, the difference between the printed form of a intrinsic and a record is trivial:
38+
39+
1. An extra 2 spaces of indentation are added.
40+
2. The call `(tail|notail|musttail)? call void @llvm.dbg._` is replaced with `#dbg_`.
41+
3. The leading `metadata ` is removed from each argument to the intrinsic.
42+
4. The DILocation changes from being a debug attachment, `!dbg !<Num>`, to being the final intrinsic argument without a leading `!dbg`.
43+
44+
Altogether, this gives you:
45+
46+
```
47+
; Intrinsic:
48+
call void @llvm.dbg.value(metadata i32 %add, metadata !10, metadata !DIExpression()), !dbg !20
49+
; Record:
50+
#dbg_value(i32 %add, !10, !DIExpression(), !20)
51+
```
52+
53+
### Test updates
54+
55+
Any tests downstream of the main LLVM repo that test the IR output of LLVM may break as a result of the change to using records. Updating an individual test to expect records instead of intrinsics should be trivial, given the update rules above. Updating many tests may be burdensome however; to update the lit tests in the main repository, the following steps were used:
56+
57+
1. Collect the list of failing lit tests into a single file, `failing-tests.txt`, separated by (and ending with) newlines.
58+
2. Use the following line to split the failing tests into tests that use update_test_checks and tests that don't:
59+
```
60+
$ while IFS= read -r f; do grep -q "Assertions have been autogenerated by" "$f" && echo "$f" >> update-checks-tests.txt || echo "$f" >> manual-tests.txt; done < failing-tests.txt
61+
```
62+
3. For the tests that use update_test_checks, run the appropriate update_test_checks script - for the main LLVM repo, this was achieved with:
63+
```
64+
$ xargs ./llvm/utils/update_test_checks.py --opt-binary ./build/bin/opt < update-checks-tests.txt
65+
$ xargs ./llvm/utils/update_cc_test_checks.py --llvm-bin ./build/bin/ < update-checks-tests.txt
66+
```
67+
4. The remaining tests can be manually updated, although if there is a large number of tests then the following scripts may be useful; firstly, a script used to extract the check-line prefixes from a file:
68+
```
69+
$ cat ./get-checks.sh
70+
#!/bin/bash
71+
72+
# Always add CHECK, since it's more effort than it's worth to filter files where
73+
# every RUN line uses other check prefixes.
74+
# Then detect every instance of "check-prefix(es)=..." and add the
75+
# comma-separated arguments as extra checks.
76+
for filename in "$@"
77+
do
78+
echo "$filename,CHECK"
79+
allchecks=$(grep -Eo 'check-prefix(es)?[ =][A-Z0-9_,-]+' $filename | sed -E 's/.+[= ]([A-Z0-9_,-]+).*/\1/g; s/,/\n/g')
80+
for check in $allchecks; do
81+
echo "$filename,$check"
82+
done
83+
done
84+
```
85+
Then a second script to perform the work of actually updating the check-lines in each of the failing tests, with a series of simple substitution patterns:
86+
```
87+
$ cat ./substitute-checks.sh
88+
#!/bin/bash
89+
90+
file="$1"
91+
check="$2"
92+
93+
# Any test that explicitly tests debug intrinsic output is not suitable to
94+
# update by this script.
95+
if grep -q "write-experimental-debuginfo=false" "$file"; then
96+
exit 0
97+
fi
98+
99+
sed -i -E -e "
100+
/(#|;|\/\/).*$check[A-Z0-9_\-]*:/!b
101+
/DIGlobalVariableExpression/b
102+
/!llvm.dbg./bpostcall
103+
s/((((((no|must)?tail )?call.*)?void )?@)?llvm.)?dbg\.([a-z]+)/#dbg_\7/
104+
:postcall
105+
/declare #dbg_/d
106+
s/metadata //g
107+
s/metadata\{/{/g
108+
s/DIExpression\(([^)]*)\)\)(,( !dbg)?)?/DIExpression(\1),/
109+
/#dbg_/!b
110+
s/((\))?(,) )?!dbg (![0-9]+)/\3\4\2/
111+
s/((\))?(, ))?!dbg/\3/
112+
" "$file"
113+
```
114+
Both of these scripts combined can be used on the list in `manual-tests.txt` as follows:
115+
```
116+
$ cat manual-tests.txt | xargs ./get-checks.sh | sort | uniq | awk -F ',' '{ system("./substitute-checks.sh " $1 " " $2) }'
117+
```
118+
These scripts dealt successfully with the vast majority of checks in `clang/test` and `llvm/test`.
119+
5. Verify the resulting tests pass, and detect any failing tests:
120+
```
121+
$ xargs ./build/bin/llvm-lit -q < failing-tests.txt
122+
********************
123+
Failed Tests (5):
124+
LLVM :: DebugInfo/Generic/dbg-value-lower-linenos.ll
125+
LLVM :: Transforms/HotColdSplit/transfer-debug-info.ll
126+
LLVM :: Transforms/ObjCARC/basic.ll
127+
LLVM :: Transforms/ObjCARC/ensure-that-exception-unwind-path-is-visited.ll
128+
LLVM :: Transforms/SafeStack/X86/debug-loc2.ll
129+
130+
131+
Total Discovered Tests: 295
132+
Failed: 5 (1.69%)
133+
```
134+
6. Some tests may have failed - the update scripts are simplistic and preserve no context across lines, and so there are cases that they will not handle; the remaining cases must be manually updated (or handled by further scripts).
135+
33136
# C-API changes
34137
35138
All the functions that have been added are temporary and will be deprecated in the future. The intention is that they'll help downstream projects adapt during the transition period.

0 commit comments

Comments
 (0)