Unexported symbols in lldb-shell :: Unwind/windows-unaligned-x86_64.test

[kendalharland]

CC @labath @mstorsjo

Related to swiftlang#9141

Overview

My local build of this test produces a binary whose realign_stack and call_func symbols are not visible to lldb so the test fails to match to the expected output of thread backtrace as these symbol names appear as empty strings in the stack trace output.

To make this test pass, I had to create this local patch which is certainly not the correct way to fix things but demonstrates my problem well.

Background

In particular, here are the problems I observe:

1. lld-link does not produce a binary that exports the realign_stack and call_func symbols which are defined as .globl in windows-unaligned-x86_64-asm.s, even though lldb-pdbutil indicates that these symbols are “public” in the final executable (The commands below are run in powershell but I built and ran these tests in cmd.exe):

PS C:\workspace\llvm-project> C:\workspace\llvm-project\build\bin\llvm-pdbutil.exe pretty --externals C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.pdb | Select-String -Pattern realign_stack

  public [0x0000104e] realign_stack

PS C:\workspace\llvm-project> C:\workspace\llvm-project\build\bin\llvm-pdbutil.exe pretty --externals C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.pdb | Select-String -Pattern call_func

  public [0x00001040] call_func

LLDB's image lookup command fails to find these symbols by address or name.

As a workaround I’ve manually exported the symbols in build.py using e.g. /EXPORT:realign_stack

2. LLDB doesn’t appear to load PDB symbols from a separate .pdb file automatically. It’s unclear if the original test author ran into this problem from their PR discussion so as a workaround I added an explicit add-dsym ... call to load the symbols.

3. LLDB seems to have a hard time parsing PDB files and crashes when add-dsym is called. I believe this is a known issue [1], [2]. Given that there are 2 PDB readers to choose from: LLDB’s implementation (default) and the Windows Native reader, as a workaround I’ve used the native one by setting LLDB_USE_NATIVE_PDB_READER=1.

Without my local patch, the full failure output with verbose information is below. As you can see, func and main are symbolized in lldb, but the two frames between those calls have no symbol name. The Clang and lld-link invocations are also shown below.

-- Testing: 1 tests, 1 workers --
FAIL: lldb-shell :: Unwind/windows-unaligned-x86_64.test (1 of 1)
******************** TEST 'lldb-shell :: Unwind/windows-unaligned-x86_64.test' FAILED ********************
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 7
'C:\Program Files\Python312\python.exe' C:\workspace\llvm-project\lldb\test\Shell\helper\build.py --compiler=any --arch=64 --tools-dir=C:/workspace/llvm-project/build/./bin --libs-dir=C:/workspace/llvm-project/build/./lib --verbose C:\workspace\llvm-project\lldb\test\Shell\Unwind/Inputs/windows-unaligned-x86_64.cpp C:\workspace\llvm-project\lldb\test\Shell\Unwind/Inputs/windows-unaligned-x86_64-asm.s -o C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp
# executed command: 'C:\Program Files\Python312\python.exe' 'C:\workspace\llvm-project\lldb\test\Shell\helper\build.py' --compiler=any --arch=64 --tools-dir=C:/workspace/llvm-project/build/./bin --libs-dir=C:/workspace/llvm-project/build/./lib --verbose 'C:\workspace\llvm-project\lldb\test\Shell\Unwind/Inputs/windows-unaligned-x86_64.cpp' 'C:\workspace\llvm-project\lldb\test\Shell\Unwind/Inputs/windows-unaligned-x86_64-asm.s' -o 'C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp'
# .---command stdout------------
# | Script Arguments:
# |   Arch: 64
# |   Compiler: any
# |   Outdir: C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output
# |   Output: C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp
# |   Nodefaultlib: False
# |   Opt: none
# |   Mode: compile-and-link
# |   Clean: True
# |   Verbose: True
# |   Dryrun: False
# |   Inputs: C:\workspace\llvm-project\lldb\test\Shell\Unwind\Inputs\windows-unaligned-x86_64.cpp
# |           C:\workspace\llvm-project\lldb\test\Shell\Unwind\Inputs\windows-unaligned-x86_64-asm.s
# |   C/C++ Standard: None
# | Script Environment:
# |     COMSPEC = C:\WINDOWS\system32\cmd.exe
# |     INCLUDE = C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\include
# |               C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\ATLMFC\include
# |               C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\VS\include
# |               C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\ucrt
# |               C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\um
# |               C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\shared
# |               C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\winrt
# |               C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\cppwinrt
# |               C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um
# |     LD_LIBRARY_PATH = c:\workspace\llvm-project\build\bin
# |                       c:\workspace\llvm-project\build\lib
# |     LIB = C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\ATLMFC\lib\x64
# |           C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\lib\x64
# |           C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64
# |           C:\Program Files (x86)\Windows Kits\10\lib\10.0.26100.0\ucrt\x64
# |           C:\Program Files (x86)\Windows Kits\10\\lib\10.0.26100.0\\um\x64
# |     LLVM_DISABLE_CRASH_REPORT = 1
# |     PATH = c:\workspace\llvm-project\build\bin
# |            c:\program files\git\usr\bin
# |            c:\program files\microsoft visual studio\2022\community\vc\tools\llvm\x64\bin
# |            c:\program files\microsoft visual studio\2022\community\dia sdk\bin\amd64
# |            c:\program files\microsoft visual studio\2022\community\dia sdk\bin
# |            c:\binarycache\llvm\clang\bin
# |            c:\workspace\llvm-project\build\bin
# |            c:\program files\microsoft visual studio\2022\community\vc\tools\llvm\x64\bin
# |            c:\program files\microsoft visual studio\2022\community\dia sdk\bin\amd64
# |            c:\program files\microsoft visual studio\2022\community\vc\tools\msvc\14.40.33807\bin\hostx64\x64
# |            c:\program files\microsoft visual studio\2022\community\common7\ide\vc\vcpackages
# |            c:\program files\microsoft visual studio\2022\community\common7\ide\commonextensions\microsoft\teamfoundation\team explorer
# |            c:\program files (x86)\microsoft sdks\windows\v10.0a\bin\netfx 4.8 tools\x64
# |            c:\program files (x86)\windows kits\10\bin\10.0.26100.0\x64
# |            c:\program files (x86)\windows kits\10\bin\x64
# |            c:\program files\microsoft visual studio\2022\community\msbuild\current\bin\amd64
# |            c:\windows\microsoft.net\framework64\v4.0.30319
# |            c:\program files\microsoft visual studio\2022\community\common7\ide
# |            c:\program files\microsoft visual studio\2022\community\common7\tools
# |            c:\program files\python312\scripts
# |            c:\program files\python312
# |            c:\program files (x86)\microsoft sdks\azure\cli2\wbin
# |            c:\windows\system32
# |            c:\windows
# |            c:\windows\system32\wbem
# |            c:\windows\system32\windowspowershell\v1.0
# |            c:\windows\system32\openssh
# |            c:\program files\dotnet
# |            c:\program files\github cli
# |            c:\programdata\chocolatey\bin
# |            c:\program files\microsoft sql server\150\tools\binn
# |            c:\program files\nodejs
# |            c:\program files\cmake\bin
# |            c:\program files\amazon\awscliv2
# |            c:\program files\yubico\yubikey manager cli
# |            c:\program files\tailscale
# |            c:\program files (x86)\windows kits\10\windows performance toolkit
# |            c:\program files\nicklockwood\swiftformat\usr\bin
# |            c:\program files\mongodb\swiftlint\usr\bin
# |            c:\program files\git\cmd
# |            c:\program files\powershell\7
# |            c:\users\kenda\appdata\local\microsoft\windowsapps
# |            c:\program files\neovim\bin
# |            c:\users\kenda\appdata\local\microsoft\winget\packages\jqlang.jq_microsoft.winget.source_8wekyb3d8bbwe
# |            c:\users\kenda\appdata\roaming\npm
# |            c:\users\kenda\appdata\local\microsoft\winget\packages\microsoft.nuget_microsoft.winget.source_8wekyb3d8bbwe
# |            c:\users\kenda\appdata\local\programs\microsoft vs code\bin
# |            c:\users\kenda\appdata\local\microsoft\winget\links
# |            c:\program files\gnuwin\bin
# |            c:\program files\swig\swigwin-4.2.1
# |            c:\users\kenda\.dotnet\tools
# |            c:\users\kenda\bin
# |            c:\program files\perl\perl\bin
# |            c:\program files\microsoft visual studio\2022\community\vc\tools\msvc\14.40.33807\bin\hostx64\x64
# |            .
# |            c:\program files\microsoft visual studio\2022\community\vc\tools\llvm\x64\bin
# |            c:\program files\microsoft visual studio\2022\community\common7\ide\commonextensions\microsoft\cmake\cmake\bin
# |            c:\program files\microsoft visual studio\2022\community\common7\ide\commonextensions\microsoft\cmake\ninja
# |     PATHEXT = .COM
# |               .EXE
# |               .BAT
# |               .CMD
# |               .VBS
# |               .VBE
# |               .JS
# |               .JSE
# |               .WSF
# |               .WSH
# |               .MSC
# |               .PY
# |               .PYW
# |     PYTHONBUFFERED = 1
# |     SYSTEMDRIVE = C:
# |     SYSTEMROOT = C:\WINDOWS
# |     TEMP = C:\Users\Kenda\AppData\Local\Temp\lit-tmp-cu8fy0m1
# |     TEMPDIR = C:\Users\Kenda\AppData\Local\Temp\lit-tmp-cu8fy0m1
# |     TMP = C:\Users\Kenda\AppData\Local\Temp\lit-tmp-cu8fy0m1
# |     TMPDIR = C:\Users\Kenda\AppData\Local\Temp\lit-tmp-cu8fy0m1
# |     USERPROFILE = C:\Users\Kenda
# |     VCINSTALLDIR = C:\Program Files\Microsoft Visual Studio\2022\Community\VC\
# |     VCTOOLSINSTALLDIR = C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\
# |     VSINSTALLDIR = C:\Program Files\Microsoft Visual Studio\2022\Community\
# |     WINDOWSSDKDIR = C:\Program Files (x86)\Windows Kits\10\
# |     WINDOWSSDKLIBVERSION = 10.0.26100.0\
# |     __COMPAT_LAYER = RunAsInvoker
# | Using %VCToolsInstallDir% C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\
# | Using %WindowsSdkDir% C:\Program Files (x86)\Windows Kits\10\
# | Using %WindowsSDKLibVersion% 10.0.26100.0
# | Cleaning C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.ilk
# |   The file does not exist.
# | Cleaning C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64-asm.ilk
# |   The file does not exist.
# | Cleaning C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp-windows-unaligned-x86_64.obj
# |   The file was successfully cleaned.
# | Cleaning C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp-windows-unaligned-x86_64-asm.obj
# |   The file was successfully cleaned.
# | Cleaning C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.pdb
# |   The file was successfully cleaned.
# | Cleaning C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp
# |   The file was successfully cleaned.
# |
# |
# |
# | compiling windows-unaligned-x86_64.cpp -> windows-unaligned-x86_64.test.tmp-windows-unaligned-x86_64.obj
# |   Command Line: C:\workspace\llvm-project\build\bin\clang-cl.EXE -m64 /Od /Z7 -Xclang -fkeep-static-consts -fms-compatibility-version=19 /c /FoC:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp-windows-unaligned-x86_64.obj -- C:\workspace\llvm-project\lldb\test\Shell\Unwind\Inputs\windows-unaligned-x86_64.cpp
# |   Env:
# |     INCLUDE = C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\ATLMFC\include
# |               C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\include
# |               C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\ucrt
# |               C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\shared
# |               C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\um
# |               C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\winrt
# |               C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\cppwinrt
# |     SystemDrive = C:
# |     SystemRoot = C:\WINDOWS
# |     TMP = C:\Users\Kenda\AppData\Local\Temp\lit-tmp-cu8fy0m1
# |     TEMP = C:\Users\Kenda\AppData\Local\Temp\lit-tmp-cu8fy0m1
# |   STDOUT:
# |
# |
# |
# |
# | compiling windows-unaligned-x86_64-asm.s -> windows-unaligned-x86_64.test.tmp-windows-unaligned-x86_64-asm.obj
# |   Command Line: C:\workspace\llvm-project\build\bin\clang-cl.EXE -m64 /Od /Z7 -Xclang -fkeep-static-consts -fms-compatibility-version=19 /c /FoC:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp-windows-unaligned-x86_64-asm.obj -- C:\workspace\llvm-project\lldb\test\Shell\Unwind\Inputs\windows-unaligned-x86_64-asm.s
# |   Env:
# |     INCLUDE = C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\ATLMFC\include
# |               C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\include
# |               C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\ucrt
# |               C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\shared
# |               C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\um
# |               C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\winrt
# |               C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\cppwinrt
# |     SystemDrive = C:
# |     SystemRoot = C:\WINDOWS
# |     TMP = C:\Users\Kenda\AppData\Local\Temp\lit-tmp-cu8fy0m1
# |     TEMP = C:\Users\Kenda\AppData\Local\Temp\lit-tmp-cu8fy0m1
# |   STDOUT:
# |
# |
# |
# |
# | linking windows-unaligned-x86_64.test.tmp-windows-unaligned-x86_64.obj+windows-unaligned-x86_64.test.tmp-windows-unaligned-x86_64-asm.obj -> windows-unaligned-x86_64.test.tmp
# |   Command Line: C:\workspace\llvm-project\build\bin\lld-link.EXE /DEBUG:FULL /INCREMENTAL:NO /PDB:C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.pdb /OUT:C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp-windows-unaligned-x86_64.obj C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp-windows-unaligned-x86_64-asm.obj
# |   Env:
# |     LIB = C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\ATLMFC\lib\x64
# |           C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\lib\x64
# |           C:\Program Files (x86)\Windows Kits\10\lib\10.0.26100.0\ucrt\x64
# |           C:\Program Files (x86)\Windows Kits\10\lib\10.0.26100.0\um\x64
# |     SystemDrive = C:
# |     SystemRoot = C:\WINDOWS
# |     TMP = C:\Users\Kenda\AppData\Local\Temp\lit-tmp-cu8fy0m1
# |     TEMP = C:\Users\Kenda\AppData\Local\Temp\lit-tmp-cu8fy0m1
# |   STDOUT:
# |
# `-----------------------------
# RUN: at line 8
env LLDB_USE_NATIVE_PDB_READER=1 c:\workspace\llvm-project\build\bin\lldb.exe --no-lldbinit -S C:/workspace/llvm-project/build/tools/lldb\test\Shell\lit-lldb-init-quiet C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp -s C:\workspace\llvm-project\lldb\test\Shell\Unwind\windows-unaligned-x86_64.test -o exit | c:\workspace\llvm-project\build\bin\filecheck.exe C:\workspace\llvm-project\lldb\test\Shell\Unwind\windows-unaligned-x86_64.test
# executed command: env LLDB_USE_NATIVE_PDB_READER=1 'c:\workspace\llvm-project\build\bin\lldb.exe' --no-lldbinit -S 'C:/workspace/llvm-project/build/tools/lldb\test\Shell\lit-lldb-init-quiet' 'C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp' -s 'C:\workspace\llvm-project\lldb\test\Shell\Unwind\windows-unaligned-x86_64.test' -o exit
# note: command had no output on stdout or stderr
# executed command: 'c:\workspace\llvm-project\build\bin\filecheck.exe' 'C:\workspace\llvm-project\lldb\test\Shell\Unwind\windows-unaligned-x86_64.test'
# .---command stderr------------
# | C:\workspace\llvm-project\lldb\test\Shell\Unwind\windows-unaligned-x86_64.test:26:10: error: CHECK: expected string not found in input
# | # CHECK: frame #1: {{.*}}`realign_stack
# |          ^
# | <stdin>:16:69: note: scanning from here
# |  frame #0: 0x00007ff79aba1000 windows-unaligned-x86_64.test.tmp`func(arg=<unavailable>) at windows-unaligned-x86_64.cpp:3
# |                                                                     ^
# | <stdin>:27:2: note: possible intended match here
# |  frame #1: 0x00007ff79aba1059 windows-unaligned-x86_64.test.tmp
# |  ^
# |
# | Input file: <stdin>
# | Check file: C:\workspace\llvm-project\lldb\test\Shell\Unwind\windows-unaligned-x86_64.test
# |
# | -dump-input=help explains the following input dump.
# |
# | Input was:
# | <<<<<<
# |             .
# |             .
# |             .
# |            11: Breakpoint 1: where = windows-unaligned-x86_64.test.tmp`func at windows-unaligned-x86_64.cpp:3, address = 0x0000000140001000
# |            12: (lldb) process launch
# |            13: Process 49344 launched: 'C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp' (x86_64)
# |            14: Process 49344 stopped
# |            15: * thread #1, stop reason = breakpoint 1.1
# |            16:  frame #0: 0x00007ff79aba1000 windows-unaligned-x86_64.test.tmp`func(arg=<unavailable>) at windows-unaligned-x86_64.cpp:3
# | check:26'0                                                                         X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
# |            17:  1 extern "C" void call_func(void (*ptr)(int a), int a);
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            18:  2
# | check:26'0     ~~~~
# |            19: -> 3 extern "C" void func(int arg) { }
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            20:  4
# | check:26'0     ~~~~
# |            21:  5 int main(int argc, char **argv) {
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            22:  6 call_func(func, 42);
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~
# |            23:  7 return 0;
# | check:26'0     ~~~~~~~~~~~~~
# |            24: (lldb) thread backtrace
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~
# |            25: * thread #1, stop reason = breakpoint 1.1
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            26:  * frame #0: 0x00007ff79aba1000 windows-unaligned-x86_64.test.tmp`func(arg=<unavailable>) at windows-unaligned-x86_64.cpp:3
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            27:  frame #1: 0x00007ff79aba1059 windows-unaligned-x86_64.test.tmp
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# | check:26'1      ?                                                               possible intended match
# |            28:  frame #2: 0x00007ff79aba1049 windows-unaligned-x86_64.test.tmp
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            29:  frame #3: 0x00007ff79aba1036 windows-unaligned-x86_64.test.tmp`main(argc=1, argv=0x000001d785f4da20) at windows-unaligned-x86_64.cpp:6
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            30:  frame #4: 0x00007ff79aba12e0 windows-unaligned-x86_64.test.tmp`__scrt_common_main_seh [inlined] invoke_main at exe_common.inl:288
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            31:  frame #5: 0x00007ff79aba12be windows-unaligned-x86_64.test.tmp`__scrt_common_main_seh at exe_common.inl:288
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            32:  frame #6: 0x00007ffb47eb257d kernel32.dll`BaseThreadInitThunk + 29
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |             .
# |             .
# |             .
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

Questions

1. Is there any contributor that is currently running this test in their CI, and if so does it pass? (Perhaps @dzhidzhoev).
2. What toolchain differences would cause the realign_stack and call_func symbols not to exported in one build vs another? Different assembler? Compiler version? Linker version? I would like to fix this locally and then update the test's REQUIRES statement if possible.
3. What needs to be done to make LLDB's built in PDB reader function locally? If this test is running in someone's CI without LLDB_USE_NATIVE_PBD_READER=1, there must be some steps I skipped in setting up PDB support and I'd like to document these.

[llvmbot]

@llvm/issue-subscribers-lldb

Author: Kendal Harland (kendalharland)

CC @labath @mstorsjo

Overview

My local build of this test produces a binary whose realign_stack and call_func symbols are not visible to lldb so the test fails to match to the expected output of thread backtrace as these symbol names appear as empty strings in the stack trace output.

To make this test pass, I had to create this local patch which is certainly not the correct way to fix things but demonstrates my problem well.

Background

In particular, here are the problems I observe:

1. lld-link does not produce a binary that exports the realign_stack and call_func symbols which are defined as .globl in windows-unaligned-x86_64-asm.s, even though lldb-pdbutil indicates that these symbols are “public” in the final executable (The commands below are run in powershell but I built and ran these tests in cmd.exe):

PS C:\workspace\llvm-project> C:\workspace\llvm-project\build\bin\llvm-pdbutil.exe pretty --externals C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.pdb | Select-String -Pattern realign_stack

  public [0x0000104e] realign_stack

PS C:\workspace\llvm-project> C:\workspace\llvm-project\build\bin\llvm-pdbutil.exe pretty --externals C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.pdb | Select-String -Pattern call_func

  public [0x00001040] call_func

As a workaround I’ve manually exported the symbols in build.py using e.g. /EXPORT:realign_stack

2. LLDB doesn’t appear to load PDB symbols from a separate .pdb file automatically. It’s unclear if the original test author ran into this problem from their PR discussion so as a workaround I added an explicit add-dysm ... call to load the symbols.

3. LLDB seems to have a hard time parsing PDB files and crashes when add-dsym is called. I believe this is a known issue [1], [2]. Given that there are 2 PDB readers to choose from: LLDB’s implementation (default) and the Windows Native reader, as a workaround I’ve used the native one by setting LLDB_USE_NATIVE_PDB_READER=1.

Without my local patch, the full failure output with verbose information is below. As you can see, func and main are symbolized in lldb, but the two frames between those calls have no symbol name. The Clang and lld-link invocations are also shown below.

-- Testing: 1 tests, 1 workers --
FAIL: lldb-shell :: Unwind/windows-unaligned-x86_64.test (1 of 1)
******************** TEST 'lldb-shell :: Unwind/windows-unaligned-x86_64.test' FAILED ********************
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 7
'C:\Program Files\Python312\python.exe' C:\workspace\llvm-project\lldb\test\Shell\helper\build.py --compiler=any --arch=64 --tools-dir=C:/workspace/llvm-project/build/./bin --libs-dir=C:/workspace/llvm-project/build/./lib --verbose C:\workspace\llvm-project\lldb\test\Shell\Unwind/Inputs/windows-unaligned-x86_64.cpp C:\workspace\llvm-project\lldb\test\Shell\Unwind/Inputs/windows-unaligned-x86_64-asm.s -o C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp
# executed command: 'C:\Program Files\Python312\python.exe' 'C:\workspace\llvm-project\lldb\test\Shell\helper\build.py' --compiler=any --arch=64 --tools-dir=C:/workspace/llvm-project/build/./bin --libs-dir=C:/workspace/llvm-project/build/./lib --verbose 'C:\workspace\llvm-project\lldb\test\Shell\Unwind/Inputs/windows-unaligned-x86_64.cpp' 'C:\workspace\llvm-project\lldb\test\Shell\Unwind/Inputs/windows-unaligned-x86_64-asm.s' -o 'C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp'
# .---command stdout------------
# | Script Arguments:
# |   Arch: 64
# |   Compiler: any
# |   Outdir: C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output
# |   Output: C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp
# |   Nodefaultlib: False
# |   Opt: none
# |   Mode: compile-and-link
# |   Clean: True
# |   Verbose: True
# |   Dryrun: False
# |   Inputs: C:\workspace\llvm-project\lldb\test\Shell\Unwind\Inputs\windows-unaligned-x86_64.cpp
# |           C:\workspace\llvm-project\lldb\test\Shell\Unwind\Inputs\windows-unaligned-x86_64-asm.s
# |   C/C++ Standard: None
# | Script Environment:
# |     COMSPEC = C:\WINDOWS\system32\cmd.exe
# |     INCLUDE = C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\include
# |               C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\ATLMFC\include
# |               C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\VS\include
# |               C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\ucrt
# |               C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\um
# |               C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\shared
# |               C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\winrt
# |               C:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\cppwinrt
# |               C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um
# |     LD_LIBRARY_PATH = c:\workspace\llvm-project\build\bin
# |                       c:\workspace\llvm-project\build\lib
# |     LIB = C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\ATLMFC\lib\x64
# |           C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\lib\x64
# |           C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64
# |           C:\Program Files (x86)\Windows Kits\10\lib\10.0.26100.0\ucrt\x64
# |           C:\Program Files (x86)\Windows Kits\10\\lib\10.0.26100.0\\um\x64
# |     LLVM_DISABLE_CRASH_REPORT = 1
# |     PATH = c:\workspace\llvm-project\build\bin
# |            c:\program files\git\usr\bin
# |            c:\program files\microsoft visual studio\2022\community\vc\tools\llvm\x64\bin
# |            c:\program files\microsoft visual studio\2022\community\dia sdk\bin\amd64
# |            c:\program files\microsoft visual studio\2022\community\dia sdk\bin
# |            c:\binarycache\llvm\clang\bin
# |            c:\workspace\llvm-project\build\bin
# |            c:\program files\microsoft visual studio\2022\community\vc\tools\llvm\x64\bin
# |            c:\program files\microsoft visual studio\2022\community\dia sdk\bin\amd64
# |            c:\program files\microsoft visual studio\2022\community\vc\tools\msvc\14.40.33807\bin\hostx64\x64
# |            c:\program files\microsoft visual studio\2022\community\common7\ide\vc\vcpackages
# |            c:\program files\microsoft visual studio\2022\community\common7\ide\commonextensions\microsoft\teamfoundation\team explorer
# |            c:\program files (x86)\microsoft sdks\windows\v10.0a\bin\netfx 4.8 tools\x64
# |            c:\program files (x86)\windows kits\10\bin\10.0.26100.0\x64
# |            c:\program files (x86)\windows kits\10\bin\x64
# |            c:\program files\microsoft visual studio\2022\community\msbuild\current\bin\amd64
# |            c:\windows\microsoft.net\framework64\v4.0.30319
# |            c:\program files\microsoft visual studio\2022\community\common7\ide
# |            c:\program files\microsoft visual studio\2022\community\common7\tools
# |            c:\program files\python312\scripts
# |            c:\program files\python312
# |            c:\program files (x86)\microsoft sdks\azure\cli2\wbin
# |            c:\windows\system32
# |            c:\windows
# |            c:\windows\system32\wbem
# |            c:\windows\system32\windowspowershell\v1.0
# |            c:\windows\system32\openssh
# |            c:\program files\dotnet
# |            c:\program files\github cli
# |            c:\programdata\chocolatey\bin
# |            c:\program files\microsoft sql server\150\tools\binn
# |            c:\program files\nodejs
# |            c:\program files\cmake\bin
# |            c:\program files\amazon\awscliv2
# |            c:\program files\yubico\yubikey manager cli
# |            c:\program files\tailscale
# |            c:\program files (x86)\windows kits\10\windows performance toolkit
# |            c:\program files\nicklockwood\swiftformat\usr\bin
# |            c:\program files\mongodb\swiftlint\usr\bin
# |            c:\program files\git\cmd
# |            c:\program files\powershell\7
# |            c:\users\kenda\appdata\local\microsoft\windowsapps
# |            c:\program files\neovim\bin
# |            c:\users\kenda\appdata\local\microsoft\winget\packages\jqlang.jq_microsoft.winget.source_8wekyb3d8bbwe
# |            c:\users\kenda\appdata\roaming\npm
# |            c:\users\kenda\appdata\local\microsoft\winget\packages\microsoft.nuget_microsoft.winget.source_8wekyb3d8bbwe
# |            c:\users\kenda\appdata\local\programs\microsoft vs code\bin
# |            c:\users\kenda\appdata\local\microsoft\winget\links
# |            c:\program files\gnuwin\bin
# |            c:\program files\swig\swigwin-4.2.1
# |            c:\users\kenda\.dotnet\tools
# |            c:\users\kenda\bin
# |            c:\program files\perl\perl\bin
# |            c:\program files\microsoft visual studio\2022\community\vc\tools\msvc\14.40.33807\bin\hostx64\x64
# |            .
# |            c:\program files\microsoft visual studio\2022\community\vc\tools\llvm\x64\bin
# |            c:\program files\microsoft visual studio\2022\community\common7\ide\commonextensions\microsoft\cmake\cmake\bin
# |            c:\program files\microsoft visual studio\2022\community\common7\ide\commonextensions\microsoft\cmake\ninja
# |     PATHEXT = .COM
# |               .EXE
# |               .BAT
# |               .CMD
# |               .VBS
# |               .VBE
# |               .JS
# |               .JSE
# |               .WSF
# |               .WSH
# |               .MSC
# |               .PY
# |               .PYW
# |     PYTHONBUFFERED = 1
# |     SYSTEMDRIVE = C:
# |     SYSTEMROOT = C:\WINDOWS
# |     TEMP = C:\Users\Kenda\AppData\Local\Temp\lit-tmp-cu8fy0m1
# |     TEMPDIR = C:\Users\Kenda\AppData\Local\Temp\lit-tmp-cu8fy0m1
# |     TMP = C:\Users\Kenda\AppData\Local\Temp\lit-tmp-cu8fy0m1
# |     TMPDIR = C:\Users\Kenda\AppData\Local\Temp\lit-tmp-cu8fy0m1
# |     USERPROFILE = C:\Users\Kenda
# |     VCINSTALLDIR = C:\Program Files\Microsoft Visual Studio\2022\Community\VC\
# |     VCTOOLSINSTALLDIR = C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\
# |     VSINSTALLDIR = C:\Program Files\Microsoft Visual Studio\2022\Community\
# |     WINDOWSSDKDIR = C:\Program Files (x86)\Windows Kits\10\
# |     WINDOWSSDKLIBVERSION = 10.0.26100.0\
# |     __COMPAT_LAYER = RunAsInvoker
# | Using %VCToolsInstallDir% C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\
# | Using %WindowsSdkDir% C:\Program Files (x86)\Windows Kits\10\
# | Using %WindowsSDKLibVersion% 10.0.26100.0
# | Cleaning C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.ilk
# |   The file does not exist.
# | Cleaning C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64-asm.ilk
# |   The file does not exist.
# | Cleaning C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp-windows-unaligned-x86_64.obj
# |   The file was successfully cleaned.
# | Cleaning C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp-windows-unaligned-x86_64-asm.obj
# |   The file was successfully cleaned.
# | Cleaning C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.pdb
# |   The file was successfully cleaned.
# | Cleaning C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp
# |   The file was successfully cleaned.
# |
# |
# |
# | compiling windows-unaligned-x86_64.cpp -> windows-unaligned-x86_64.test.tmp-windows-unaligned-x86_64.obj
# |   Command Line: C:\workspace\llvm-project\build\bin\clang-cl.EXE -m64 /Od /Z7 -Xclang -fkeep-static-consts -fms-compatibility-version=19 /c /FoC:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp-windows-unaligned-x86_64.obj -- C:\workspace\llvm-project\lldb\test\Shell\Unwind\Inputs\windows-unaligned-x86_64.cpp
# |   Env:
# |     INCLUDE = C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\ATLMFC\include
# |               C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\include
# |               C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\ucrt
# |               C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\shared
# |               C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\um
# |               C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\winrt
# |               C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\cppwinrt
# |     SystemDrive = C:
# |     SystemRoot = C:\WINDOWS
# |     TMP = C:\Users\Kenda\AppData\Local\Temp\lit-tmp-cu8fy0m1
# |     TEMP = C:\Users\Kenda\AppData\Local\Temp\lit-tmp-cu8fy0m1
# |   STDOUT:
# |
# |
# |
# |
# | compiling windows-unaligned-x86_64-asm.s -> windows-unaligned-x86_64.test.tmp-windows-unaligned-x86_64-asm.obj
# |   Command Line: C:\workspace\llvm-project\build\bin\clang-cl.EXE -m64 /Od /Z7 -Xclang -fkeep-static-consts -fms-compatibility-version=19 /c /FoC:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp-windows-unaligned-x86_64-asm.obj -- C:\workspace\llvm-project\lldb\test\Shell\Unwind\Inputs\windows-unaligned-x86_64-asm.s
# |   Env:
# |     INCLUDE = C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\ATLMFC\include
# |               C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\include
# |               C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\ucrt
# |               C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\shared
# |               C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\um
# |               C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\winrt
# |               C:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\cppwinrt
# |     SystemDrive = C:
# |     SystemRoot = C:\WINDOWS
# |     TMP = C:\Users\Kenda\AppData\Local\Temp\lit-tmp-cu8fy0m1
# |     TEMP = C:\Users\Kenda\AppData\Local\Temp\lit-tmp-cu8fy0m1
# |   STDOUT:
# |
# |
# |
# |
# | linking windows-unaligned-x86_64.test.tmp-windows-unaligned-x86_64.obj+windows-unaligned-x86_64.test.tmp-windows-unaligned-x86_64-asm.obj -> windows-unaligned-x86_64.test.tmp
# |   Command Line: C:\workspace\llvm-project\build\bin\lld-link.EXE /DEBUG:FULL /INCREMENTAL:NO /PDB:C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.pdb /OUT:C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp-windows-unaligned-x86_64.obj C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp-windows-unaligned-x86_64-asm.obj
# |   Env:
# |     LIB = C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\ATLMFC\lib\x64
# |           C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\lib\x64
# |           C:\Program Files (x86)\Windows Kits\10\lib\10.0.26100.0\ucrt\x64
# |           C:\Program Files (x86)\Windows Kits\10\lib\10.0.26100.0\um\x64
# |     SystemDrive = C:
# |     SystemRoot = C:\WINDOWS
# |     TMP = C:\Users\Kenda\AppData\Local\Temp\lit-tmp-cu8fy0m1
# |     TEMP = C:\Users\Kenda\AppData\Local\Temp\lit-tmp-cu8fy0m1
# |   STDOUT:
# |
# `-----------------------------
# RUN: at line 8
env LLDB_USE_NATIVE_PDB_READER=1 c:\workspace\llvm-project\build\bin\lldb.exe --no-lldbinit -S C:/workspace/llvm-project/build/tools/lldb\test\Shell\lit-lldb-init-quiet C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp -s C:\workspace\llvm-project\lldb\test\Shell\Unwind\windows-unaligned-x86_64.test -o exit | c:\workspace\llvm-project\build\bin\filecheck.exe C:\workspace\llvm-project\lldb\test\Shell\Unwind\windows-unaligned-x86_64.test
# executed command: env LLDB_USE_NATIVE_PDB_READER=1 'c:\workspace\llvm-project\build\bin\lldb.exe' --no-lldbinit -S 'C:/workspace/llvm-project/build/tools/lldb\test\Shell\lit-lldb-init-quiet' 'C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp' -s 'C:\workspace\llvm-project\lldb\test\Shell\Unwind\windows-unaligned-x86_64.test' -o exit
# note: command had no output on stdout or stderr
# executed command: 'c:\workspace\llvm-project\build\bin\filecheck.exe' 'C:\workspace\llvm-project\lldb\test\Shell\Unwind\windows-unaligned-x86_64.test'
# .---command stderr------------
# | C:\workspace\llvm-project\lldb\test\Shell\Unwind\windows-unaligned-x86_64.test:26:10: error: CHECK: expected string not found in input
# | # CHECK: frame #<!-- -->1: {{.*}}`realign_stack
# |          ^
# | &lt;stdin&gt;:16:69: note: scanning from here
# |  frame #<!-- -->0: 0x00007ff79aba1000 windows-unaligned-x86_64.test.tmp`func(arg=&lt;unavailable&gt;) at windows-unaligned-x86_64.cpp:3
# |                                                                     ^
# | &lt;stdin&gt;:27:2: note: possible intended match here
# |  frame #<!-- -->1: 0x00007ff79aba1059 windows-unaligned-x86_64.test.tmp
# |  ^
# |
# | Input file: &lt;stdin&gt;
# | Check file: C:\workspace\llvm-project\lldb\test\Shell\Unwind\windows-unaligned-x86_64.test
# |
# | -dump-input=help explains the following input dump.
# |
# | Input was:
# | &lt;&lt;&lt;&lt;&lt;&lt;
# |             .
# |             .
# |             .
# |            11: Breakpoint 1: where = windows-unaligned-x86_64.test.tmp`func at windows-unaligned-x86_64.cpp:3, address = 0x0000000140001000
# |            12: (lldb) process launch
# |            13: Process 49344 launched: 'C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.tmp' (x86_64)
# |            14: Process 49344 stopped
# |            15: * thread #<!-- -->1, stop reason = breakpoint 1.1
# |            16:  frame #<!-- -->0: 0x00007ff79aba1000 windows-unaligned-x86_64.test.tmp`func(arg=&lt;unavailable&gt;) at windows-unaligned-x86_64.cpp:3
# | check:26'0                                                                         X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
# |            17:  1 extern "C" void call_func(void (*ptr)(int a), int a);
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            18:  2
# | check:26'0     ~~~~
# |            19: -&gt; 3 extern "C" void func(int arg) { }
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            20:  4
# | check:26'0     ~~~~
# |            21:  5 int main(int argc, char **argv) {
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            22:  6 call_func(func, 42);
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~
# |            23:  7 return 0;
# | check:26'0     ~~~~~~~~~~~~~
# |            24: (lldb) thread backtrace
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~
# |            25: * thread #<!-- -->1, stop reason = breakpoint 1.1
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            26:  * frame #<!-- -->0: 0x00007ff79aba1000 windows-unaligned-x86_64.test.tmp`func(arg=&lt;unavailable&gt;) at windows-unaligned-x86_64.cpp:3
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            27:  frame #<!-- -->1: 0x00007ff79aba1059 windows-unaligned-x86_64.test.tmp
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# | check:26'1      ?                                                               possible intended match
# |            28:  frame #<!-- -->2: 0x00007ff79aba1049 windows-unaligned-x86_64.test.tmp
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            29:  frame #<!-- -->3: 0x00007ff79aba1036 windows-unaligned-x86_64.test.tmp`main(argc=1, argv=0x000001d785f4da20) at windows-unaligned-x86_64.cpp:6
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            30:  frame #<!-- -->4: 0x00007ff79aba12e0 windows-unaligned-x86_64.test.tmp`__scrt_common_main_seh [inlined] invoke_main at exe_common.inl:288
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            31:  frame #<!-- -->5: 0x00007ff79aba12be windows-unaligned-x86_64.test.tmp`__scrt_common_main_seh at exe_common.inl:288
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            32:  frame #<!-- -->6: 0x00007ffb47eb257d kernel32.dll`BaseThreadInitThunk + 29
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |             .
# |             .
# |             .
# | &gt;&gt;&gt;&gt;&gt;&gt;
# `-----------------------------
# error: command failed with exit status: 1

Questions

1. Is there any contributor that is currently running this test in their CI, and if so does it pass? (Perhaps @dzhidzhoev).
2. What toolchain differences would cause the realign_stack and call_func symbols not to exported in one build vs another? Different assembler? Compiler version? Linker version? I would like to fix this locally and then update the test's REQUIRES statement if possible.
3. What needs to be done to make LLDB's built in PDB reader function locally? If this test is running in someone's CI without LLDB_USE_NATIVE_PBD_READER=1, there must be some steps I skipped in setting up PDB support and I'd like to document these.

[dzhidzhoev]

Is there any contributor that is currently running this test in their CI, and if so does it pass? (Perhaps @dzhidzhoev).

For us, it fails on breakpoint set -n func line.

[kendalharland]

Is there any contributor that is currently running this test in their CI, and if so does it pass? (Perhaps @dzhidzhoev).

For us, it fails on breakpoint set -n func line.

Thanks for confirming! IIRC this is because LLDB does not load the debug information automatically (I am also unsure if it is supposed to) This error should be resolved via the extra add-dsym call in my patch above, then the test will go on to fail when it attempts to match realign_stack and call_func in thread backtrace output.

[kendalharland]

Related to #100474

[mstorsjo]

This test does pass for me. In a build with -DLLVM_ENABLE_PROJECTS="clang;lld;lldb" -DLLVM_ENABLE_DIA_SDK=TRUE (where the build is set up with MSVC), the test does pass, without any modifications.

LLVM's own PDB reader is indeed a bit incomplete, but AFAIK all buildbots that run configurations with LLDB on Windows run with -DLLVM_ENABLE_DIA_SDK=TRUE.

[labath]

CC @labath @mstorsjo

Overview

My local build of this test produces a binary whose realign_stack and call_func symbols are not visible to lldb so the test fails to match to the expected output of thread backtrace as these symbol names appear as empty strings in the stack trace output.

To make this test pass, I had to create this local patch which is certainly not the correct way to fix things but demonstrates my problem well.

Background

In particular, here are the problems I observe:

1. lld-link does not produce a binary that exports the `realign_stack` and `call_func` symbols  which are defined as `.globl` in [windows-unaligned-x86_64-asm.s](https://github.com/llvm/llvm-project/blob/90065da6d5a5f661b60c2f75b0f2dc094d27f4f5/lldb/test/Shell/Unwind/Inputs/windows-unaligned-x86_64-asm.s), 

I think this part is expected. .globl makes the symbol visible to other compile units in the same library. To export it to other libraries, extra steps (like the ones you did are needed). At least on windows that is -- with elf, global symbols are exported by default, and must be explicitly hidden if desired.

even though lldb-pdbutil indicates that these symbols are “public” in the final executable (The commands below are run in powershell but I built and ran these tests in cmd.exe):

PS C:\workspace\llvm-project> C:\workspace\llvm-project\build\bin\llvm-pdbutil.exe pretty --externals C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.pdb | Select-String -Pattern realign_stack

  public [0x0000104e] realign_stack

PS C:\workspace\llvm-project> C:\workspace\llvm-project\build\bin\llvm-pdbutil.exe pretty --externals C:\workspace\llvm-project\build\tools\lldb\test\Shell\Unwind\Output\windows-unaligned-x86_64.test.pdb | Select-String -Pattern call_func

  public [0x00001040] call_func

LLDB's image lookup command fails to find these symbols by address or name.

I guess that's a bug. If they're in the debug info, lldb should find them, whether or not they are exported.

As a workaround I’ve manually exported the symbols in build.py using e.g. /EXPORT:realign_stack

2. LLDB doesn’t appear to load PDB symbols from a separate `.pdb` file automatically. 

It should, and I think it does so in some cases, but it's quite possible this does not work all the time.

It’s unclear if the original test author ran into this problem from their PR discussion so as a workaround I added an explicit add-dsym ... call to load the symbols.

2. What toolchain differences would cause the `realign_stack` and `call_func` symbols not to exported in one build vs another? Different assembler? Compiler version? Linker version? I would like to fix this locally and then update the test's `REQUIRES` statement if possible.

Like I said above, I think the exportedness of the symbol is not the issue, and I don't expect any windows compiler to export them by default (well, maybe some cygwin stuff, which tries to emulate elf behavior).

That said, all of the above things could affect the exact kind of debug info (PDB) being produced, and thus lldb's ability to parse it.

3. What needs to be done to make LLDB's built in PDB reader function locally? If this test is running in someone's CI without `LLDB_USE_NATIVE_PBD_READER=1`, there must be some steps I skipped in setting up PDB support and I'd like to document these.

From where I'm sitting, the ideal outcome would be to flip the settings to use the "native" pdb reader by default. :)

[kendalharland]

Thanks for the added context @labath! Flipping the settings to use the native reader by default seems reasonable to me.

Like I said above, I think the exportedness of the symbol is not the issue, and I don't expect any windows compiler to export them by default (well, maybe some cygwin stuff, which tries to emulate elf behavior).

Makes sense. This tells me that although I can fix the problem locally by /EXPORT ing the symbols, it's not really fixing the true issue. This leaves me only to figure out why these symbols are not rendered in my thread backtrace output even with the Native PDB reader enabled, locally. Instead their backtrace lines contain empty strings where the names should be. Maybe @mstorsjo can help? Would you have any clue as to where I should start debugging this?

My full set of setup and build commands is:

# In an Admin cmd.exe
regsvr32 "C:\Program Files\Microsoft Visual Studio\2022\Community\DIA SDK\bin\msdia140.dll"
regsvr32 "C:\Program Files\Microsoft Visual Studio\2022\Community\DIA SDK\bin\amd64\msdia140.dll"

# In a non Admin cmd.exe
"C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\VsDevCmd.bat" -host_arch=amd64 -arch=amd64
"C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\VsDevCmd.bat" -test

set LLDB_USE_LLDB_SERVER=1
set MSVC_INSTALL_DIR=C:\Program Files\Microsoft Visual Studio\2022\Community
set "PATH=%MSVC_INSTALL_DIR%\DIA SDK\bin\amd64;%PATH%"
set "PATH=%MSVC_INSTALL_DIR%\VC\Tools\Llvm\x64\bin;%PATH%"
set "PATH=C:\workspace\llvm-project\build\bin;%PATH%"
set CC=clang-cl
set CXX=clang-cl

cmake -G Ninja ^
      -B build ^
      -D Python3_EXECUTABLE="C:\Program Files\Python312\python.exe" ^
      -D CMAKE_C_COMPILER_LAUNCHER=ccache ^
      -D CMAKE_CXX_COMPILER_LAUNCHER=ccache ^
      -D CMAKE_BUILD_TYPE="Release" ^
      -D LLVM_CCACHE_BUILD="ON" ^
      -D LLVM_ENABLE_PROJECTS="llvm;clang;lldb;lld" ^
      -D LLVM_ENABLE_LLD="YES" ^
      -D LLVM_ENABLE_DIA_SDK="ON" ^
      -D LLVM_LIT_ARGS=-v ^
      -D LLDB_ENABLE_PYTHON=1 ^
      -D LLDB_PYTHON_EXE_RELATIVE_PATH="python.exe" ^
      -D LLDB_TEST_USER_ARGS=--skip-category=watchpoint ^
      -D LLDB_ENFORCE_STRICT_TEST_REQUIREMENTS=1 ^
      llvm

ninja -C build check-lldb

I believe the main difference here might be that you are setting up your build with MSVC and I'm using Clang with MSVC CLI. I can start there and see if that fixes the problem.

[kendalharland]

Can confirm that switching to MSVC does not fix this for me.

[mstorsjo]

For some reason, the test is now failing for me too, in my local test environment. I have no idea how it worked for me two weeks ago though.

But now I see what's going wrong.

The issue is that when building the assembly source file, Shell/Unwind/Inputs/windows-unaligned-x86_64-asm.s, the output object file gets DWARF debug info generated. This is a change that happened in Clang 18, which seems to have started occurring since f58330c. (This seems to be an unintended effect of this commit.)

When the assembly files is compiled, the command run looks like clang-cl /c /Z7 /Foout.o input.s. The /Z7 is included to enable generation of codeview debug info in the output object file - but when the input is a plain .s assembly file, it seems like it triggers generating some DWARF debug info.

To see this in action, run the following:

> bin\clang-cl /Z7 /c /Foout.o ..\..\lldb\test\Shell\Unwind\Inputs\windows-unaligned-x86_64-asm.s
> bin\llvm-readobj --sections out.o

Note that the output contains a couple of sections like .debug_info and .debug_line. When the /Z7 option is omitted, these sections aren't generated. When using a clang-cl from version 17 or earlier, those sections aren't generated either.

When linking the output binary, we do generate PDB debug info, as that's what we request. But as one object file did include DWARF debug info, the output also contains a little bit of DWARF. When LLDB inspects the binary, it looks for both DWARF and PDB. In this case, it finds a little bit of DWARF debug info, and LLDB concludes the search for related debug info, not looking for the PDB. That's why it can't resolve the symbols that normally would be provided by the debug info (either PDB, or DWARF when the test is run in mingw mode).

So in this case, there are many ways around this (some more proof of concept, others more reasonable):

• Run llvm-strip on the output binary, to remove any unintended DWARF debug info. (This only should be done when the toolchain is operating in clang-cl mode though, in mingw mode, DWARF would be the default and intended debug output format, and shouldn't be stripped.)
• Run llvm-strip --strip-debug on the object file corresponding to the assembly input
• Don't pass /Z7 to the compiler, in the build.py helper script, when the input is an assembly file
• Make Clang/LLVM not generate DWARF debug info for assembly outputs, when the target triple is an MSVC target, unless an option like -gdwarf has been passed. I.e. fix the apparently unintended consequence of f58330c for this case.

This does seem like a clear regression in the Clang 18 timeframe.

But what's puzzling me is how the test does seem to work for some, like I would presume for @dzhidzhoev and probably also on some buildbots, and like how it did seem to work for me 2 weeks ago? Can the build.py script end up running e.g. a host installed clang-cl version rather than the one that is recently built, in the same build tree?