mingw-64 x86_64 toolchain gdb hangs during debuginfo tests

[brson]

On the bots, msys2 comes with version 7.6.1 of gdb. The installed x86_64-4.9.1-win32-seh-rt_v3-rev1 toolchain comes with gdb 7.8.1. When running the test suite with the latter, gdb hangs on all debuginfo tests; with the former it succeeds.

cc @michaelwoerister @vadimcn @klutzy

[brson]

To complete #16457 I'm just going to change PATH to prefer the msys2 gdb.

[michaelwoerister]

For me it also hangs with GDB 7.6.1 from msys2. GDB seems to run into some kind of endless loop since it utilizes 100% of one CPU core.

The freeze always happens when the debugger encounters the function _zzz() (taking test/debuginfo/basic-types.rs as example here). Setting a breakpoint in the functions works fine, but actually breaking in it or stepping into it from main() triggers the freeze.

Modifying the function to contain a call to ::std::io::print() or declaring two integer variables in it will "solve" the problem. Declaring just one variable in it doesn't help. My suspiscion is that LLVM generates some debuginfo that GDB can't deal with on Windows. I don't know yet how to further debug this problem. I'll probably have to find out where GDB actually hangs.

[michaelwoerister]

This is what I found out so far:

• The error does not occur with GDB 7.6.2 which comes with my new install of msys2
• The error does with GDB 7.8 from the toolchain mentioned above.
• The error is specific to 64 bit. With the equivalent 32 bit toolchain everything works just fine.
• GDB gets stuck in some kind of infinite loop. Here are some stack traces:

#0  0x00000000005418ba in memory_xfer_partial_1 ()
#1  0x0000000000541d28 in target_xfer_partial ()
#2  0x00000000005423dd in target_read ()
#3  0x0000000000542439 in target_read_memory ()
#4  0x0000000000410563 in amd64_windows_frame_decode_insns ()
#5  0x0000000000410964 in amd64_windows_frame_cache ()
#6  0x0000000000410b00 in amd64_windows_frame_this_id ()
#7  0x00000000005df7ee in get_prev_frame_if_no_cycle ()
#8  0x00000000005e1a89 in get_prev_frame_always ()
#9  0x00000000005e2201 in get_prev_frame ()
#10 0x00000000005e24cc in unwind_to_current_frame ()
#11 0x000000000051a008 in catch_exceptions_with_msg ()
#12 0x000000000051a141 in catch_exceptions ()
#13 0x00000000005dfb09 in get_current_frame ()
#14 0x000000000050ccb4 in handle_inferior_event.part ()
#15 0x000000000050e61c in wait_for_inferior ()
#16 0x000000000050e950 in proceed ()
#17 0x0000000000502ae2 in run_command_1 ()
#18 0x00000000005d7beb in execute_command ()
#19 0x00000000005d89fd in command_loop ()
#20 0x00000000005d8acb in read_command_file ()
#21 0x00000000004577d2 in script_from_file ()
#22 0x000000000045994b in source_script_with_search ()
#23 0x000000000051cb03 in catch_command_errors_const.constprop ()
#24 0x000000000051d72c in captured_main ()
#25 0x000000000051a1c5 in catch_errors ()
#26 0x000000000051dbd0 in gdb_main ()
#27 0x0000000000724a88 in main ()


#0  0x000000000041056b in amd64_windows_frame_decode_insns ()
#1  0x0000000000410964 in amd64_windows_frame_cache ()
#2  0x0000000000410b00 in amd64_windows_frame_this_id ()
#3  0x00000000005df7ee in get_prev_frame_if_no_cycle ()
#4  0x00000000005e1a89 in get_prev_frame_always ()
#5  0x00000000005e2201 in get_prev_frame ()
#6  0x00000000005e24cc in unwind_to_current_frame ()
#7  0x000000000051a008 in catch_exceptions_with_msg ()
#8  0x000000000051a141 in catch_exceptions ()
#9  0x00000000005dfb09 in get_current_frame ()
#10 0x000000000050ccb4 in handle_inferior_event.part ()
#11 0x000000000050e61c in wait_for_inferior ()
#12 0x000000000050e950 in proceed ()
#13 0x0000000000502ae2 in run_command_1 ()
#14 0x00000000005d7beb in execute_command ()
#15 0x00000000005d89fd in command_loop ()
#16 0x00000000005d8acb in read_command_file ()
#17 0x00000000004577d2 in script_from_file ()
#18 0x000000000045994b in source_script_with_search ()
#19 0x000000000051cb03 in catch_command_errors_const.constprop ()
#20 0x000000000051d72c in captured_main ()
#21 0x000000000051a1c5 in catch_errors ()
#22 0x000000000051dbd0 in gdb_main ()
#23 0x0000000000724a88 in main ()


#0  0x0000000000541b4d in memory_xfer_partial_1 ()
#1  0x0000000000541d28 in target_xfer_partial ()
#2  0x00000000005423dd in target_read ()
#3  0x0000000000542439 in target_read_memory ()
#4  0x0000000000410563 in amd64_windows_frame_decode_insns ()
#5  0x0000000000410964 in amd64_windows_frame_cache ()
#6  0x0000000000410b00 in amd64_windows_frame_this_id ()
#7  0x00000000005df7ee in get_prev_frame_if_no_cycle ()
#8  0x00000000005e1a89 in get_prev_frame_always ()
#9  0x00000000005e2201 in get_prev_frame ()
#10 0x00000000005e24cc in unwind_to_current_frame ()
#11 0x000000000051a008 in catch_exceptions_with_msg ()
#12 0x000000000051a141 in catch_exceptions ()
#13 0x00000000005dfb09 in get_current_frame ()
#14 0x000000000050ccb4 in handle_inferior_event.part ()
#15 0x000000000050e61c in wait_for_inferior ()
#16 0x000000000050e950 in proceed ()
#17 0x0000000000502ae2 in run_command_1 ()
#18 0x00000000005d7beb in execute_command ()
#19 0x00000000005d89fd in command_loop ()
#20 0x00000000005d8acb in read_command_file ()
#21 0x00000000004577d2 in script_from_file ()
#22 0x000000000045994b in source_script_with_search ()
#23 0x000000000051cb03 in catch_command_errors_const.constprop ()
#24 0x000000000051d72c in captured_main ()
#25 0x000000000051a1c5 in catch_errors ()
#26 0x000000000051dbd0 in gdb_main ()
#27 0x0000000000724a88 in main ()


#0  0x00000000777f16aa in ntdll!ZwReadVirtualMemory ()
   from C:\Windows\SYSTEM32\ntdll.dll
#1  0x000007fefd7355d4 in ReadProcessMemory ()
   from C:\Windows\system32\KernelBase.dll
#2  0x00000000776cc183 in ReadProcessMemory ()
   from C:\Windows\system32\kernel32.dll
#3  0x000000000042afcf in windows_xfer_partial ()
#4  0x0000000000540702 in raw_memory_xfer_partial ()
#5  0x0000000000541d28 in target_xfer_partial ()
#6  0x00000000005423dd in target_read ()
#7  0x0000000000542439 in target_read_memory ()
#8  0x0000000000410563 in amd64_windows_frame_decode_insns ()
#9  0x0000000000410964 in amd64_windows_frame_cache ()
#10 0x0000000000410b00 in amd64_windows_frame_this_id ()
#11 0x00000000005df7ee in get_prev_frame_if_no_cycle ()
#12 0x00000000005e1a89 in get_prev_frame_always ()
#13 0x00000000005e2201 in get_prev_frame ()
#14 0x00000000005e24cc in unwind_to_current_frame ()
#15 0x000000000051a008 in catch_exceptions_with_msg ()
#16 0x000000000051a141 in catch_exceptions ()
#17 0x00000000005dfb09 in get_current_frame ()
#18 0x000000000050ccb4 in handle_inferior_event.part ()
#19 0x000000000050e61c in wait_for_inferior ()
#20 0x000000000050e950 in proceed ()
#21 0x0000000000502ae2 in run_command_1 ()
#22 0x00000000005d7beb in execute_command ()
#23 0x00000000005d89fd in command_loop ()
#24 0x00000000005d8acb in read_command_file ()
#25 0x00000000004577d2 in script_from_file ()
#26 0x000000000045994b in source_script_with_search ()
#27 0x000000000051cb03 in catch_command_errors_const.constprop ()
#28 0x000000000051d72c in captured_main ()
#29 0x000000000051a1c5 in catch_errors ()
#30 0x000000000051dbd0 in gdb_main ()
#31 0x0000000000724a88 in main ()


#0  0x000000000053c8e0 in lookup_mem_region ()
#1  0x0000000000541911 in memory_xfer_partial_1 ()
#2  0x0000000000541d28 in target_xfer_partial ()
#3  0x00000000005423dd in target_read ()
#4  0x0000000000542439 in target_read_memory ()
#5  0x0000000000410563 in amd64_windows_frame_decode_insns ()
#6  0x0000000000410964 in amd64_windows_frame_cache ()
#7  0x0000000000410b00 in amd64_windows_frame_this_id ()
#8  0x00000000005df7ee in get_prev_frame_if_no_cycle ()
#9  0x00000000005e1a89 in get_prev_frame_always ()
#10 0x00000000005e2201 in get_prev_frame ()
#11 0x00000000005e24cc in unwind_to_current_frame ()
#12 0x000000000051a008 in catch_exceptions_with_msg ()
#13 0x000000000051a141 in catch_exceptions ()
#14 0x00000000005dfb09 in get_current_frame ()
#15 0x000000000050ccb4 in handle_inferior_event.part ()
#16 0x000000000050e61c in wait_for_inferior ()
#17 0x000000000050e950 in proceed ()
#18 0x0000000000502ae2 in run_command_1 ()
#19 0x00000000005d7beb in execute_command ()
#20 0x00000000005d89fd in command_loop ()
#21 0x00000000005d8acb in read_command_file ()
#22 0x00000000004577d2 in script_from_file ()
#23 0x000000000045994b in source_script_with_search ()
#24 0x000000000051cb03 in catch_command_errors_const.constprop ()
#25 0x000000000051d72c in captured_main ()
#26 0x000000000051a1c5 in catch_errors ()
#27 0x000000000051dbd0 in gdb_main ()
#28 0x0000000000724a88 in main ()

• The GDB's amd64_windows_frame_decode_insns() function, which is near the top of all backtraces, looks like it could easily get stuck in an infinite loop.

I suspect that LLVM produces some kind of machine code sequence that GDB can't deal with, but it does so only for very small functions like fn zzz() { () } which is used as marker throughout the debuginfo tests.

My suggested way forward for next week:

• File a bug report for GDB
• Work around the hang issue by either (a) modify the marker function so it's not poisonous anymore, or (b) only use line-numbers to set breakpoints in the test cases as is already done in the LLDB tests.

[michaelwoerister]

Bug report about underlying GDB issue:
https://sourceware.org/bugzilla/show_bug.cgi?id=17521

[michaelwoerister]

@brson,
since this should be fixed now, should we try to use the 'proper' GDB version on the test bots again?

[brson]

@michaelwoerister it does not need to be fixed right now (though it would be good to fix). I will try switching the bots back to the other toolchain soon.

[michaelwoerister]

OK, there's no hurry from my side. Just wanted to let you know that this should be a non-issue by now.

[brson]

Trying to switch the toolchain today.

[michaelwoerister]

fingers crossed

[brson]

Done. Thanks @michaelwoerister