concurrent_events tests: wrong breakpoint hit with multiple concurrent breakpoints

[llvmbot]

Bugzilla Link	49433
Version	unspecified
OS	Linux
Reporter	LLVM Bugzilla Contributor
CC	@JDevlieghere,@labath

Extended Description

All of the tests involving multiple breakpoints suffer from randomly establishing the wrong breakpoint hit count, e.g.:

FAIL: LLDB (/home/mgorny/llvm-project/build.arm64/bin/clang-aarch64) :: test (TestConcurrentTwoBreakpointThreads.ConcurrentTwoBreakpointThreads)

FAIL: test (TestConcurrentTwoBreakpointThreads.ConcurrentTwoBreakpointThreads)
Test two threads that trigger a breakpoint.

Traceback (most recent call last):
File "/home/mgorny/llvm-project/lldb/packages/Python/lldbsuite/test/decorators.py", line 149, in wrapper
return func(*args, **kwargs)
File "/home/mgorny/llvm-project/lldb/test/API/functionalities/thread/concurrent_events/TestConcurrentTwoBreakpointThreads.py", line 19, in test
self.do_thread_actions(num_breakpoint_threads=2)
File "/home/mgorny/llvm-project/lldb/packages/Python/lldbsuite/test/concurrent_base.py", line 266, in do_thread_actions
breakpoint_hit_count))
AssertionError: 2 != 3 : Expected 2 breakpoint hits, but got 3
Config=aarch64-/home/mgorny/llvm-project/build.arm64/bin/clang

It seems to be a problem in lldb/lldb-server logic somewhere that needs further debugging.

[labath]

Are you using any kind of virtualisation by any chance?

I can see happen this pretty reliably in VirtualBox, for instance. However, I haven't seen it happen on physical hardware in like, 4 years. So my money is on something in the virtualization stack eating some events...

[llvmbot]

Are you using any kind of virtualisation by any chance?

Yes, it's qemu-system-aarch64. Real hardware is impacted by bug #​48017.

I can see happen this pretty reliably in VirtualBox, for instance. However,
I haven't seen it happen on physical hardware in like, 4 years. So my money
is on something in the virtualization stack eating some events...

Well, here the event count is larger than it should be, so I don't think they're being eaten ;-). I suspect some event is wrongly attributed as the breakpoint hit.

[labath]

Ah, I missed the aarch64 part -- my bad.

I'm not that familiar with aarch64, so it could indeed be some architecture-specific thing (though the arm bots seem to be mostly fine).

[jasonmolenda]

For what it's worth, I've been looking into these same failures on Darwin systems and I believe I know what's going on. On AArch64 systems, an instruction that triggers a watchpoint is unrolled and we get the exception that it will trigger a watchpoint when it executes. If the selected thread (on a private stop) has a watchpoint reason, lldb disables the watchpoint, instruction steps the thread, re-enables the watchpoint, and shows the old/new values for write watchpoints. However, it doesn't do this to other threads that also hit a watchpoint at the same time. The problem is that the watchpoint hit-count is incremented (ignore-count decremented) at the point when we stop, for all threads that hit the watchpoint.

If 3 threads hit the same watchpoint simultaneously, hitcount += 3, we instruction-step one of them and report that thread as having hit a watchpoint. The other 2 threads (with debugserver anyway) lose their stop reason when we were instruction-stepping the first one (the Darwin kernel has delivered the exception and it's no longer posted for them I guess). lldb says that one thread has hit a watchpoint. Then you continue execution, and the two remaining threads -- which are sitting on an instruction that is about to hit the watchpoint -- will now hit it. hit-count += 2, we instruction step over one of them, then continue again and now hit-count += 1, and we instruction step the final watchpoint. Note that once we've finally got past this point, our hit-count is now 6 (3+2+1).

This is different than Intel where we receive the exception after the watchpoint instruction has completed.

Either we can delay modifying hit-count/ignore-count until we've completed instruction steps, or we can instruction step any thread that is on the initial watchpoint-hit exception when we stop. Jim prefers the latter, and so that's what we'll need to do to fix this. So when we hit 3 watchpoints simultaneously, we instruction step all 3 of them and then make it a public stop and report the modified values to the user, and hit-count will be 3.

[jasonmolenda]

Jim has put up a patch which I believe will address this https://reviews.llvm.org/D129814