[openmp] TEST 'libomp :: affinity/redetect.c' FAILED on s390x Ubuntu 23.10

[iii-i]

The test fails 100% of the time:

llvm-project/build$ bin/llvm-lit -sv projects/openmp/runtime/test/affinity/redetect.c
FAIL: libomp :: affinity/redetect.c (1 of 1)
******************** TEST 'libomp :: affinity/redetect.c' FAILED ********************
Exit Code: 1       

Command Output (stdout): 
--                   
# RUN: at line 1                                                                                         
llvm-project/build/./bin/clang -fopenmp   -I llvm-project/build/projects/openmp/runtime/src -I llvm-project/openmp/runtime/test -L llvm-project/build/./lib  -fno-omit-frame-pointer -mbackchain -I llvm-project/openmp/runtime/test/ompt llvm-project/openmp/runtime/test/affinity/redetect.c -o llvm-project/build/projects/openmp/runtime/test/affinity/Output/redetect.c.tmp -l
m -latomic
# executed command: llvm-project/build/./bin/clang -fopenmp -I llvm-project/build/projects/openmp/runtime/src -I llvm-project/openmp/runtime/test -L llvm-project/build/./lib -fno-omit-frame-pointer -mbackchain -I llvm-project/openmp/runtime/test/ompt llvm-project/openmp/runtime/test/affinity/redetect.c -o llvm-project/build/projects/openmp/runtime/test/affinity/Output/redetect.c.tmp -lm -latomic
# RUN: at line 2
llvm-project/build/projects/openmp/runtime/test/affinity/Output/redetect.c.tmp
# executed command: llvm-project/build/projects/openmp/runtime/test/affinity/Output/redetect.c.tmp
# .---command stdout------------
# | Orignal Mask: {0-59}
# | Hello! Thread 48 executed single region in parent process
# | Hello! Thread 51 executed single region in child process
# | Orignal Mask: {0-59}
# | Hello! Thread 48 executed single region in parent process
# `-----------------------------
# .---command stderr------------
# | Original Mask = {1-59}
# | New Mask = {0-59}
# | Child affinity mask did not reset properly
# `-----------------------------
# error: command failed with exit status: 1

[llvmbot]

@llvm/issue-subscribers-openmp

Author: Ilya Leoshkevich (iii-i)

The test fails 100% of the time:

llvm-project/build$ bin/llvm-lit -sv projects/openmp/runtime/test/affinity/redetect.c
FAIL: libomp :: affinity/redetect.c (1 of 1)
******************** TEST 'libomp :: affinity/redetect.c' FAILED ********************
Exit Code: 1       

Command Output (stdout): 
--                   
# RUN: at line 1                                                                                         
llvm-project/build/./bin/clang -fopenmp   -I llvm-project/build/projects/openmp/runtime/src -I llvm-project/openmp/runtime/test -L llvm-project/build/./lib  -fno-omit-frame-pointer -mbackchain -I llvm-project/openmp/runtime/test/ompt llvm-project/openmp/runtime/test/affinity/redetect.c -o llvm-project/build/projects/openmp/runtime/test/affinity/Output/redetect.c.tmp -l
m -latomic
# executed command: llvm-project/build/./bin/clang -fopenmp -I llvm-project/build/projects/openmp/runtime/src -I llvm-project/openmp/runtime/test -L llvm-project/build/./lib -fno-omit-frame-pointer -mbackchain -I llvm-project/openmp/runtime/test/ompt llvm-project/openmp/runtime/test/affinity/redetect.c -o llvm-project/build/projects/openmp/runtime/test/affinity/Output/redetect.c.tmp -lm -latomic
# RUN: at line 2
llvm-project/build/projects/openmp/runtime/test/affinity/Output/redetect.c.tmp
# executed command: llvm-project/build/projects/openmp/runtime/test/affinity/Output/redetect.c.tmp
# .---command stdout------------
# | Orignal Mask: {0-59}
# | Hello! Thread 48 executed single region in parent process
# | Hello! Thread 51 executed single region in child process
# | Orignal Mask: {0-59}
# | Hello! Thread 48 executed single region in parent process
# `-----------------------------
# .---command stderr------------
# | Original Mask = {1-59}
# | New Mask = {0-59}
# | Child affinity mask did not reset properly
# `-----------------------------
# error: command failed with exit status: 1

[uweigand]

Hi @iii-i I was so far unable to reproduce this anywhere. However, I just found an endian bug related to NUMA topology, which could possibly be related to this issue: #117073

Could you check whether this fixes this problem as well?

[iii-i]

I can still reproduce this on Ubuntu 24.04 running inside a Docker container.
LLVM commit 668f2c7.
Cherry-picking 77acb60 does not help.

[uweigand]

This turned out to be a problem with the host GCC miscompiling libomp, specifically the __kmp_atfork_child routine. To trigger the problem you need to use the host compiler to build libomp (i.e. configure using LLVM_ENABLE_PROJECTS="openmp" rather than LLVM_ENABLE_RUNTIMES="openmp"), and the host compiler needs to be a GCC version affected by the bug (certain versions of GCC 13 and 14), and you need to do a Release build with optimizations enabled. You also need to target a processor with vector support (z13 or later), which the Ubuntu system compiler does by default.

The GCC bug is reported here https://gcc.gnu.org/bugzilla/show_bug.cgi?id=116997 and has been fixed on all affected branches.