On Haswell: error: always_inline function '_mm512_setzero_si512' requires target feature 'evex512' while compiling nodejs 21.1.0

[ms178]

Trying to compile nodejs 21.1.0 with c0a7391, I see the following AVX-512 related errors, my Haswell-EP clearly doesn't support these instructions.

./deps/simdutf/simdutf.cpp:18171:26: error: always_inline function '_mm512_setzero_si512' requires target feature 'evex512', but would be inlined into function 'detect_encodings' that is compiled without support for 'evex512'
 18171 |     __m512i currentmax = _mm512_setzero_si512();
       |                          ^
../deps/simdutf/simdutf.cpp:18171:26: error: AVX vector return of type '__m512i' (vector of 8 'long long' values) without 'evex512' enabled changes the ABI
../deps/simdutf/simdutf.cpp:18173:20: error: always_inline function '_mm512_loadu_si512' requires target feature 'evex512', but would be inlined into function 'detect_encodings' that is compiled without support for 'evex512'
 18173 |       __m512i in = _mm512_loadu_si512((__m512i *)buf);
       |                    ^
../deps/simdutf/simdutf.cpp:18173:20: error: AVX vector return of type '__m512i' (vector of 8 'long long' values) without 'evex512' enabled changes the ABI
../deps/simdutf/simdutf.cpp:18174:43: error: always_inline function '_mm512_set1_epi16' requires target feature 'evex512', but would be inlined into function 'detect_encodings' that is compiled without support for 'evex512'
 18174 |       __m512i diff = _mm512_sub_epi16(in, _mm512_set1_epi16(uint16_t(0xD800)));
       |                                           ^
../deps/simdutf/simdutf.cpp:18174:43: error: AVX vector return of type '__m512i' (vector of 8 'long long' values) without 'evex512' enabled changes the ABI
../deps/simdutf/simdutf.cpp:18174:22: error: always_inline function '_mm512_sub_epi16' requires target feature 'evex512', but would be inlined into function 'detect_encodings' that is compiled without support for 'evex512'
 18174 |       __m512i diff = _mm512_sub_epi16(in, _mm512_set1_epi16(uint16_t(0xD800)));
       |                      ^
../deps/simdutf/simdutf.cpp:18174:22: error: AVX vector argument of type '__m512i' (vector of 8 'long long' values) without 'evex512' enabled changes the ABI
../deps/simdutf/simdutf.cpp:18176:11: error: '__builtin_ia32_ucmpw512_mask' needs target feature avx512bw,evex512
 18176 |           _mm512_cmplt_epu16_mask(diff, _mm512_set1_epi16(uint16_t(0x0800)));
       |           ^
/home/marcus/llvm18/lib/clang/18/include/avx512bwintrin.h:367:5: note: expanded from macro '_mm512_cmplt_epu16_mask'
  367 |     _mm512_cmp_epu16_mask((A), (B), _MM_CMPINT_LT)
      |     ^
/home/marcus/llvm18/lib/clang/18/include/avx512bwintrin.h:266:15: note: expanded from macro '_mm512_cmp_epu16_mask'
  266 |   ((__mmask32)__builtin_ia32_ucmpw512_mask((__v32hi)(__m512i)(a), \
      |               ^
../deps/simdutf/simdutf.cpp:18176:41: error: always_inline function '_mm512_set1_epi16' requires target feature 'evex512', but would be inlined into function 'detect_encodings' that is compiled without support for 'evex512'
 18176 |           _mm512_cmplt_epu16_mask(diff, _mm512_set1_epi16(uint16_t(0x0800)));
       |                                         ^
../deps/simdutf/simdutf.cpp:18176:41: error: AVX vector return of type '__m512i' (vector of 8 'long long' values) without 'evex512' enabled changes the ABI
../deps/simdutf/simdutf.cpp:18190:38: error: '__builtin_ia32_ucmpw512_mask' needs target feature avx512bw,evex512
 18190 |           __mmask32 highsurrogates = _mm512_cmplt_epu16_mask(
       |                                      ^
/home/marcus/llvm18/lib/clang/18/include/avx512bwintrin.h:367:5: note: expanded from macro '_mm512_cmplt_epu16_mask'
  367 |     _mm512_cmp_epu16_mask((A), (B), _MM_CMPINT_LT)
      |     ^
/home/marcus/llvm18/lib/clang/18/include/avx512bwintrin.h:266:15: note: expanded from macro '_mm512_cmp_epu16_mask'
  266 |   ((__mmask32)__builtin_ia32_ucmpw512_mask((__v32hi)(__m512i)(a), \
      |               ^
../deps/simdutf/simdutf.cpp:18191:21: error: always_inline function '_mm512_set1_epi16' requires target feature 'evex512', but would be inlined into function 'detect_encodings' that is compiled without support for 'evex512'
 18191 |               diff, _mm512_set1_epi16(uint16_t(0x0400)));
       |                     ^
../deps/simdutf/simdutf.cpp:18191:21: error: AVX vector return of type '__m512i' (vector of 8 'long long' values) without 'evex512' enabled changes the ABI
../deps/simdutf/simdutf.cpp:18234:20: error: always_inline function '_mm512_max_epu32' requires target feature 'evex512', but would be inlined into function 'detect_encodings' that is compiled without support for 'evex512'
 18234 |       currentmax = _mm512_max_epu32(in, currentmax);
       |                    ^

Flags used:

export CC=clang
export CXX=clang++
export CC_LD=lld
export CXX_LD=lld
export AR=llvm-ar
export NM=llvm-nm
export STRIP=llvm-strip
export OBJCOPY=llvm-objcopy
export OBJDUMP=llvm-objdump
export READELF=llvm-readelf
export RANLIB=llvm-ranlib
export HOSTCC=clang
export HOSTCXX=clang++
export HOSTAR=llvm-ar
export CPPFLAGS="-D_FORTIFY_SOURCE=0"
export CFLAGS="-O3 -march=native -mtune=native -maes -mbmi2 -mpclmul -mllvm -inline-threshold=1000 -mllvm -polly -mllvm -polly-position=early -mllvm -polly-dependences-computeout=900000000 -mllvm -polly-detect-profitability-min-per-loop-insts=40 -mllvm -polly-tiling=true -mllvm -polly-prevect-width=256 -mllvm -polly-vectorizer=stripmine -mllvm -polly-invariant-load-hoisting -mllvm -polly-loopfusion-greedy -mllvm -polly-run-inliner -mllvm -polly-run-dce -mllvm -polly-enable-delicm=true -mllvm -polly -fmerge-all-constants -mllvm -extra-vectorizer-passes -mllvm -enable-cond-stores-vec -mllvm -slp-vectorize-hor-store -mllvm -enable-loopinterchange -mllvm -enable-loop-distribute -mllvm -enable-unroll-and-jam -mllvm -enable-loop-flatten -mllvm -interleave-small-loop-scalar-reduction -mllvm -unroll-runtime-multi-exit -mllvm -aggressive-ext-opt -fno-math-errno -fno-trapping-math -falign-functions=32 -funroll-loops -fno-semantic-interposition -fcf-protection=none -mharden-sls=none -fomit-frame-pointer -mprefer-vector-width=256 -flto=thin -fwhole-program-vtables -fsplit-lto-unit -mllvm -adce-remove-loops -mllvm -enable-ext-tsp-block-placement=1 -mllvm -enable-gvn-hoist -mllvm -enable-dfa-jump-thread -Wno-error=backend-plugin -fgnuc-version=13.2.1 -ffp-contract=fast -fdata-sections -ffunction-sections -fno-unique-section-names -fsplit-machine-functions -freroll-loops"
export CXXFLAGS="${CFLAGS} -Wp,-U_GLIBCXX_ASSERTIONS"
export LDFLAGS="-Wl,--lto-CGO3 -Wl,--gc-sections -Wl,--icf=all -Wl,--lto-O3,-O3,-Bsymbolic-functions,--as-needed -fcf-protection=none -mharden-sls=none -Wl,-mllvm,-polly -Wl,-mllvm,-polly-invariant-load-hoisting -Wl,-mllvm -Wl,-polly-loopfusion-greedy -Wl,-mllvm -Wl,-polly-run-inliner -Wl,-mllvm -Wl,-polly-run-dce -Wl,-mllvm -Wl,-polly-enable-delicm=true -Wl,-mllvm -Wl,-extra-vectorizer-passes -Wl,-mllvm -Wl,-enable-cond-stores-vec -Wl,-mllvm -Wl,-slp-vectorize-hor-store -Wl,-mllvm -Wl,-enable-loopinterchange -Wl,-mllvm -Wl,-enable-loop-distribute -Wl,-mllvm -Wl,-enable-unroll-and-jam -Wl,-mllvm -Wl,-enable-loop-flatten -Wl,-mllvm -Wl,-interleave-small-loop-scalar-reduction -Wl,-mllvm -Wl,-unroll-runtime-multi-exit -Wl,-mllvm -Wl,-aggressive-ext-opt -Wl,-mllvm -Wl,-enable-interleaved-mem-accesses -Wl,-mllvm -Wl,-enable-masked-interleaved-mem-accesses -march=native -maes -mbmi2 -mpclmul -flto=thin -fwhole-program-vtables -fuse-ld=lld -Wl,-zmax-page-size=0x200000 -freroll-loops -Wl,-mllvm -Wl,-adce-remove-loops -Wl,-mllvm -Wl,-enable-ext-tsp-block-placement=1 -Wl,-mllvm -Wl,-enable-gvn-hoist -Wl,-mllvm -Wl,-enable-dfa-jump-thread -Wl,--push-state -Wl,-whole-archive -ljemalloc_pic -Wl,--pop-state -lpthread -lstdc++ -lm -ldl -Wl,--undefined-version"
CCLDFLAGS="$LDFLAGS"
CXXLDFLAGS="$LDFLAGS"
export ASFLAGS="-D__AVX__=1 -D__AVX2__=1 -msse2avx -D__FMA__=1"

[edit: Removed the PGO-profile from the flags, it is not needed to re-produce the issue]

[ms178]

Pinging @phoebewang for awareness.

[phoebewang]

Thanks @ms178 for reporting the issue. Could you provide a single file with compile command? You can use -E to precompile headers to the file.
I also have a pr #71318 to solve some known issues. You may have a try with it when it's landed if you are not in hurry.

[llvmbot]

@llvm/issue-subscribers-backend-x86

Author: None (ms178)

Trying to compile nodejs 21.1.0 with c0a7391, I see the following AVX-512 related errors, my Haswell-EP clearly doesn't support these instructions.

./deps/simdutf/simdutf.cpp:18171:26: error: always_inline function '_mm512_setzero_si512' requires target feature 'evex512', but would be inlined into function 'detect_encodings' that is compiled without support for 'evex512'
 18171 |     __m512i currentmax = _mm512_setzero_si512();
       |                          ^
../deps/simdutf/simdutf.cpp:18171:26: error: AVX vector return of type '__m512i' (vector of 8 'long long' values) without 'evex512' enabled changes the ABI
../deps/simdutf/simdutf.cpp:18173:20: error: always_inline function '_mm512_loadu_si512' requires target feature 'evex512', but would be inlined into function 'detect_encodings' that is compiled without support for 'evex512'
 18173 |       __m512i in = _mm512_loadu_si512((__m512i *)buf);
       |                    ^
../deps/simdutf/simdutf.cpp:18173:20: error: AVX vector return of type '__m512i' (vector of 8 'long long' values) without 'evex512' enabled changes the ABI
../deps/simdutf/simdutf.cpp:18174:43: error: always_inline function '_mm512_set1_epi16' requires target feature 'evex512', but would be inlined into function 'detect_encodings' that is compiled without support for 'evex512'
 18174 |       __m512i diff = _mm512_sub_epi16(in, _mm512_set1_epi16(uint16_t(0xD800)));
       |                                           ^
../deps/simdutf/simdutf.cpp:18174:43: error: AVX vector return of type '__m512i' (vector of 8 'long long' values) without 'evex512' enabled changes the ABI
../deps/simdutf/simdutf.cpp:18174:22: error: always_inline function '_mm512_sub_epi16' requires target feature 'evex512', but would be inlined into function 'detect_encodings' that is compiled without support for 'evex512'
 18174 |       __m512i diff = _mm512_sub_epi16(in, _mm512_set1_epi16(uint16_t(0xD800)));
       |                      ^
../deps/simdutf/simdutf.cpp:18174:22: error: AVX vector argument of type '__m512i' (vector of 8 'long long' values) without 'evex512' enabled changes the ABI
../deps/simdutf/simdutf.cpp:18176:11: error: '__builtin_ia32_ucmpw512_mask' needs target feature avx512bw,evex512
 18176 |           _mm512_cmplt_epu16_mask(diff, _mm512_set1_epi16(uint16_t(0x0800)));
       |           ^
/home/marcus/llvm18/lib/clang/18/include/avx512bwintrin.h:367:5: note: expanded from macro '_mm512_cmplt_epu16_mask'
  367 |     _mm512_cmp_epu16_mask((A), (B), _MM_CMPINT_LT)
      |     ^
/home/marcus/llvm18/lib/clang/18/include/avx512bwintrin.h:266:15: note: expanded from macro '_mm512_cmp_epu16_mask'
  266 |   ((__mmask32)__builtin_ia32_ucmpw512_mask((__v32hi)(__m512i)(a), \
      |               ^
../deps/simdutf/simdutf.cpp:18176:41: error: always_inline function '_mm512_set1_epi16' requires target feature 'evex512', but would be inlined into function 'detect_encodings' that is compiled without support for 'evex512'
 18176 |           _mm512_cmplt_epu16_mask(diff, _mm512_set1_epi16(uint16_t(0x0800)));
       |                                         ^
../deps/simdutf/simdutf.cpp:18176:41: error: AVX vector return of type '__m512i' (vector of 8 'long long' values) without 'evex512' enabled changes the ABI
../deps/simdutf/simdutf.cpp:18190:38: error: '__builtin_ia32_ucmpw512_mask' needs target feature avx512bw,evex512
 18190 |           __mmask32 highsurrogates = _mm512_cmplt_epu16_mask(
       |                                      ^
/home/marcus/llvm18/lib/clang/18/include/avx512bwintrin.h:367:5: note: expanded from macro '_mm512_cmplt_epu16_mask'
  367 |     _mm512_cmp_epu16_mask((A), (B), _MM_CMPINT_LT)
      |     ^
/home/marcus/llvm18/lib/clang/18/include/avx512bwintrin.h:266:15: note: expanded from macro '_mm512_cmp_epu16_mask'
  266 |   ((__mmask32)__builtin_ia32_ucmpw512_mask((__v32hi)(__m512i)(a), \
      |               ^
../deps/simdutf/simdutf.cpp:18191:21: error: always_inline function '_mm512_set1_epi16' requires target feature 'evex512', but would be inlined into function 'detect_encodings' that is compiled without support for 'evex512'
 18191 |               diff, _mm512_set1_epi16(uint16_t(0x0400)));
       |                     ^
../deps/simdutf/simdutf.cpp:18191:21: error: AVX vector return of type '__m512i' (vector of 8 'long long' values) without 'evex512' enabled changes the ABI
../deps/simdutf/simdutf.cpp:18234:20: error: always_inline function '_mm512_max_epu32' requires target feature 'evex512', but would be inlined into function 'detect_encodings' that is compiled without support for 'evex512'
 18234 |       currentmax = _mm512_max_epu32(in, currentmax);
       |                    ^

Flags used:

export CC=clang
export CXX=clang++
export CC_LD=lld
export CXX_LD=lld
export AR=llvm-ar
export NM=llvm-nm
export STRIP=llvm-strip
export OBJCOPY=llvm-objcopy
export OBJDUMP=llvm-objdump
export READELF=llvm-readelf
export RANLIB=llvm-ranlib
export HOSTCC=clang
export HOSTCXX=clang++
export HOSTAR=llvm-ar
export CPPFLAGS="-D_FORTIFY_SOURCE=0"
export CFLAGS="-O3 -march=native -mtune=native -maes -mbmi2 -mpclmul -mllvm -inline-threshold=1000 -mllvm -polly -mllvm -polly-position=early -mllvm -polly-dependences-computeout=900000000 -mllvm -polly-detect-profitability-min-per-loop-insts=40 -mllvm -polly-tiling=true -mllvm -polly-prevect-width=256 -mllvm -polly-vectorizer=stripmine -mllvm -polly-invariant-load-hoisting -mllvm -polly-loopfusion-greedy -mllvm -polly-run-inliner -mllvm -polly-run-dce -mllvm -polly-enable-delicm=true -mllvm -polly -fmerge-all-constants -mllvm -extra-vectorizer-passes -mllvm -enable-cond-stores-vec -mllvm -slp-vectorize-hor-store -mllvm -enable-loopinterchange -mllvm -enable-loop-distribute -mllvm -enable-unroll-and-jam -mllvm -enable-loop-flatten -mllvm -interleave-small-loop-scalar-reduction -mllvm -unroll-runtime-multi-exit -mllvm -aggressive-ext-opt -fno-math-errno -fno-trapping-math -falign-functions=32 -funroll-loops -fno-semantic-interposition -fcf-protection=none -mharden-sls=none -fomit-frame-pointer -mprefer-vector-width=256 -flto=thin -fwhole-program-vtables -fsplit-lto-unit -mllvm -adce-remove-loops -mllvm -enable-ext-tsp-block-placement=1 -mllvm -enable-gvn-hoist -mllvm -enable-dfa-jump-thread -Wno-error=backend-plugin -fgnuc-version=13.2.1 -fprofile-use=/home/marcus/Downloads/all.profdata -ffp-contract=fast -fdata-sections -ffunction-sections -fno-unique-section-names -fsplit-machine-functions -freroll-loops"
export CXXFLAGS="${CFLAGS} -Wp,-U_GLIBCXX_ASSERTIONS"
export LDFLAGS="-Wl,--lto-CGO3 -Wl,--gc-sections -Wl,--icf=all -Wl,--lto-O3,-O3,-Bsymbolic-functions,--as-needed -fcf-protection=none -mharden-sls=none -Wl,-mllvm,-polly -Wl,-mllvm,-polly-invariant-load-hoisting -Wl,-mllvm -Wl,-polly-loopfusion-greedy -Wl,-mllvm -Wl,-polly-run-inliner -Wl,-mllvm -Wl,-polly-run-dce -Wl,-mllvm -Wl,-polly-enable-delicm=true -Wl,-mllvm -Wl,-extra-vectorizer-passes -Wl,-mllvm -Wl,-enable-cond-stores-vec -Wl,-mllvm -Wl,-slp-vectorize-hor-store -Wl,-mllvm -Wl,-enable-loopinterchange -Wl,-mllvm -Wl,-enable-loop-distribute -Wl,-mllvm -Wl,-enable-unroll-and-jam -Wl,-mllvm -Wl,-enable-loop-flatten -Wl,-mllvm -Wl,-interleave-small-loop-scalar-reduction -Wl,-mllvm -Wl,-unroll-runtime-multi-exit -Wl,-mllvm -Wl,-aggressive-ext-opt -Wl,-mllvm -Wl,-enable-interleaved-mem-accesses -Wl,-mllvm -Wl,-enable-masked-interleaved-mem-accesses -march=native -maes -mbmi2 -mpclmul -flto=thin -fwhole-program-vtables -fuse-ld=lld -Wl,-zmax-page-size=0x200000 -fprofile-use=/home/marcus/Downloads/all.profdata -freroll-loops -Wl,-mllvm -Wl,-adce-remove-loops -Wl,-mllvm -Wl,-enable-ext-tsp-block-placement=1 -Wl,-mllvm -Wl,-enable-gvn-hoist -Wl,-mllvm -Wl,-enable-dfa-jump-thread -Wl,--push-state -Wl,-whole-archive -ljemalloc_pic -Wl,--pop-state -lpthread -lstdc++ -lm -ldl -Wl,--undefined-version"
CCLDFLAGS="$LDFLAGS"
CXXLDFLAGS="$LDFLAGS"
export ASFLAGS="-D__AVX__=1 -D__AVX2__=1 -msse2avx -D__FMA__=1"

[ms178]

Thanks for taking a look! As I am not a developer, I kindly ask for more guidance as to what to do next. From the console output I get, the offending file is either:
simdutf.cpp.txt

or

cached-powers.o.txt

With the following PKGBUILD (and rebuild.list file in the same directory), the seen error should re-produce easily on any Arch-based distribution using makepkg -si --cleanbuild --skippgpchecks --skippgpchecksums when using the mentioned Clang-revision.

PKGBUILD.txt
rebuild.list.txt

[phoebewang]

Sorry, I cannot confirm the problem from the source file and I don't know the mechanism how it been built.

Here are some guesses:

1. The -march=native with a non AVX512 target cannot compile code that calls to 512-bit intriniscs.
2. I see the error was reported about calling _mm512_setzero_si512, the code should not be built in this case.
3. I see the code is in icelake namespace. So it looks like the code should not be built on non icelake target.

In a word, I suspect something wrong with the code or build command. I don't see anything wrong in the compiler.

[ms178]

One important observation: If I exchange -march=native with -march=haswell in my CFLAGS (and the same change for the mtune flag), the compilation runs fine.

[ms178]

Skimming through past committs, I found: 8a58407

There might have been other changes as AVX-512 code was worked on recently a lot. I wonder why AVX-512 instructions still get used when -march=native -detection should detect that the CPU target does not support AVX-512.

[ms178]

The distro's default Clang 16.0.6 had no such issue with -march=native with the new nodejs version.

[phoebewang]

I wonder why AVX-512 instructions still get used when -march=native -detection should detect that the CPU target does not support AVX-512.

I have the same question. Usually, when writing code calling to AVX512 intrinisics, there will be a detection mechanism provided together to make sure it won't be built for non AVX512 target. I didn't figure out it by just reading the single file.

It's probably a bug in 8a58407 exposed with the detection mechanism. But I need more code to analyze it.

Here are ways I can think to get it:

1. Try clang -c -march=native simdutf.cpp to see if the same error can be reprodued with it. You may need to use -IPATH to specify the path of header files;
2. If 1) works, use clang -E simdutf.cpp > simdutf.i to precompile dependent headers into the single file;
3. Verify if the the same error can be reprodued with clang -c -march=native simdutf.i. Upload simdutf.i if it can be reproduced.

[ms178]

Here are the related files which also reproduce the issue when using -march=native on my Haswell-EP with the mentioned Clang-18 revision:

simdutf.h.txt
simdutf.i.txt
simdutf.cpp.txt

[phoebewang]

@ms178 Thanks for the files. I have root caused the problem that can be shown in this simple case https://godbolt.org/z/5Tcqd5KMo

It's caused by a bug in 24194090e17b
I have checked this issue cannot be reproducible with the new design #71318. I'll try to land it ASAP.

[ms178]

As the mentioned fix has landed, I am closing this one.