Skip to content

[SPIRV][RFC] Rework / extend support for memory scopes #106429

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 28 commits into from
Sep 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
d41faf6
Tweak AMDGCNSPIRV ABI to allow for the correct handling of aggregates…
AlexVlx Aug 10, 2024
757e119
Fix formatting error.
AlexVlx Aug 10, 2024
af1a416
Merge branch 'main' of https://github.com/llvm/llvm-project into amdg…
AlexVlx Aug 13, 2024
0cb89f6
Merge branch 'main' of https://github.com/llvm/llvm-project into amdg…
AlexVlx Aug 19, 2024
11162b0
Merge branch 'main' of https://github.com/llvm/llvm-project into amdg…
AlexVlx Aug 19, 2024
13f83ac
No else after return.
AlexVlx Aug 19, 2024
fe68593
Merge branch 'main' of https://github.com/llvm/llvm-project into amdg…
AlexVlx Aug 21, 2024
e466a56
Merge branch 'main' of https://github.com/llvm/llvm-project into amdg…
AlexVlx Aug 23, 2024
734630a
Merge branch 'main' of https://github.com/llvm/llvm-project into amdg…
AlexVlx Aug 23, 2024
1637876
Merge branch 'main' of https://github.com/llvm/llvm-project into amdg…
AlexVlx Aug 25, 2024
1d3fedb
Merge branch 'main' of https://github.com/llvm/llvm-project into amdg…
AlexVlx Aug 27, 2024
440a0ef
Merge branch 'main' of https://github.com/llvm/llvm-project into amdg…
AlexVlx Aug 28, 2024
daa76c3
Re-work SPIR-V support for memory scopes.
AlexVlx Aug 28, 2024
25378a7
Fix formatting.
AlexVlx Aug 28, 2024
fc422b1
Merge branch 'main' of https://github.com/llvm/llvm-project into amdg…
AlexVlx Aug 28, 2024
b6fd508
Merge branch 'main' of https://github.com/llvm/llvm-project into amdg…
AlexVlx Sep 4, 2024
79acf40
Incorporate review feedback.
AlexVlx Sep 4, 2024
bc712a3
Merge branch 'main' of https://github.com/llvm/llvm-project into amdg…
AlexVlx Sep 18, 2024
e984939
No need for aliases / special handling of System scope.
AlexVlx Sep 18, 2024
ced6877
Remove & replace SyncScopeIDs struct.
AlexVlx Sep 18, 2024
ec0eb50
Fix formatting.
AlexVlx Sep 18, 2024
8621e36
Merge branch 'main' of https://github.com/llvm/llvm-project into amdg…
AlexVlx Sep 18, 2024
9f87c0f
Merge branch 'main' of https://github.com/llvm/llvm-project into amdg…
AlexVlx Sep 20, 2024
e2f72fb
Incorporate review feedback.
AlexVlx Sep 20, 2024
96a79e7
Try to work around confusing Scope re-use.
AlexVlx Sep 20, 2024
75776cc
Merge branch 'main' of https://github.com/llvm/llvm-project into amdg…
AlexVlx Sep 24, 2024
92f739c
Add test for OCL defaulting to `device` scope.
AlexVlx Sep 24, 2024
2cb4b46
Merge branch 'main' of https://github.com/llvm/llvm-project into amdg…
AlexVlx Sep 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions clang/lib/Basic/Targets/SPIR.h
Original file line number Diff line number Diff line change
Expand Up @@ -335,6 +335,9 @@ class LLVM_LIBRARY_VISIBILITY SPIRV32TargetInfo : public BaseSPIRVTargetInfo {
PointerWidth = PointerAlign = 32;
SizeType = TargetInfo::UnsignedInt;
PtrDiffType = IntPtrType = TargetInfo::SignedInt;
// SPIR-V has core support for atomic ops, and Int32 is always available;
// we take the maximum because it's possible the Host supports wider types.
MaxAtomicInlineWidth = std::max<unsigned char>(MaxAtomicInlineWidth, 32);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't there a 64-bit atomic extension? How are extensions supposed to work here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm assuming that the SPIRV32 target exists for cases where the Int64 capability is never enabled, but it would probably be useful to have that assumption checked. For SPIR-V the model for extensions / capabilities in LLVM seems to be push i.e. extensions get enabled / checked iff a feature requiring the extension / capability is encountered when translating (legacy) / lowering (the experimental BE). FWIW, my reading of the SPIR-V spec is that the Int64 capability is core.

resetDataLayout("e-p:32:32-i64:64-v16:16-v24:32-v32:32-v48:64-"
"v96:128-v192:256-v256:256-v512:512-v1024:1024-G1");
}
Expand All @@ -356,6 +359,9 @@ class LLVM_LIBRARY_VISIBILITY SPIRV64TargetInfo : public BaseSPIRVTargetInfo {
PointerWidth = PointerAlign = 64;
SizeType = TargetInfo::UnsignedLong;
PtrDiffType = IntPtrType = TargetInfo::SignedLong;
// SPIR-V has core support for atomic ops, and Int64 is always available;
// we take the maximum because it's possible the Host supports wider types.
MaxAtomicInlineWidth = std::max<unsigned char>(MaxAtomicInlineWidth, 64);
resetDataLayout("e-i64:64-v16:16-v24:32-v32:32-v48:64-"
"v96:128-v192:256-v256:256-v512:512-v1024:1024-G1");
}
Expand Down
13 changes: 12 additions & 1 deletion clang/lib/CodeGen/CGAtomic.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -766,8 +766,19 @@ static void EmitAtomicOp(CodeGenFunction &CGF, AtomicExpr *Expr, Address Dest,
// LLVM atomic instructions always have synch scope. If clang atomic
// expression has no scope operand, use default LLVM synch scope.
if (!ScopeModel) {
llvm::SyncScope::ID SS;
if (CGF.getLangOpts().OpenCL)
// OpenCL approach is: "The functions that do not have memory_scope
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which flavor of atomic operations does this function correspond to?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the primary entry point for Atomic emission, so things like the Clang builtins (which do not carry scopes) would end up here.

// argument have the same semantics as the corresponding functions with
// the memory_scope argument set to memory_scope_device." See ref.:
// https://registry.khronos.org/OpenCL/specs/3.0-unified/html/OpenCL_C.html#atomic-functions
SS = CGF.getTargetHooks().getLLVMSyncScopeID(CGF.getLangOpts(),
SyncScope::OpenCLDevice,
Order, CGF.getLLVMContext());
else
SS = llvm::SyncScope::System;
EmitAtomicOp(CGF, Expr, Dest, Ptr, Val1, Val2, IsWeak, FailureOrder, Size,
Order, CGF.CGM.getLLVMContext().getOrInsertSyncScopeID(""));
Order, SS);
return;
}

Expand Down
36 changes: 36 additions & 0 deletions clang/lib/CodeGen/Targets/SPIR.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,36 @@ class SPIRVTargetCodeGenInfo : public CommonSPIRTargetCodeGenInfo {
SPIRVTargetCodeGenInfo(CodeGen::CodeGenTypes &CGT)
: CommonSPIRTargetCodeGenInfo(std::make_unique<SPIRVABIInfo>(CGT)) {}
void setCUDAKernelCallingConvention(const FunctionType *&FT) const override;
llvm::SyncScope::ID getLLVMSyncScopeID(const LangOptions &LangOpts,
SyncScope Scope,
llvm::AtomicOrdering Ordering,
llvm::LLVMContext &Ctx) const override;
};

inline StringRef mapClangSyncScopeToLLVM(SyncScope Scope) {
switch (Scope) {
case SyncScope::HIPSingleThread:
case SyncScope::SingleScope:
return "singlethread";
case SyncScope::HIPWavefront:
case SyncScope::OpenCLSubGroup:
case SyncScope::WavefrontScope:
return "subgroup";
case SyncScope::HIPWorkgroup:
case SyncScope::OpenCLWorkGroup:
case SyncScope::WorkgroupScope:
return "workgroup";
case SyncScope::HIPAgent:
case SyncScope::OpenCLDevice:
case SyncScope::DeviceScope:
return "device";
case SyncScope::SystemScope:
case SyncScope::HIPSystem:
case SyncScope::OpenCLAllSVMDevices:
return "";
}
return "";
}
} // End anonymous namespace.

void CommonSPIRABIInfo::setCCs() {
Expand Down Expand Up @@ -188,6 +217,13 @@ void SPIRVTargetCodeGenInfo::setCUDAKernelCallingConvention(
}
}

llvm::SyncScope::ID
SPIRVTargetCodeGenInfo::getLLVMSyncScopeID(const LangOptions &, SyncScope Scope,
llvm::AtomicOrdering,
llvm::LLVMContext &Ctx) const {
return Ctx.getOrInsertSyncScopeID(mapClangSyncScopeToLLVM(Scope));
}

/// Construct a SPIR-V target extension type for the given OpenCL image type.
static llvm::Type *getSPIRVImageType(llvm::LLVMContext &Ctx, StringRef BaseType,
StringRef OpenCLName,
Expand Down
336 changes: 223 additions & 113 deletions clang/test/CodeGen/scoped-atomic-ops.c

Large diffs are not rendered by default.

235 changes: 235 additions & 0 deletions clang/test/CodeGenOpenCL/atomic-builtins-default-to-device-scope.cl
Original file line number Diff line number Diff line change
@@ -0,0 +1,235 @@
// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py UTC_ARGS: --version 5
// RUN: %clang_cc1 %s -cl-std=CL2.0 -emit-llvm -O3 -o - -triple=amdgcn-amd-amdhsa \
// RUN: | FileCheck %s --check-prefix=AMDGCN
// RUN: %clang_cc1 %s -cl-std=CL2.0 -emit-llvm -O3 -o - -triple=spirv64-unknown-unknown \
// RUN: | FileCheck %s --check-prefix=SPIRV

// AMDGCN-LABEL: define dso_local i32 @load(
// AMDGCN-SAME: ptr nocapture noundef readonly [[P:%.*]]) local_unnamed_addr #[[ATTR0:[0-9]+]] {
// AMDGCN-NEXT: [[ENTRY:.*:]]
// AMDGCN-NEXT: [[TMP0:%.*]] = load atomic i32, ptr [[P]] syncscope("agent") seq_cst, align 4
// AMDGCN-NEXT: ret i32 [[TMP0]]
//
// SPIRV-LABEL: define spir_func i32 @load(
// SPIRV-SAME: ptr addrspace(4) nocapture noundef readonly [[P:%.*]]) local_unnamed_addr #[[ATTR0:[0-9]+]] {
// SPIRV-NEXT: [[ENTRY:.*:]]
// SPIRV-NEXT: [[TMP0:%.*]] = load atomic i32, ptr addrspace(4) [[P]] syncscope("device") seq_cst, align 4
// SPIRV-NEXT: ret i32 [[TMP0]]
//
int load(int *p) { return __atomic_load_n(p, __ATOMIC_SEQ_CST); }
// AMDGCN-LABEL: define dso_local void @store(
// AMDGCN-SAME: ptr nocapture noundef writeonly [[P:%.*]], i32 noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// AMDGCN-NEXT: [[ENTRY:.*:]]
// AMDGCN-NEXT: store atomic i32 [[X]], ptr [[P]] syncscope("agent") seq_cst, align 4
// AMDGCN-NEXT: ret void
//
// SPIRV-LABEL: define spir_func void @store(
// SPIRV-SAME: ptr addrspace(4) nocapture noundef writeonly [[P:%.*]], i32 noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// SPIRV-NEXT: [[ENTRY:.*:]]
// SPIRV-NEXT: store atomic i32 [[X]], ptr addrspace(4) [[P]] syncscope("device") seq_cst, align 4
// SPIRV-NEXT: ret void
//
void store(int *p, int x) { return __atomic_store_n(p, x, __ATOMIC_SEQ_CST); }
// AMDGCN-LABEL: define dso_local i32 @add(
// AMDGCN-SAME: ptr nocapture noundef [[P:%.*]], i32 noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// AMDGCN-NEXT: [[ENTRY:.*:]]
// AMDGCN-NEXT: [[TMP0:%.*]] = atomicrmw add ptr [[P]], i32 [[X]] syncscope("agent") seq_cst, align 4
// AMDGCN-NEXT: ret i32 [[TMP0]]
//
// SPIRV-LABEL: define spir_func i32 @add(
// SPIRV-SAME: ptr addrspace(4) nocapture noundef [[P:%.*]], i32 noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// SPIRV-NEXT: [[ENTRY:.*:]]
// SPIRV-NEXT: [[TMP0:%.*]] = atomicrmw add ptr addrspace(4) [[P]], i32 [[X]] syncscope("device") seq_cst, align 4
// SPIRV-NEXT: ret i32 [[TMP0]]
//
int add(int *p, int x) { return __atomic_fetch_add(p, x, __ATOMIC_SEQ_CST); }
// AMDGCN-LABEL: define dso_local float @fadd(
// AMDGCN-SAME: ptr nocapture noundef [[P:%.*]], float noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// AMDGCN-NEXT: [[ENTRY:.*:]]
// AMDGCN-NEXT: [[TMP0:%.*]] = atomicrmw fadd ptr [[P]], float [[X]] syncscope("agent") seq_cst, align 4
// AMDGCN-NEXT: ret float [[TMP0]]
//
// SPIRV-LABEL: define spir_func float @fadd(
// SPIRV-SAME: ptr addrspace(4) nocapture noundef [[P:%.*]], float noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// SPIRV-NEXT: [[ENTRY:.*:]]
// SPIRV-NEXT: [[TMP0:%.*]] = atomicrmw fadd ptr addrspace(4) [[P]], float [[X]] syncscope("device") seq_cst, align 4
// SPIRV-NEXT: ret float [[TMP0]]
//
float fadd(float *p, float x) { return __atomic_fetch_add(p, x, __ATOMIC_SEQ_CST); }
// AMDGCN-LABEL: define dso_local i32 @sub(
// AMDGCN-SAME: ptr nocapture noundef [[P:%.*]], i32 noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// AMDGCN-NEXT: [[ENTRY:.*:]]
// AMDGCN-NEXT: [[TMP0:%.*]] = atomicrmw sub ptr [[P]], i32 [[X]] syncscope("agent") seq_cst, align 4
// AMDGCN-NEXT: ret i32 [[TMP0]]
//
// SPIRV-LABEL: define spir_func i32 @sub(
// SPIRV-SAME: ptr addrspace(4) nocapture noundef [[P:%.*]], i32 noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// SPIRV-NEXT: [[ENTRY:.*:]]
// SPIRV-NEXT: [[TMP0:%.*]] = atomicrmw sub ptr addrspace(4) [[P]], i32 [[X]] syncscope("device") seq_cst, align 4
// SPIRV-NEXT: ret i32 [[TMP0]]
//
int sub(int *p, int x) { return __atomic_fetch_sub(p, x, __ATOMIC_SEQ_CST); }
// AMDGCN-LABEL: define dso_local float @fsub(
// AMDGCN-SAME: ptr nocapture noundef [[P:%.*]], float noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// AMDGCN-NEXT: [[ENTRY:.*:]]
// AMDGCN-NEXT: [[TMP0:%.*]] = atomicrmw fsub ptr [[P]], float [[X]] syncscope("agent") seq_cst, align 4
// AMDGCN-NEXT: ret float [[TMP0]]
//
// SPIRV-LABEL: define spir_func float @fsub(
// SPIRV-SAME: ptr addrspace(4) nocapture noundef [[P:%.*]], float noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// SPIRV-NEXT: [[ENTRY:.*:]]
// SPIRV-NEXT: [[TMP0:%.*]] = atomicrmw fsub ptr addrspace(4) [[P]], float [[X]] syncscope("device") seq_cst, align 4
// SPIRV-NEXT: ret float [[TMP0]]
//
float fsub(float *p, float x) { return __atomic_fetch_sub(p, x, __ATOMIC_SEQ_CST); }
// AMDGCN-LABEL: define dso_local i32 @and(
// AMDGCN-SAME: ptr nocapture noundef [[P:%.*]], i32 noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// AMDGCN-NEXT: [[ENTRY:.*:]]
// AMDGCN-NEXT: [[TMP0:%.*]] = atomicrmw and ptr [[P]], i32 [[X]] syncscope("agent") seq_cst, align 4
// AMDGCN-NEXT: ret i32 [[TMP0]]
//
// SPIRV-LABEL: define spir_func i32 @and(
// SPIRV-SAME: ptr addrspace(4) nocapture noundef [[P:%.*]], i32 noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// SPIRV-NEXT: [[ENTRY:.*:]]
// SPIRV-NEXT: [[TMP0:%.*]] = atomicrmw and ptr addrspace(4) [[P]], i32 [[X]] syncscope("device") seq_cst, align 4
// SPIRV-NEXT: ret i32 [[TMP0]]
//
int and(int *p, int x) { return __atomic_fetch_and(p, x, __ATOMIC_SEQ_CST); }
// AMDGCN-LABEL: define dso_local i32 @nand(
// AMDGCN-SAME: ptr nocapture noundef [[P:%.*]], i32 noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// AMDGCN-NEXT: [[ENTRY:.*:]]
// AMDGCN-NEXT: [[TMP0:%.*]] = atomicrmw nand ptr [[P]], i32 [[X]] syncscope("agent") seq_cst, align 4
// AMDGCN-NEXT: ret i32 [[TMP0]]
//
// SPIRV-LABEL: define spir_func i32 @nand(
// SPIRV-SAME: ptr addrspace(4) nocapture noundef [[P:%.*]], i32 noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// SPIRV-NEXT: [[ENTRY:.*:]]
// SPIRV-NEXT: [[TMP0:%.*]] = atomicrmw nand ptr addrspace(4) [[P]], i32 [[X]] syncscope("device") seq_cst, align 4
// SPIRV-NEXT: ret i32 [[TMP0]]
//
int nand(int *p, int x) { return __atomic_fetch_nand(p, x, __ATOMIC_SEQ_CST); }
// AMDGCN-LABEL: define dso_local i32 @or(
// AMDGCN-SAME: ptr nocapture noundef [[P:%.*]], i32 noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// AMDGCN-NEXT: [[ENTRY:.*:]]
// AMDGCN-NEXT: [[TMP0:%.*]] = atomicrmw or ptr [[P]], i32 [[X]] syncscope("agent") seq_cst, align 4
// AMDGCN-NEXT: ret i32 [[TMP0]]
//
// SPIRV-LABEL: define spir_func i32 @or(
// SPIRV-SAME: ptr addrspace(4) nocapture noundef [[P:%.*]], i32 noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// SPIRV-NEXT: [[ENTRY:.*:]]
// SPIRV-NEXT: [[TMP0:%.*]] = atomicrmw or ptr addrspace(4) [[P]], i32 [[X]] syncscope("device") seq_cst, align 4
// SPIRV-NEXT: ret i32 [[TMP0]]
//
int or(int *p, int x) { return __atomic_fetch_or(p, x, __ATOMIC_SEQ_CST); }
// AMDGCN-LABEL: define dso_local i32 @xor(
// AMDGCN-SAME: ptr nocapture noundef [[P:%.*]], i32 noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// AMDGCN-NEXT: [[ENTRY:.*:]]
// AMDGCN-NEXT: [[TMP0:%.*]] = atomicrmw xor ptr [[P]], i32 [[X]] syncscope("agent") seq_cst, align 4
// AMDGCN-NEXT: ret i32 [[TMP0]]
//
// SPIRV-LABEL: define spir_func i32 @xor(
// SPIRV-SAME: ptr addrspace(4) nocapture noundef [[P:%.*]], i32 noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// SPIRV-NEXT: [[ENTRY:.*:]]
// SPIRV-NEXT: [[TMP0:%.*]] = atomicrmw xor ptr addrspace(4) [[P]], i32 [[X]] syncscope("device") seq_cst, align 4
// SPIRV-NEXT: ret i32 [[TMP0]]
//
int xor(int *p, int x) { return __atomic_fetch_xor(p, x, __ATOMIC_SEQ_CST); }
// AMDGCN-LABEL: define dso_local i32 @min(
// AMDGCN-SAME: ptr nocapture noundef [[P:%.*]], i32 noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// AMDGCN-NEXT: [[ENTRY:.*:]]
// AMDGCN-NEXT: [[TMP0:%.*]] = atomicrmw min ptr [[P]], i32 [[X]] syncscope("agent") seq_cst, align 4
// AMDGCN-NEXT: ret i32 [[TMP0]]
//
// SPIRV-LABEL: define spir_func i32 @min(
// SPIRV-SAME: ptr addrspace(4) nocapture noundef [[P:%.*]], i32 noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// SPIRV-NEXT: [[ENTRY:.*:]]
// SPIRV-NEXT: [[TMP0:%.*]] = atomicrmw min ptr addrspace(4) [[P]], i32 [[X]] syncscope("device") seq_cst, align 4
// SPIRV-NEXT: ret i32 [[TMP0]]
//
int min(int *p, int x) { return __atomic_fetch_min(p, x, __ATOMIC_SEQ_CST); }
// AMDGCN-LABEL: define dso_local float @fmin(
// AMDGCN-SAME: ptr nocapture noundef [[P:%.*]], float noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// AMDGCN-NEXT: [[ENTRY:.*:]]
// AMDGCN-NEXT: [[TMP0:%.*]] = atomicrmw fmin ptr [[P]], float [[X]] syncscope("agent") seq_cst, align 4
// AMDGCN-NEXT: ret float [[TMP0]]
//
// SPIRV-LABEL: define spir_func float @fmin(
// SPIRV-SAME: ptr addrspace(4) nocapture noundef [[P:%.*]], float noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// SPIRV-NEXT: [[ENTRY:.*:]]
// SPIRV-NEXT: [[TMP0:%.*]] = atomicrmw fmin ptr addrspace(4) [[P]], float [[X]] syncscope("device") seq_cst, align 4
// SPIRV-NEXT: ret float [[TMP0]]
//
float fmin(float *p, float x) { return __atomic_fetch_min(p, x, __ATOMIC_SEQ_CST); }
// AMDGCN-LABEL: define dso_local i32 @max(
// AMDGCN-SAME: ptr nocapture noundef [[P:%.*]], i32 noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// AMDGCN-NEXT: [[ENTRY:.*:]]
// AMDGCN-NEXT: [[TMP0:%.*]] = atomicrmw max ptr [[P]], i32 [[X]] syncscope("agent") seq_cst, align 4
// AMDGCN-NEXT: ret i32 [[TMP0]]
//
// SPIRV-LABEL: define spir_func i32 @max(
// SPIRV-SAME: ptr addrspace(4) nocapture noundef [[P:%.*]], i32 noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// SPIRV-NEXT: [[ENTRY:.*:]]
// SPIRV-NEXT: [[TMP0:%.*]] = atomicrmw max ptr addrspace(4) [[P]], i32 [[X]] syncscope("device") seq_cst, align 4
// SPIRV-NEXT: ret i32 [[TMP0]]
//
int max(int *p, int x) { return __atomic_fetch_max(p, x, __ATOMIC_SEQ_CST); }
// AMDGCN-LABEL: define dso_local float @fmax(
// AMDGCN-SAME: ptr nocapture noundef [[P:%.*]], float noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// AMDGCN-NEXT: [[ENTRY:.*:]]
// AMDGCN-NEXT: [[TMP0:%.*]] = atomicrmw fmax ptr [[P]], float [[X]] syncscope("agent") seq_cst, align 4
// AMDGCN-NEXT: ret float [[TMP0]]
//
// SPIRV-LABEL: define spir_func float @fmax(
// SPIRV-SAME: ptr addrspace(4) nocapture noundef [[P:%.*]], float noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// SPIRV-NEXT: [[ENTRY:.*:]]
// SPIRV-NEXT: [[TMP0:%.*]] = atomicrmw fmax ptr addrspace(4) [[P]], float [[X]] syncscope("device") seq_cst, align 4
// SPIRV-NEXT: ret float [[TMP0]]
//
float fmax(float *p, float x) { return __atomic_fetch_max(p, x, __ATOMIC_SEQ_CST); }
// AMDGCN-LABEL: define dso_local i32 @xchg(
// AMDGCN-SAME: ptr nocapture noundef [[P:%.*]], i32 noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// AMDGCN-NEXT: [[ENTRY:.*:]]
// AMDGCN-NEXT: [[TMP0:%.*]] = atomicrmw xchg ptr [[P]], i32 [[X]] syncscope("agent") seq_cst, align 4
// AMDGCN-NEXT: ret i32 [[TMP0]]
//
// SPIRV-LABEL: define spir_func i32 @xchg(
// SPIRV-SAME: ptr addrspace(4) nocapture noundef [[P:%.*]], i32 noundef [[X:%.*]]) local_unnamed_addr #[[ATTR0]] {
// SPIRV-NEXT: [[ENTRY:.*:]]
// SPIRV-NEXT: [[TMP0:%.*]] = atomicrmw xchg ptr addrspace(4) [[P]], i32 [[X]] syncscope("device") seq_cst, align 4
// SPIRV-NEXT: ret i32 [[TMP0]]
//
int xchg(int *p, int x) { return __atomic_exchange_n(p, x, __ATOMIC_SEQ_CST); }
// AMDGCN-LABEL: define dso_local range(i32 0, 2) i32 @cmpxchg(
// AMDGCN-SAME: ptr nocapture noundef [[P:%.*]], i32 noundef [[X:%.*]], i32 noundef [[Y:%.*]]) local_unnamed_addr #[[ATTR0]] {
// AMDGCN-NEXT: [[ENTRY:.*:]]
// AMDGCN-NEXT: [[TMP0:%.*]] = cmpxchg ptr [[P]], i32 [[X]], i32 [[Y]] syncscope("agent") seq_cst seq_cst, align 4
// AMDGCN-NEXT: [[TMP1:%.*]] = extractvalue { i32, i1 } [[TMP0]], 1
// AMDGCN-NEXT: [[CONV:%.*]] = zext i1 [[TMP1]] to i32
// AMDGCN-NEXT: ret i32 [[CONV]]
//
// SPIRV-LABEL: define spir_func range(i32 0, 2) i32 @cmpxchg(
// SPIRV-SAME: ptr addrspace(4) nocapture noundef [[P:%.*]], i32 noundef [[X:%.*]], i32 noundef [[Y:%.*]]) local_unnamed_addr #[[ATTR0]] {
// SPIRV-NEXT: [[ENTRY:.*:]]
// SPIRV-NEXT: [[TMP0:%.*]] = cmpxchg ptr addrspace(4) [[P]], i32 [[X]], i32 [[Y]] syncscope("device") seq_cst seq_cst, align 4
// SPIRV-NEXT: [[TMP1:%.*]] = extractvalue { i32, i1 } [[TMP0]], 1
// SPIRV-NEXT: [[CONV:%.*]] = zext i1 [[TMP1]] to i32
// SPIRV-NEXT: ret i32 [[CONV]]
//
int cmpxchg(int *p, int x, int y) { return __atomic_compare_exchange(p, &x, &y, 0, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST); }
// AMDGCN-LABEL: define dso_local range(i32 0, 2) i32 @cmpxchg_weak(
// AMDGCN-SAME: ptr nocapture noundef [[P:%.*]], i32 noundef [[X:%.*]], i32 noundef [[Y:%.*]]) local_unnamed_addr #[[ATTR0]] {
// AMDGCN-NEXT: [[ENTRY:.*:]]
// AMDGCN-NEXT: [[TMP0:%.*]] = cmpxchg weak ptr [[P]], i32 [[X]], i32 [[Y]] syncscope("agent") seq_cst seq_cst, align 4
// AMDGCN-NEXT: [[TMP1:%.*]] = extractvalue { i32, i1 } [[TMP0]], 1
// AMDGCN-NEXT: [[CONV:%.*]] = zext i1 [[TMP1]] to i32
// AMDGCN-NEXT: ret i32 [[CONV]]
//
// SPIRV-LABEL: define spir_func range(i32 0, 2) i32 @cmpxchg_weak(
// SPIRV-SAME: ptr addrspace(4) nocapture noundef [[P:%.*]], i32 noundef [[X:%.*]], i32 noundef [[Y:%.*]]) local_unnamed_addr #[[ATTR0]] {
// SPIRV-NEXT: [[ENTRY:.*:]]
// SPIRV-NEXT: [[TMP0:%.*]] = cmpxchg weak ptr addrspace(4) [[P]], i32 [[X]], i32 [[Y]] syncscope("device") seq_cst seq_cst, align 4
// SPIRV-NEXT: [[TMP1:%.*]] = extractvalue { i32, i1 } [[TMP0]], 1
// SPIRV-NEXT: [[CONV:%.*]] = zext i1 [[TMP1]] to i32
// SPIRV-NEXT: ret i32 [[CONV]]
//
int cmpxchg_weak(int *p, int x, int y) { return __atomic_compare_exchange(p, &x, &y, 1, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST); }
1 change: 1 addition & 0 deletions clang/test/Sema/scoped-atomic-ops.c
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
// RUN: %clang_cc1 -x c -triple=amdgcn-amd-amdhsa -verify -fsyntax-only %s
// RUN: %clang_cc1 -x c -triple=x86_64-pc-linux-gnu -verify -fsyntax-only %s
// RUN: %clang_cc1 -x c -triple=spirv64-unknown-unknown -verify -fsyntax-only %s

int fi1a(int *i) {
int v;
Expand Down
3 changes: 2 additions & 1 deletion llvm/lib/Target/SPIRV/SPIRVEmitIntrinsics.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1351,7 +1351,8 @@ Instruction *SPIRVEmitIntrinsics::visitAtomicCmpXchgInst(AtomicCmpXchgInst &I) {
SmallVector<Value *> Args;
for (auto &Op : I.operands())
Args.push_back(Op);
Args.push_back(B.getInt32(I.getSyncScopeID()));
Args.push_back(B.getInt32(
static_cast<uint32_t>(getMemScope(I.getContext(), I.getSyncScopeID()))));
Args.push_back(B.getInt32(
static_cast<uint32_t>(getMemSemantics(I.getSuccessOrdering()))));
Args.push_back(B.getInt32(
Expand Down
Loading