[AMDGPU][Attributor] Rework update of AAAMDWavesPerEU
#123995
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This stack of pull requests is managed by Graphite. Learn more about stacking. |
|
@llvm/pr-subscribers-clang @llvm/pr-subscribers-backend-amdgpu Author: Shilei Tian (shiltian) ChangesPatch is 164.14 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/123995.diff 31 Files Affected:
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAttributor.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAttributor.cpp
index 2bc68cf2fd6a4a..dc5ba1afcfdf7a 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUAttributor.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAttributor.cpp
@@ -1109,74 +1109,38 @@ struct AAAMDWavesPerEU : public AAAMDSizeRangeAttribute {
Function *F = getAssociatedFunction();
auto &InfoCache = static_cast<AMDGPUInformationCache &>(A.getInfoCache());
- auto TakeRange = [&](std::pair<unsigned, unsigned> R) {
- auto [Min, Max] = R;
- ConstantRange Range(APInt(32, Min), APInt(32, Max + 1));
- IntegerRangeState RangeState(Range);
- clampStateAndIndicateChange(this->getState(), RangeState);
- indicateOptimisticFixpoint();
- };
-
- std::pair<unsigned, unsigned> MaxWavesPerEURange{
- 1U, InfoCache.getMaxWavesPerEU(*F)};
-
// If the attribute exists, we will honor it if it is not the default.
if (auto Attr = InfoCache.getWavesPerEUAttr(*F)) {
+ std::pair<unsigned, unsigned> MaxWavesPerEURange{
+ 1U, InfoCache.getMaxWavesPerEU(*F)};
if (*Attr != MaxWavesPerEURange) {
- TakeRange(*Attr);
+ auto [Min, Max] = *Attr;
+ ConstantRange Range(APInt(32, Min), APInt(32, Max + 1));
+ IntegerRangeState RangeState(Range);
+ clampStateAndIndicateChange(this->getState(), RangeState);
+ indicateOptimisticFixpoint();
return;
}
}
- // Unlike AAAMDFlatWorkGroupSize, it's getting trickier here. Since the
- // calculation of waves per EU involves flat work group size, we can't
- // simply use an assumed flat work group size as a start point, because the
- // update of flat work group size is in an inverse direction of waves per
- // EU. However, we can still do something if it is an entry function. Since
- // an entry function is a terminal node, and flat work group size either
- // from attribute or default will be used anyway, we can take that value and
- // calculate the waves per EU based on it. This result can't be updated by
- // no means, but that could still allow us to propagate it.
- if (AMDGPU::isEntryFunctionCC(F->getCallingConv())) {
- std::pair<unsigned, unsigned> FlatWorkGroupSize;
- if (auto Attr = InfoCache.getFlatWorkGroupSizeAttr(*F))
- FlatWorkGroupSize = *Attr;
- else
- FlatWorkGroupSize = InfoCache.getDefaultFlatWorkGroupSize(*F);
- TakeRange(InfoCache.getEffectiveWavesPerEU(*F, MaxWavesPerEURange,
- FlatWorkGroupSize));
- }
+ if (AMDGPU::isEntryFunctionCC(F->getCallingConv()))
+ indicatePessimisticFixpoint();
}
ChangeStatus updateImpl(Attributor &A) override {
- auto &InfoCache = static_cast<AMDGPUInformationCache &>(A.getInfoCache());
ChangeStatus Change = ChangeStatus::UNCHANGED;
auto CheckCallSite = [&](AbstractCallSite CS) {
Function *Caller = CS.getInstruction()->getFunction();
- Function *Func = getAssociatedFunction();
- LLVM_DEBUG(dbgs() << '[' << getName() << "] Call " << Caller->getName()
- << "->" << Func->getName() << '\n');
-
const auto *CallerInfo = A.getAAFor<AAAMDWavesPerEU>(
*this, IRPosition::function(*Caller), DepClassTy::REQUIRED);
- const auto *AssumedGroupSize = A.getAAFor<AAAMDFlatWorkGroupSize>(
- *this, IRPosition::function(*Func), DepClassTy::REQUIRED);
- if (!CallerInfo || !AssumedGroupSize || !CallerInfo->isValidState() ||
- !AssumedGroupSize->isValidState())
+ if (!CallerInfo || !CallerInfo->isValidState())
return false;
-
- unsigned Min, Max;
- std::tie(Min, Max) = InfoCache.getEffectiveWavesPerEU(
- *Caller,
- {CallerInfo->getAssumed().getLower().getZExtValue(),
- CallerInfo->getAssumed().getUpper().getZExtValue() - 1},
- {AssumedGroupSize->getAssumed().getLower().getZExtValue(),
- AssumedGroupSize->getAssumed().getUpper().getZExtValue() - 1});
- ConstantRange CallerRange(APInt(32, Min), APInt(32, Max + 1));
+ unsigned Min = CallerInfo->getAssumed().getLower().getZExtValue();
+ unsigned Max = CallerInfo->getAssumed().getUpper().getZExtValue();
+ ConstantRange CallerRange(APInt(32, Min), APInt(32, Max));
IntegerRangeState CallerRangeState(CallerRange);
Change |= clampStateAndIndicateChange(this->getState(), CallerRangeState);
-
return true;
};
@@ -1329,6 +1293,59 @@ static void addPreloadKernArgHint(Function &F, TargetMachine &TM) {
}
}
+static void checkWavesPerEU(Module &M, TargetMachine &TM) {
+ for (Function &F : M) {
+ const GCNSubtarget &ST = TM.getSubtarget<GCNSubtarget>(F);
+
+ auto FlatWgrpSizeAttr =
+ AMDGPU::getIntegerPairAttribute(F, "amdgpu-flat-work-group-size");
+ auto WavesPerEUAttr = AMDGPU::getIntegerPairAttribute(
+ F, "amdgpu-waves-per-eu", /*OnlyFirstRequired=*/true);
+
+ unsigned MinWavesPerEU = ST.getMinWavesPerEU();
+ unsigned MaxWavesPerEU = ST.getMaxWavesPerEU();
+
+ unsigned MinFlatWgrpSize = 1U;
+ unsigned MaxFlatWgrpSize = 1024U;
+ if (FlatWgrpSizeAttr.has_value()) {
+ MinFlatWgrpSize = FlatWgrpSizeAttr->first;
+ MaxFlatWgrpSize = *(FlatWgrpSizeAttr->second);
+ }
+
+ // Start with the max range.
+ unsigned Min = MinWavesPerEU;
+ unsigned Max = MaxWavesPerEU;
+
+ // If the attribute exists, set them to the value from the attribute.
+ if (WavesPerEUAttr.has_value()) {
+ Min = WavesPerEUAttr->first;
+ if (WavesPerEUAttr->second.has_value())
+ Max = *(WavesPerEUAttr->second);
+ }
+
+ // Compute the range from flat workgroup size.
+ auto [MinFromFlatWgrpSize, MaxFromFlatWgrpSize] =
+ ST.getWavesPerEU(F, std::make_pair(MinFlatWgrpSize, MaxFlatWgrpSize));
+
+ // For the lower bound, we have to "tighten" it.
+ Min = std::max(Min, MinFromFlatWgrpSize);
+ // For the upper bound, we have to "extend" it.
+ Max = std::max(Max, MaxFromFlatWgrpSize);
+
+ // Clamp the range to the max range.
+ Min = std::max(Min, MinWavesPerEU);
+ Max = std::min(Max, MaxWavesPerEU);
+
+ // Update the attribute if it is not the max.
+ if (Min != MinWavesPerEU || Max != MaxWavesPerEU) {
+ SmallString<10> Buffer;
+ raw_svector_ostream OS(Buffer);
+ OS << Min << ',' << Max;
+ F.addFnAttr("amdgpu-waves-per-eu", OS.str());
+ }
+ }
+}
+
static bool runImpl(Module &M, AnalysisGetter &AG, TargetMachine &TM,
AMDGPUAttributorOptions Options,
ThinOrFullLTOPhase LTOPhase) {
@@ -1421,8 +1438,14 @@ static bool runImpl(Module &M, AnalysisGetter &AG, TargetMachine &TM,
}
}
- ChangeStatus Change = A.run();
- return Change == ChangeStatus::CHANGED;
+ bool Changed = A.run() == ChangeStatus::CHANGED;
+
+ if (Changed && (LTOPhase == ThinOrFullLTOPhase::None ||
+ LTOPhase == ThinOrFullLTOPhase::FullLTOPostLink ||
+ LTOPhase == ThinOrFullLTOPhase::ThinLTOPostLink))
+ checkWavesPerEU(M, TM);
+
+ return Changed;
}
class AMDGPUAttributorLegacy : public ModulePass {
diff --git a/llvm/test/CodeGen/AMDGPU/addrspacecast-constantexpr.ll b/llvm/test/CodeGen/AMDGPU/addrspacecast-constantexpr.ll
index d316e10037757b..51f0348c050cd9 100644
--- a/llvm/test/CodeGen/AMDGPU/addrspacecast-constantexpr.ll
+++ b/llvm/test/CodeGen/AMDGPU/addrspacecast-constantexpr.ll
@@ -232,9 +232,9 @@ attributes #1 = { nounwind }
; AKF_HSA: attributes #[[ATTR0:[0-9]+]] = { nocallback nofree nounwind willreturn memory(argmem: readwrite) }
; AKF_HSA: attributes #[[ATTR1]] = { nounwind }
;.
-; ATTRIBUTOR_HSA: attributes #[[ATTR0:[0-9]+]] = { nocallback nofree nounwind willreturn memory(argmem: readwrite) }
-; ATTRIBUTOR_HSA: attributes #[[ATTR1]] = { nounwind "amdgpu-no-agpr" "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-flat-scratch-init" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "uniform-work-group-size"="false" }
-; ATTRIBUTOR_HSA: attributes #[[ATTR2]] = { nounwind "amdgpu-no-agpr" "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-flat-scratch-init" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "uniform-work-group-size"="false" }
+; ATTRIBUTOR_HSA: attributes #[[ATTR0:[0-9]+]] = { nocallback nofree nounwind willreturn memory(argmem: readwrite) "amdgpu-waves-per-eu"="4,10" }
+; ATTRIBUTOR_HSA: attributes #[[ATTR1]] = { nounwind "amdgpu-no-agpr" "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-flat-scratch-init" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "amdgpu-waves-per-eu"="4,10" "uniform-work-group-size"="false" }
+; ATTRIBUTOR_HSA: attributes #[[ATTR2]] = { nounwind "amdgpu-no-agpr" "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-flat-scratch-init" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "amdgpu-waves-per-eu"="4,10" "uniform-work-group-size"="false" }
;.
; AKF_HSA: [[META0:![0-9]+]] = !{i32 1, !"amdhsa_code_object_version", i32 500}
;.
diff --git a/llvm/test/CodeGen/AMDGPU/amdgpu-attributor-no-agpr.ll b/llvm/test/CodeGen/AMDGPU/amdgpu-attributor-no-agpr.ll
index 33e7e7a7a019e3..e28caf8f455e21 100644
--- a/llvm/test/CodeGen/AMDGPU/amdgpu-attributor-no-agpr.ll
+++ b/llvm/test/CodeGen/AMDGPU/amdgpu-attributor-no-agpr.ll
@@ -254,11 +254,11 @@ define amdgpu_kernel void @indirect_calls_none_agpr(i1 %cond) {
attributes #0 = { "amdgpu-no-agpr" }
;.
-; CHECK: attributes #[[ATTR0]] = { "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-flat-scratch-init" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "target-cpu"="gfx90a" "uniform-work-group-size"="false" }
-; CHECK: attributes #[[ATTR1]] = { "amdgpu-no-agpr" "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-flat-scratch-init" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "target-cpu"="gfx90a" "uniform-work-group-size"="false" }
-; CHECK: attributes #[[ATTR2]] = { "target-cpu"="gfx90a" "uniform-work-group-size"="false" }
-; CHECK: attributes #[[ATTR3:[0-9]+]] = { convergent nocallback nofree nosync nounwind willreturn memory(none) "target-cpu"="gfx90a" }
-; CHECK: attributes #[[ATTR4:[0-9]+]] = { nocallback nofree nosync nounwind speculatable willreturn memory(none) "target-cpu"="gfx90a" }
-; CHECK: attributes #[[ATTR5:[0-9]+]] = { nocallback nofree nounwind willreturn memory(argmem: readwrite) "target-cpu"="gfx90a" }
+; CHECK: attributes #[[ATTR0]] = { "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-flat-scratch-init" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "amdgpu-waves-per-eu"="4,8" "target-cpu"="gfx90a" "uniform-work-group-size"="false" }
+; CHECK: attributes #[[ATTR1]] = { "amdgpu-no-agpr" "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-flat-scratch-init" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "amdgpu-waves-per-eu"="4,8" "target-cpu"="gfx90a" "uniform-work-group-size"="false" }
+; CHECK: attributes #[[ATTR2]] = { "amdgpu-waves-per-eu"="4,8" "target-cpu"="gfx90a" "uniform-work-group-size"="false" }
+; CHECK: attributes #[[ATTR3:[0-9]+]] = { convergent nocallback nofree nosync nounwind willreturn memory(none) "amdgpu-waves-per-eu"="4,8" "target-cpu"="gfx90a" }
+; CHECK: attributes #[[ATTR4:[0-9]+]] = { nocallback nofree nosync nounwind speculatable willreturn memory(none) "amdgpu-waves-per-eu"="4,8" "target-cpu"="gfx90a" }
+; CHECK: attributes #[[ATTR5:[0-9]+]] = { nocallback nofree nounwind willreturn memory(argmem: readwrite) "amdgpu-waves-per-eu"="4,8" "target-cpu"="gfx90a" }
; CHECK: attributes #[[ATTR6]] = { "amdgpu-no-agpr" }
;.
diff --git a/llvm/test/CodeGen/AMDGPU/annotate-existing-abi-attributes.ll b/llvm/test/CodeGen/AMDGPU/annotate-existing-abi-attributes.ll
index 7e0208cd1f45aa..28722021e0448f 100644
--- a/llvm/test/CodeGen/AMDGPU/annotate-existing-abi-attributes.ll
+++ b/llvm/test/CodeGen/AMDGPU/annotate-existing-abi-attributes.ll
@@ -117,14 +117,14 @@ define void @call_no_dispatch_id() {
ret void
}
;.
-; CHECK: attributes #[[ATTR0]] = { "amdgpu-no-workitem-id-x" "uniform-work-group-size"="false" }
-; CHECK: attributes #[[ATTR1]] = { "amdgpu-no-workitem-id-y" "uniform-work-group-size"="false" }
-; CHECK: attributes #[[ATTR2]] = { "amdgpu-no-workitem-id-z" "uniform-work-group-size"="false" }
-; CHECK: attributes #[[ATTR3]] = { "amdgpu-no-workgroup-id-x" "uniform-work-group-size"="false" }
-; CHECK: attributes #[[ATTR4]] = { "amdgpu-no-workgroup-id-y" "uniform-work-group-size"="false" }
-; CHECK: attributes #[[ATTR5]] = { "amdgpu-no-workgroup-id-z" "uniform-work-group-size"="false" }
-; CHECK: attributes #[[ATTR6]] = { "amdgpu-no-dispatch-ptr" "uniform-work-group-size"="false" }
-; CHECK: attributes #[[ATTR7]] = { "amdgpu-no-queue-ptr" "uniform-work-group-size"="false" }
-; CHECK: attributes #[[ATTR8]] = { "amdgpu-no-implicitarg-ptr" "uniform-work-group-size"="false" }
-; CHECK: attributes #[[ATTR9]] = { "amdgpu-no-dispatch-id" "uniform-work-group-size"="false" }
+; CHECK: attributes #[[ATTR0]] = { "amdgpu-no-workitem-id-x" "amdgpu-waves-per-eu"="4,10" "uniform-work-group-size"="false" }
+; CHECK: attributes #[[ATTR1]] = { "amdgpu-no-workitem-id-y" "amdgpu-waves-per-eu"="4,10" "uniform-work-group-size"="false" }
+; CHECK: attributes #[[ATTR2]] = { "amdgpu-no-workitem-id-z" "amdgpu-waves-per-eu"="4,10" "uniform-work-group-size"="false" }
+; CHECK: attributes #[[ATTR3]] = { "amdgpu-no-workgroup-id-x" "amdgpu-waves-per-eu"="4,10" "uniform-work-group-size"="false" }
+; CHECK: attributes #[[ATTR4]] = { "amdgpu-no-workgroup-id-y" "amdgpu-waves-per-eu"="4,10" "uniform-work-group-size"="false" }
+; CHECK: attributes #[[ATTR5]] = { "amdgpu-no-workgroup-id-z" "amdgpu-waves-per-eu"="4,10" "uniform-work-group-size"="false" }
+; CHECK: attributes #[[ATTR6]] = { "amdgpu-no-dispatch-ptr" "amdgpu-waves-per-eu"="4,10" "uniform-work-group-size"="false" }
+; CHECK: attributes #[[ATTR7]] = { "amdgpu-no-queue-ptr" "amdgpu-waves-per-eu"="4,10" "uniform-work-group-size"="false" }
+; CHECK: attributes #[[ATTR8]] = { "amdgpu-no-implicitarg-ptr" "amdgpu-waves-per-eu"="4,10" "uniform-work-group-size"="false" }
+; CHECK: attributes #[[ATTR9]] = { "amdgpu-no-dispatch-id" "amdgpu-waves-per-eu"="4,10" "uniform-work-group-size"="false" }
;.
diff --git a/llvm/test/CodeGen/AMDGPU/annotate-kernel-features-hsa-call.ll b/llvm/test/CodeGen/AMDGPU/annotate-kernel-features-hsa-call.ll
index ea3f08ede2c5dc..da0f710abe08cf 100644
--- a/llvm/test/CodeGen/AMDGPU/annotate-kernel-features-hsa-call.ll
+++ b/llvm/test/CodeGen/AMDGPU/annotate-kernel-features-hsa-call.ll
@@ -688,7 +688,7 @@ define void @func_call_asm() #3 {
;
; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_call_asm
; ATTRIBUTOR_HSA-SAME: () #[[ATTR16]] {
-; ATTRIBUTOR_HSA-NEXT: call void asm sideeffect "", ""() #[[ATTR26:[0-9]+]]
+; ATTRIBUTOR_HSA-NEXT: call void asm sideeffect "", ""() #[[ATTR24:[0-9]+]]
; ATTRIBUTOR_HSA-NEXT: ret void
;
call void asm sideeffect "", ""() #3
@@ -717,7 +717,7 @@ define amdgpu_kernel void @func_kern_defined() #3 {
; AKF_HSA-NEXT: ret void
;
; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_kern_defined
-; ATTRIBUTOR_HSA-SAME: () #[[ATTR17:[0-9]+]] {
+; ATTRIBUTOR_HSA-SAME: () #[[ATTR16]] {
; ATTRIBUTOR_HSA-NEXT: call void @defined.func()
; ATTRIBUTOR_HSA-NEXT: ret void
;
@@ -845,7 +845,7 @@ define amdgpu_kernel void @kern_sanitize_address() #4 {
; AKF_HSA-NEXT: ret void
;
; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@kern_sanitize_address
-; ATTRIBUTOR_HSA-SAME: () #[[ATTR18:[0-9]+]] {
+; ATTRIBUTOR_HSA-SAME: () #[[ATTR17:[0-9]+]] {
; ATTRIBUTOR_HSA-NEXT: store volatile i32 0, ptr addrspace(1) null, align 4
; ATTRIBUTOR_HSA-NEXT: ret void
;
@@ -861,7 +861,7 @@ define void @func_sanitize_address() #4 {
; AKF_HSA-NEXT: ret void
;
; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_sanitize_address
-; ATTRIBUTOR_HSA-SAME: () #[[ATTR18]] {
+; ATTRIBUTOR_HSA-SAME: () #[[ATTR17]] {
; ATTRIBUTOR_HSA-NEXT: store volatile i32 0, ptr addrspace(1) null, align 4
; ATTRIBUTOR_HSA-NEXT: ret void
;
@@ -877,7 +877,7 @@ define void @func_indirect_sanitize_address() #3 {
; AKF_HSA-NEXT: ret void
;
; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_indirect_sanitize_address
-; ATTRIBUTOR_HSA-SAME: () #[[ATTR19:[0-9]+]] {
+; ATTRIBUTOR_HSA-SAME: () #[[ATTR18:[0-9]+]] {
; ATTRIBUTOR_HSA-NEXT: call void @func_sanitize_address()
; ATTRIBUTOR_HSA-NEXT: ret void
;
@@ -893,7 +893,7 @@ define amdgpu_kernel void @kern_indirect_sanitize_address() #3 {
; AKF_HSA-NEXT: ret void
;
; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@kern_indirect_sanitize_address
-; ATTRIBUTOR_HSA-SAME: () #[[ATTR19]] {
+; ATTRIBUTOR_HSA-SAME: () #[[ATTR18]] {
; ATTRIBUTOR_HSA-NEXT: call void @func_sanitize_address()
; ATTRIBUTOR_HSA-NEXT: ret void
;
@@ -928,7 +928,7 @@ define internal void @enqueue_block_def() #6 {
; AKF_HSA-NEXT: ret void
;
; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@enqueue_block_def
-; ATTRIBUTOR_HSA-SAME: () #[[ATTR22:[0-9]+]] {
+; ATTRIBUTOR_HSA-SAME: () #[[ATTR21:[0-9]+]] {
; ATTRIBUTOR_HSA-NEXT: ret void
;
ret void
@@ -941,7 +941,7 @@ define amdgpu_kernel void @kern_call_enqueued_block_decl() {
; AKF_HSA-NEXT: ret void
;
; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@kern_call_enqueued_block_decl
-; ATTRIBUTOR_HSA-SAME: () #[[ATTR23:[0-9]+]] {
+; ATTRIBUTOR_HSA-SAME: () #[[ATTR22:[0-9]+]] {
; ATTRIBUTOR_HSA-NEXT: call void @enqueue_block_decl()
; ATTRIBUTOR_HSA-NEXT: ret void
;
@@ -956,7 +956,7 @@ define amdgpu_kernel void @kern_call_enqueued_block_def() {
; AKF_HSA-NEXT: ret void
;
; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@kern_call_enqueued_block_def
-; ATTRIBUTOR_HSA-SAME: () #[[ATTR24:[0-9]+]] {
+; ATTRIBUTOR_HSA-SAME: () #[[ATTR23:[0-9]+]] {
; ATTRIBUTOR_HSA-NEXT: call void @enqueue_block_def()
; ATTRIBUTOR_HSA-NEXT: ret void
;
@@ -969,7 +969,7 @@ define void @unused_enqueue_block() {
; AKF_HSA-NEXT: ret void
...
[truncated]
|
shiltian
commented
Jan 22, 2025
|
I'll also make more tests if the code changes look reasonable. |
shiltian
changed the title
shiltian
force-pushed
the
users/shiltian/waves-per-eu-aa-rework
branch
from
0b40b9b to
3c03967
Compare
shiltian
force-pushed
the
users/shiltian/pass-stage-to-amdgpuattributor
branch
from
34fd831 to
1c1acf3
Compare
shiltian
force-pushed
the
users/shiltian/waves-per-eu-aa-rework
branch
from
3c03967 to
b063f21
Compare
shiltian
force-pushed
the
users/shiltian/pass-stage-to-amdgpuattributor
branch
2 times, most recently
from
33c1672 to
f044145
Compare
shiltian
force-pushed
the
users/shiltian/waves-per-eu-aa-rework
branch
from
b063f21 to
66cd929
Compare
shiltian
changed the title
AAAMDWavesPerEU
shiltian
force-pushed
the
users/shiltian/waves-per-eu-aa-rework
branch
from
d1dd771 to
fafefe4
Compare
shiltian
force-pushed
the
users/shiltian/pass-stage-to-amdgpuattributor
branch
from
55196cc to
9b9a320
Compare
|
ping |
|
Out of curiosity: I believe this is case three, why would we not want to use the user-provided values? |
A library use does not have the usage context. It may have specified a bounds to avoid negatively impacting occupancy of some unknown calling kernel. If we know the bounds of the concrete calling kernels, the function should specialize for its bound |
shiltian
force-pushed
the
users/shiltian/waves-per-eu-aa-rework
branch
from
fafefe4 to
64d1817
Compare
shiltian
force-pushed
the
users/shiltian/pass-stage-to-amdgpuattributor
branch
from
9b9a320 to
3118593
Compare
|
✅ With the latest revision this PR passed the undef deprecator. |
shiltian
force-pushed
the
users/shiltian/pass-stage-to-amdgpuattributor
branch
from
3118593 to
c3685eb
Compare
shiltian
force-pushed
the
users/shiltian/waves-per-eu-aa-rework
branch
from
64d1817 to
2750f73
Compare
shiltian
force-pushed
the
users/shiltian/waves-per-eu-aa-rework
branch
from
2750f73 to
7167ec8
Compare
arsenm
reviewed
Mar 24, 2025
shiltian
force-pushed
the
users/shiltian/waves-per-eu-aa-rework
branch
from
1a376e2 to
0532ee5
Compare
shiltian
force-pushed
the
users/shiltian/waves-per-eu-aa-rework
branch
from
0532ee5 to
f89193e
Compare
shiltian
force-pushed
the
users/shiltian/pass-stage-to-amdgpuattributor
branch
2 times, most recently
from
b7efaf3 to
aafd825
Compare
shiltian
force-pushed
the
users/shiltian/waves-per-eu-aa-rework
branch
from
f89193e to
eff8de0
Compare
arsenm
reviewed
May 1, 2025
| /// during attributor run because the two attributes grow in opposite direction, | ||
| /// we should not use any intermediate value to calculate waves per eu until we | ||
| /// have a determined flat workgroup size. | ||
| static void updateWavesPerEU(Module &M, TargetMachine &TM) { |
There was a problem hiding this comment.
It just occurred to me that AMDGPUAttributor is only a Module pass. Should we add or replace it with a CGSCC pass?
There was a problem hiding this comment.
We should probably add a CGSCC pass.
After a second thought, no, since we don't want to run it multiple times.
| @@ -1408,8 +1433,14 @@ static bool runImpl(Module &M, AnalysisGetter &AG, TargetMachine &TM, | |||
| } | |||
| } | |||
|
|
|||
| ChangeStatus Change = A.run(); | |||
| return Change == ChangeStatus::CHANGED; | |||
| bool Changed = A.run() == ChangeStatus::CHANGED; | |||
There was a problem hiding this comment.
Can this be more refined on waves per eu or flat workgroup size changing?
There was a problem hiding this comment.
I didn't follow your question. Can you expatiate it?
shiltian
force-pushed
the
users/shiltian/pass-stage-to-amdgpuattributor
branch
from
aafd825 to
35fda65
Compare
shiltian
force-pushed
the
users/shiltian/waves-per-eu-aa-rework
branch
5 times, most recently
from
66d27ed to
1e140dd
Compare
shiltian
force-pushed
the
users/shiltian/pass-stage-to-amdgpuattributor
branch
from
35fda65 to
ed136cc
Compare
shiltian
force-pushed
the
users/shiltian/waves-per-eu-aa-rework
branch
from
1e140dd to
7c79a09
Compare
Base automatically changed from
users/shiltian/pass-stage-to-amdgpuattributor
to
main
May 2, 2025 15:33
shiltian
force-pushed
the
users/shiltian/waves-per-eu-aa-rework
branch
from
7c79a09 to
6b7be9c
Compare
|
ping |
shiltian
force-pushed
the
users/shiltian/waves-per-eu-aa-rework
branch
from
6b7be9c to
c235997
Compare
jdoerfert
reviewed
May 16, 2025
shiltian
force-pushed
the
users/shiltian/waves-per-eu-aa-rework
branch
3 times, most recently
from
a24f0d0 to
72851ae
Compare
jdoerfert
approved these changes
May 17, 2025
shiltian
force-pushed
the
users/shiltian/waves-per-eu-aa-rework
branch
from
72851ae to
352a09a
Compare
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/202/builds/1279 Here is the relevant piece of the build log for the reference |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, we use
AAAMDWavesPerEUto iteratively update values based on attributes from the associated function, potentially propagating user-annotated values, along withAAAMDFlatWorkGroupSize. Similarly, we haveAAAMDFlatWorkGroupSize. However, since the value calculated through the flat workgroup size always dominates the user annotation (i.e., the attribute), runningAAAMDWavesPerEUiteratively is unnecessary if no user-annotated value exists.This PR completely rewrites how the
amdgpu-waves-per-euattribute is handled inAMDGPUAttributor. The key changes are as follows:AAAMDFlatWorkGroupSizeremains unchanged.AAAMDWavesPerEUnow only propagates user-annotated values.amdgpu-waves-per-eubased on the following rules:1,1024and compute waves per eu based on this.1,1024, but this can be adjusted if a different approach is preferred. Alternatively, we could directly use the user-annotated value.This PR also updates the logic for merging two waves per eu pairs. The current implementation, which uses
clampStateAndIndicateChangeto compute a union, might not be ideal. If we think from ensure proper resource allocation perspective, for instance, if one pair specifies a minimum of 2 waves per eu, and another specifies a minimum of 4, we should guarantee that 4 waves per eu can be supported, as failing to do so could result in excessive resource allocation per wave. A similar principle applies to the upper bound. Thus, the PR uses the following approach for merging two pairs,lo_a,up_aandlo_b,up_b:max(lo_a, lo_b), max(up_a, up_b). This ensures that resource allocation adheres to the stricter constraints from both inputs.Fix #123092.