[VPlan] Add hasScalarTail, use instead of !CM.foldTailByMasking() (NFC). #134674
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@llvm/pr-subscribers-vectorizers @llvm/pr-subscribers-llvm-transforms Author: Florian Hahn (fhahn) ChangesNow that VPlan is able to fold away redundant branches to the scalar preheader, we can directly check in VPlan if the scalar tail may execute. hasScalarTail returns true if the tail may execute. We know that the scalar tail won't execute if the scalar preheader doesn't have any predecessors, i.e. is not reachable. This removes some late uses of the legacy cost model. Full diff: https://github.com/llvm/llvm-project/pull/134674.diff 3 Files Affected:
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
index 12d615d9adbcc..8fdee76a8c49e 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
@@ -535,13 +535,13 @@ class LoopVectorizationPlanner {
/// Returns true if the per-lane cost of VectorizationFactor A is lower than
/// that of B.
bool isMoreProfitable(const VectorizationFactor &A,
- const VectorizationFactor &B) const;
+ const VectorizationFactor &B, bool HasTail) const;
/// Returns true if the per-lane cost of VectorizationFactor A is lower than
/// that of B in the context of vectorizing a loop with known \p MaxTripCount.
bool isMoreProfitable(const VectorizationFactor &A,
const VectorizationFactor &B,
- const unsigned MaxTripCount) const;
+ const unsigned MaxTripCount, bool HasTail) const;
/// Determines if we have the infrastructure to vectorize the loop and its
/// epilogue, assuming the main loop is vectorized by \p VF.
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 5df1061691a67..d60f0c3f1600a 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -4253,9 +4253,10 @@ static unsigned getEstimatedRuntimeVF(ElementCount VF,
return EstimatedVF;
}
-bool LoopVectorizationPlanner::isMoreProfitable(
- const VectorizationFactor &A, const VectorizationFactor &B,
- const unsigned MaxTripCount) const {
+bool LoopVectorizationPlanner::isMoreProfitable(const VectorizationFactor &A,
+ const VectorizationFactor &B,
+ const unsigned MaxTripCount,
+ bool HasTail) const {
InstructionCost CostA = A.Cost;
InstructionCost CostB = B.Cost;
@@ -4293,9 +4294,9 @@ bool LoopVectorizationPlanner::isMoreProfitable(
if (!MaxTripCount)
return CmpFn(CostA * EstimatedWidthB, CostB * EstimatedWidthA);
- auto GetCostForTC = [MaxTripCount, this](unsigned VF,
- InstructionCost VectorCost,
- InstructionCost ScalarCost) {
+ auto GetCostForTC = [MaxTripCount, HasTail](unsigned VF,
+ InstructionCost VectorCost,
+ InstructionCost ScalarCost) {
// If the trip count is a known (possibly small) constant, the trip count
// will be rounded up to an integer number of iterations under
// FoldTailByMasking. The total cost in that case will be
@@ -4304,9 +4305,10 @@ bool LoopVectorizationPlanner::isMoreProfitable(
// some extra overheads, but for the purpose of comparing the costs of
// different VFs we can use this to compare the total loop-body cost
// expected after vectorization.
- if (CM.foldTailByMasking())
- return VectorCost * divideCeil(MaxTripCount, VF);
- return VectorCost * (MaxTripCount / VF) + ScalarCost * (MaxTripCount % VF);
+ if (HasTail)
+ return VectorCost * (MaxTripCount / VF) +
+ ScalarCost * (MaxTripCount % VF);
+ return VectorCost * divideCeil(MaxTripCount, VF);
};
auto RTCostA = GetCostForTC(EstimatedWidthA, CostA, A.ScalarCost);
@@ -4314,10 +4316,12 @@ bool LoopVectorizationPlanner::isMoreProfitable(
return CmpFn(RTCostA, RTCostB);
}
-bool LoopVectorizationPlanner::isMoreProfitable(
- const VectorizationFactor &A, const VectorizationFactor &B) const {
+bool LoopVectorizationPlanner::isMoreProfitable(const VectorizationFactor &A,
+ const VectorizationFactor &B,
+ bool HasTail) const {
const unsigned MaxTripCount = PSE.getSmallConstantMaxTripCount();
- return LoopVectorizationPlanner::isMoreProfitable(A, B, MaxTripCount);
+ return LoopVectorizationPlanner::isMoreProfitable(A, B, MaxTripCount,
+ HasTail);
}
void LoopVectorizationPlanner::emitInvalidCostRemarks(
@@ -4607,7 +4611,7 @@ VectorizationFactor LoopVectorizationPlanner::selectVectorizationFactor() {
continue;
}
- if (isMoreProfitable(Candidate, ChosenFactor))
+ if (isMoreProfitable(Candidate, ChosenFactor, P->hasScalarTail()))
ChosenFactor = Candidate;
}
}
@@ -4621,7 +4625,8 @@ VectorizationFactor LoopVectorizationPlanner::selectVectorizationFactor() {
}
LLVM_DEBUG(if (ForceVectorization && !ChosenFactor.Width.isScalar() &&
- !isMoreProfitable(ChosenFactor, ScalarCost)) dbgs()
+ !isMoreProfitable(ChosenFactor, ScalarCost,
+ !CM.foldTailByMasking())) dbgs()
<< "LV: Vectorization seems to be not beneficial, "
<< "but was forced by a user.\n");
return ChosenFactor;
@@ -4713,7 +4718,8 @@ VectorizationFactor LoopVectorizationPlanner::selectEpilogueVectorizationFactor(
if (EpilogueVectorizationForceVF > 1) {
LLVM_DEBUG(dbgs() << "LEV: Epilogue vectorization factor is forced.\n");
- ElementCount ForcedEC = ElementCount::getFixed(EpilogueVectorizationForceVF);
+ ElementCount ForcedEC =
+ ElementCount::getFixed(EpilogueVectorizationForceVF);
if (hasPlanWithVF(ForcedEC))
return {ForcedEC, 0, 0};
@@ -4787,7 +4793,7 @@ VectorizationFactor LoopVectorizationPlanner::selectEpilogueVectorizationFactor(
}
if (Result.Width.isScalar() ||
- isMoreProfitable(NextVF, Result, MaxTripCount))
+ isMoreProfitable(NextVF, Result, MaxTripCount, !CM.foldTailByMasking()))
Result = NextVF;
}
@@ -7540,11 +7546,11 @@ VectorizationFactor LoopVectorizationPlanner::computeBestVF() {
InstructionCost Cost = cost(*P, VF);
VectorizationFactor CurrentFactor(VF, Cost, ScalarCost);
- if (isMoreProfitable(CurrentFactor, BestFactor))
+ if (isMoreProfitable(CurrentFactor, BestFactor, P->hasScalarTail()))
BestFactor = CurrentFactor;
// If profitable add it to ProfitableVF list.
- if (isMoreProfitable(CurrentFactor, ScalarFactor))
+ if (isMoreProfitable(CurrentFactor, ScalarFactor, P->hasScalarTail()))
ProfitableVFs.push_back(CurrentFactor);
}
}
diff --git a/llvm/lib/Transforms/Vectorize/VPlan.h b/llvm/lib/Transforms/Vectorize/VPlan.h
index a98d0ecb9a33b..fa9b1cd5cb029 100644
--- a/llvm/lib/Transforms/Vectorize/VPlan.h
+++ b/llvm/lib/Transforms/Vectorize/VPlan.h
@@ -3768,6 +3768,13 @@ class VPlan {
/// successors of the block in VPlan. The returned block is owned by the VPlan
/// and deleted once the VPlan is destroyed.
VPIRBasicBlock *createVPIRBasicBlock(BasicBlock *IRBB);
+
+ /// Returns true if the scalar tail may execute after the vector loop. Note
+ /// that this relies on unneeded branches to the scalar tail loop being
+ /// removed.
+ bool hasScalarTail() const {
+ return getScalarPreheader()->getNumPredecessors() != 0;
+ }
};
#if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
|
| @@ -4713,7 +4718,8 @@ VectorizationFactor LoopVectorizationPlanner::selectEpilogueVectorizationFactor( | |||
|
|
|||
| if (EpilogueVectorizationForceVF > 1) { | |||
| LLVM_DEBUG(dbgs() << "LEV: Epilogue vectorization factor is forced.\n"); | |||
| ElementCount ForcedEC = ElementCount::getFixed(EpilogueVectorizationForceVF); | |||
| ElementCount ForcedEC = | |||
Now that VPlan is able to fold away redundant branches to the scalar preheader, we can directly check in VPlan if the scalar tail may execute. hasScalarTail returns true if the tail may execute. We know that the scalar tail won't execute if the scalar preheader doesn't have any predecessors, i.e. is not reachable. This removes some late uses of the legacy cost model.
fhahn
force-pushed
the
vplan-has-scalar-tail
branch
from
05cd1bd to
3cd5cc6
Compare
alexey-bataev
approved these changes
Apr 7, 2025
david-arm
reviewed
Apr 8, 2025
| @@ -4787,7 +4792,7 @@ VectorizationFactor LoopVectorizationPlanner::selectEpilogueVectorizationFactor( | |||
| } | |||
|
|
|||
| if (Result.Width.isScalar() || | |||
| isMoreProfitable(NextVF, Result, MaxTripCount)) | |||
| isMoreProfitable(NextVF, Result, MaxTripCount, !CM.foldTailByMasking())) | |||
There was a problem hiding this comment.
Why doesn't this use the new hasScalarTail function yet?
There was a problem hiding this comment.
selectEpilogueVectorizationFactor doesn't really work on VPlans directly so it isn't readily available. I think it would be good to update to use VPlans directly in the funciton separately
llvm-sync bot
pushed a commit
to arm/arm-toolchain
that referenced
this pull request
Apr 11, 2025
…sking() (NFC). (#134674) Now that VPlan is able to fold away redundant branches to the scalar preheader, we can directly check in VPlan if the scalar tail may execute. hasScalarTail returns true if the tail may execute. We know that the scalar tail won't execute if the scalar preheader doesn't have any predecessors, i.e. is not reachable. This removes some late uses of the legacy cost model. PR: llvm/llvm-project#134674
var-const
pushed a commit
to ldionne/llvm-project
that referenced
this pull request
Apr 17, 2025
…C). (llvm#134674) Now that VPlan is able to fold away redundant branches to the scalar preheader, we can directly check in VPlan if the scalar tail may execute. hasScalarTail returns true if the tail may execute. We know that the scalar tail won't execute if the scalar preheader doesn't have any predecessors, i.e. is not reachable. This removes some late uses of the legacy cost model. PR: llvm#134674
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Now that VPlan is able to fold away redundant branches to the scalar preheader, we can directly check in VPlan if the scalar tail may execute. hasScalarTail returns true if the tail may execute.
We know that the scalar tail won't execute if the scalar preheader doesn't have any predecessors, i.e. is not reachable.
This removes some late uses of the legacy cost model.