Add checks before hoisting out in loop pipelining #90872
Conversation
|
Thank you for submitting a Pull Request (PR) to the LLVM Project! This PR will be automatically labeled and the relevant teams will be If you wish to, you can add reviewers by using the "Reviewers" section on this page. If this is not working for you, it is probably because you do not have write If you have received no comments on your PR for a week, you can request a review If you have further questions, they may be answered by the LLVM GitHub User Guide. You can also ask questions in a comment on this PR, on the LLVM Discord or on the forums. |
|
@llvm/pr-subscribers-mlir-scf @llvm/pr-subscribers-mlir Author: Fotis Kounelis (fotiskoun) ChangesCurrently, during a loop pipelining transformation, operations may be hoisted out without any checks on the loop bounds, which leads to incorrect transformations and unexpected behaviour. The following issue describes the problem more extensively, including an example. Full diff: https://github.com/llvm/llvm-project/pull/90872.diff 2 Files Affected:
diff --git a/mlir/lib/Dialect/SCF/TransformOps/SCFTransformOps.cpp b/mlir/lib/Dialect/SCF/TransformOps/SCFTransformOps.cpp
index 69f83d8bd70da1..48abb888b8d2e5 100644
--- a/mlir/lib/Dialect/SCF/TransformOps/SCFTransformOps.cpp
+++ b/mlir/lib/Dialect/SCF/TransformOps/SCFTransformOps.cpp
@@ -217,6 +217,8 @@ loopScheduling(scf::ForOp forOp,
return 1;
};
+ auto ubConstant = getConstantIntValue(forOp.getUpperBound());
+ auto lbConstant = getConstantIntValue(forOp.getLowerBound());
DenseMap<Operation *, unsigned> opCycles;
std::map<unsigned, std::vector<Operation *>> wrappedSchedule;
for (Operation &op : forOp.getBody()->getOperations()) {
@@ -227,7 +229,14 @@ loopScheduling(scf::ForOp forOp,
Operation *def = operand.getDefiningOp();
if (!def)
continue;
- earlyCycle = std::max(earlyCycle, opCycles[def] + getLatency(def));
+ if (ubConstant && lbConstant) {
+ unsigned ubInt = ubConstant.value();
+ unsigned lbInt = lbConstant.value();
+ auto minLatency = std::min(ubInt - lbInt - 1, getLatency(def));
+ earlyCycle = std::max(earlyCycle, opCycles[def] + minLatency);
+ } else {
+ earlyCycle = std::max(earlyCycle, opCycles[def] + getLatency(def));
+ }
}
opCycles[&op] = earlyCycle;
wrappedSchedule[earlyCycle % iterationInterval].push_back(&op);
diff --git a/mlir/test/Dialect/SCF/transform-ops.mlir b/mlir/test/Dialect/SCF/transform-ops.mlir
index f4b0db7fb1f92a..3aa114ee9d9024 100644
--- a/mlir/test/Dialect/SCF/transform-ops.mlir
+++ b/mlir/test/Dialect/SCF/transform-ops.mlir
@@ -300,3 +300,60 @@ module attributes {transform.with_named_sequence} {
transform.yield
}
}
+
+
+// -----
+
+// CHECK-LABEL: func.func @loop_pipeline
+func.func @loop_pipeline(%arg0: memref<4x16xf32>, %arg1: vector<16xf32>) -> vector<16xf32> {
+ %c0 = arith.constant 0 : index
+ %c1 = arith.constant 1 : index
+ %cst = arith.constant 0.000000e+00 : f32
+ %c3 = arith.constant 3 : index
+ // CHECK: vector.transfer_read
+ // CHECK: vector.transfer_read
+ // CHECK: vector.transfer_read
+ // CHECK: arith.addf
+ // CHECK: arith.addf
+ // CHECK: arith.addf
+ %0 = scf.for %arg2 = %c0 to %c3 step %c1 iter_args(%arg3 = %arg1) -> (vector<16xf32>) {
+ %1 = vector.transfer_read %arg0[%arg2, %c0], %cst {in_bounds = [true]} : memref<4x16xf32>, vector<16xf32>
+ %2 = arith.addf %1, %arg3 : vector<16xf32>
+ scf.yield %2 : vector<16xf32>
+ }
+ return %0 : vector<16xf32>
+}
+module attributes {transform.with_named_sequence} {
+ transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+ %0 = transform.structured.match ops{["scf.for"]} in %arg1 : (!transform.any_op) -> !transform.op<"scf.for">
+ %1 = transform.loop.pipeline %0 {iteration_interval = 1 : i64, read_latency = 5 : i64, scheduling_type = "full-loops"} : (!transform.op<"scf.for">) -> !transform.any_op
+ transform.yield
+ }
+}
+
+
+// -----
+
+// CHECK-LABEL: func.func @loop_pipeline_lb_gt_0
+func.func @loop_pipeline_lb_gt_0(%arg0: memref<4x16xf32>, %arg1: vector<16xf32>) -> vector<16xf32> {
+ %c1 = arith.constant 1 : index
+ %cst = arith.constant 0.000000e+00 : f32
+ %c3 = arith.constant 3 : index
+ // CHECK: vector.transfer_read
+ // CHECK: vector.transfer_read
+ // CHECK: arith.addf
+ // CHECK: arith.addf
+ %0 = scf.for %arg2 = %c1 to %c3 step %c1 iter_args(%arg3 = %arg1) -> (vector<16xf32>) {
+ %1 = vector.transfer_read %arg0[%arg2, %c1], %cst {in_bounds = [true]} : memref<4x16xf32>, vector<16xf32>
+ %2 = arith.addf %1, %arg3 : vector<16xf32>
+ scf.yield %2 : vector<16xf32>
+ }
+ return %0 : vector<16xf32>
+}
+module attributes {transform.with_named_sequence} {
+ transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+ %0 = transform.structured.match ops{["scf.for"]} in %arg1 : (!transform.any_op) -> !transform.op<"scf.for">
+ %1 = transform.loop.pipeline %0 {iteration_interval = 1 : i64, read_latency = 5 : i64, scheduling_type = "full-loops"} : (!transform.op<"scf.for">) -> !transform.any_op
+ transform.yield
+ }
+}
|
|
Ping |
| @@ -217,6 +217,8 @@ loopScheduling(scf::ForOp forOp, | |||
| return 1; | |||
| }; | |||
|
|
|||
| auto ubConstant = getConstantIntValue(forOp.getUpperBound()); | |||
There was a problem hiding this comment.
Nit: please expand auto unless the type is obvious from line-level context or impossible to spell out.
There was a problem hiding this comment.
Thanks, I changed auto to std::optional<int64_t>
fotiskoun
force-pushed
the
loop-pipelining-hoisting
branch
from
004eca3 to
fc798df
Compare
|
@ftynse I have addressed the comments. Thank you! |
|
@fotiskoun Congratulations on having your first Pull Request (PR) merged into the LLVM Project! Your changes will be combined with recent changes from other authors, then tested Please check whether problems have been caused by your change specifically, as How to do this, and the rest of the post-merge process, is covered in detail here. If your change does cause a problem, it may be reverted, or you can revert it yourself. If you don't get any reports, no action is required from you. Your changes are working as expected, well done! |
Currently, during a loop pipelining transformation, operations may be hoisted out without any checks on the loop bounds, which leads to incorrect transformations and unexpected behaviour. The following issue describes the problem more extensively, including an example.
The proposed fix adds some check in the loop bounds before and applies the maximum hoisting.