StackOverflow during staging

[hearnadam]

On my github CI, I consistently see compiler crashes during staging. The code itself has been hard to minimize due to the crash not happening locally. I tried a variety of stack sizes, but once I reach -Xss512k other parts of the code fail to compile due to non-trivial inlining.

Compiler version

• 3.6.3

Minimized code

The code is very similar to this file: https://github.com/getkyo/kyo/blob/main/kyo-examples/jvm/src/main/scala/examples/ledger/api/Endpoints.scala

It uses kyo-direct. The code is private, but happy to share privately with maintainers.

Output (click arrow to expand)

error] ## Exception when compiling 3 sources to /home/runner/work/winedb/winedb/backend/target/scala-3.6.3/classes
[error] java.lang.StackOverflowError
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1524)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1531)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1537)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform$$anonfun$1(Trees.scala:1643)
[error] scala.collection.immutable.List.mapConserve(List.scala:473)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1643)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:55)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:34)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1605)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:43)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.loop$2(tpd.scala:1289)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformStats(tpd.scala:1289)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformBlock(tpd.scala:1294)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1551)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:43)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:24)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1600)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:43)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.loop$2(tpd.scala:1289)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformStats(tpd.scala:1289)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformBlock(tpd.scala:1294)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1551)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:43)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:24)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1545)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1571)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1531)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1539)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1537)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformBlock$$anonfun$1$$anonfun$1(tpd.scala:1294)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.loop$2(tpd.scala:1276)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformStats(tpd.scala:1289)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformBlock(tpd.scala:1294)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1551)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:43)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:24)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformBlock$$anonfun$1$$anonfun$1(tpd.scala:1294)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.loop$2(tpd.scala:1276)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformStats(tpd.scala:1289)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformBlock(tpd.scala:1294)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1551)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:43)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:24)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1545)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1571)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1531)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1539)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1537)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformBlock$$anonfun$1$$anonfun$1(tpd.scala:1294)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.loop$2(tpd.scala:1276)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformStats(tpd.scala:1289)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformBlock(tpd.scala:1294)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1551)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:43)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:24)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformBlock$$anonfun$1$$anonfun$1(tpd.scala:1294)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.loop$2(tpd.scala:1276)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformStats(tpd.scala:1289)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformBlock(tpd.scala:1294)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1551)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:43)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:24)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1545)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1571)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1531)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1539)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1537)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformBlock$$anonfun$1$$anonfun$1(tpd.scala:1294)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.loop$2(tpd.scala:1276)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformStats(tpd.scala:1289)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformBlock(tpd.scala:1294)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1551)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:43)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:24)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:43)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:24)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1600)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:43)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.loop$2(tpd.scala:1289)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformStats(tpd.scala:1289)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformBlock(tpd.scala:1294)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1551)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:43)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:24)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1545)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1571)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1531)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1539)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1537)
[error] dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
[error] dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
[error] dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
[error] dotty.tools.dotc.ast.tpd$TreeMapWithPreciseStatContexts.transformBlock$$anonfun$1$$anonfun$1(tpd.scala:1294)

[WojciechMazur]

-Xss512k is ~ 0.5MB per thread, the JVM default is 1MB. The Scala compiler is not multithreading-heavy so typically you'd want to have bigger stack size, especially in case on non trivial code, like multi-level inlines.
The kyo is using lots of inlining and non trivial typing which might require a bigger stack size for the compiler.
I believe that instead of StackOverflowError the efforts should focus on dealing with the inlining.

[Gedochao]

@jchyb @dwijnand Can you think of any advice, how to help with minimising/reproducing this?

The code is private, but happy to share privately with maintainers.

@hearnadam it's hard to tell when we'll have the capacity to spend time on this in the maintenance team. When someone is available, we may reach out, but it'd be best to try to minimise this on your side, at least partially.

[jchyb]

Looking through the stack trace, nothing seems out of the ordinary - it just walks thought the program tree. I am wondering if perhaps somehow a different code is generated by Kyo locally and in the CI, as unlikely as that sounds. You can try running with the -Xcheck-macros option to check if the code generated by Kyo is correct. Otherwise I agree with @WojciechMazur that the issue might be with a too small of a stack size.