Compiler stack-overflows or fails silently on uncaught cyclic type reference

[busti]

reproduction steps

using Scala (2.13.7),

object lib {
  trait Tx[+A] {
    type J[+T] <: Tx[T]
    
    def map[B](fn: A => B): J[B]
  }

  trait TxBuilder { self =>
    type J[+T] <: Tx[T]
  }
}

trait EmitterBuilder[+O] { self =>
  val BTx: lib.TxBuilder
  type TTx[+T] = self.BTx.J[T] { type J[+U] = self.TTx[U] }

  def transform[T](fn: TTx[O] => TTx[T]): EmitterBuilder[T]

  def map[T](fn: O => T): EmitterBuilder[T] = transform[T](_.map(fn))
}

problem

In an empty Project using the above code the compiler fails either silently (running out of memory, it eventually reports that it spent the last 99% of time on GC) or with a stackoverflow exception, depending on the amount of ram given to the compiler.

The Scala 3 compiler correctly reports the issue with the following message:

illegal cyclic type reference: alias ... of type TTx refers back to the type itself

I was able to reproduce the issue on scastie aswell as on my own machine.
https://scastie.scala-lang.org/fezxomx6SbGZYLoq5dAJzQ

[joroKr21]

Minimization: type F[x] = { type U = F[Int] }
Interestingly the *-kinded version is caught: type T = { type U = T }
So it might be helpful to find the difference in implementation.