Quotes: generate arbitrary trait or class implementations

[ghik]

The problem

Using Scala 2 macros, we have built an RPC framework that can automatically generate implementations of arbitrary traits. These implementations are effectively network proxies that translate every method call into a network request. When the request is received by the RPC server, a reverse translation is performed and a real implementation of this trait is invoked.

A very simplified example of this mechanism:

trait RawRpc {
  def invoke(methodName: String, parameters: Map[String, Json]): Future[Json]
}
object RawRpc {
  def asReal[RealRpc](rawRpc: RawRpc): RealRpc = macro ...
  def asRaw[RealRpc](realRpc: RealRpc): RawRpc = macro ...
}

The asReal macro needs to do the following:

• enumerate all (abstract) methods of RealRpc
• for every parameter of every method, find a typeclass instance that would allow it to serialize it into Json
• for every method's result type, find a typeclass instance that would allow it to deserialize it from Json (assuming that the result type is a Future)
• generate an implementation of every method that forwards it to rawRpc
• inspect annotations on methods and parameters in order to treat them in some special way

The asRaw macro needs similar capabilities in order to perform a reverse translation.

Situation in Scala 3

It's unclear to me whether something like this is possible with Scala 3. Looking at the Quotes API, it should in principle be possible if I was able to generate arbitrary trees, starting with ClassDef. However, it seems that currently ClassDef.apply is commented out and marked as TODO.

So my questions are:

• First of all, is this a valid use case for Scala 3 metaprogramming?
• If yes, can this be done with Quotes API?
• Are there any serious obstacles preventing methods like ClassDef.apply from being exposed?

(Note: I asked this question briefly on Gitter and it was recommended to me to create an issue.)

[joroKr21]

We also need to be able to implement traits in order to port cats-tagless to Scala 3. This is blackbox macro territory so it should be a perfectly reasonable use case for Scala 3 macros.

[nicolasstucki]

What are the classes that you are generating? Could you give a small example of the code you would have and what code is generated?

[joroKr21]

Example from cats-tagless (where ~> is FunctionK from cats and Stream is from fs2):

trait InvariantK[Alg[_[_]]] extends Serializable {
  def imapK[F[_], G[_]](af: Alg[F])(fk: F ~> G)(gk: G ~> F): Alg[G]
}

trait FunctorK[Alg[_[_]]] extends InvariantK[Alg] {
  def mapK[F[_], G[_]](af: Alg[F])(fk: F ~> G): Alg[G]
  override def imapK[F[_], G[_]](af: Alg[F])(fk: F ~> G)(gk: G ~> F): Alg[G] = mapK(af)(fk)
}

trait ContravariantK[Alg[_[_]]] extends InvariantK[Alg] {
  def contramapK[F[_], G[_]](af: Alg[F])(fk: G ~> F): Alg[G]
  override def imapK[F[_], G[_]](af: Alg[F])(fk: F ~> G)(gk: G ~> F): Alg[G] = contramapK(af)(gk)
}

final case class User(name: String, age: Int)

trait UserRepo[F[_]] {
    type Id

    def create(user: User): F[Id]
    def findAll(olderThan: Int): Stream[F, User]
    def pickOne(ids: Id*)(f: List ~> F): F[User]
  }

  val functorK: FunctorK[UserRepo] = Derive.functorK
  // macro generated
  new FunctorK[UserRepo] {
    override def mapK[F[_], G[_]](af: UserRepo[F])(fk: F ~> G): UserRepo[G] = new UserRepo[G] {
      type Id = af.Id

      override def create(user: User): F[Id] =
        fk(af.create(user))

      override def findAll(olderThan: Int): Stream[F, User] =
        FunctorK[Stream[*[_], User]].mapK(af.findAll(olderThan))(fk)

      override def pickOne(ids: Id*)(f: FunctionK[List, G]): F[User] =
        fk(af.pickOne(ids: _*)(ContravariantK[FunctionK[List, *[_]]].contramapK(f)(fk)))
    }
  }

Sorry, I think I messed up ContravariantK[FunctionK[List, *[_]]] but I think the idea is clear - arguments are in contravariant position so we can use ContravariantK[[F[_]] => ArgType[F]] to contramapK them.

[nicolasstucki]

Good, it seems that you can create that class in a much simpler way. As you seem to know that you want to implement an instance of FunctorK/RawRpc you can just use quoted code for that part, the rest will need some reflection.

Here is the basic idea

def asRaw[RealRpc](realRpc: RealRpc): RawRpc = ${ asRawExpr('realRpc) } 
private def asRaw[RealRpc: Type](realRpc: Expr[RealRpc])(using Quotes): Expr[RawRpc] = 
  '{ 
    new RawRpc {
      def invoke(methodName: String, parameters: Map[String, Json]): Future[Json] = 
        ${invokeExpr(realRpc, 'methodName, 'parameters) }
    } 
  }

private def invokeExpr[RealRpc: Type](realRpc: Expr[RealRpc], methodName: Expr[String], parameters: Expr[Map[String, Json]])(using Quotes): Expr[RawRpc] = 
  import quotes.reflect._
  ...

[joroKr21]

Note: we need to implement both an instance of FunctorK and UserRepo - and UserRepo is provided by the user.

[nicolasstucki]

Then the second might be problematic. I will have to have a deeper look.

[ghik]

Coming back to my example with RawRpc, let's assume a typeclass for Json serialization:

trait JsonCodec[T] {
  def encode(value: T): Json
  def decode(json: Json): T
}
object JsonCodec {
  def apply[T](implicit codec: JsonCodec[T]): JsonCodec[T] = codec
}

Then let's assume an example user RPC trait:

trait MyRpc {
  def handleStuff(foo: String, bar: Int): Future[Boolean]
  def doSomethingElse(baz: Double): Future[Unit]
}

An invocation of RawRpc.asReal for MyRpc would generate something like this:

new MyRpc {
  def handleStuff(foo: String, bar: Int): Future[Boolean] =
    rawRpc.invoke("handleStuff", Map(
      "foo" -> JsonCodec[String].encode(foo), 
      "bar" -> JsonCodec[Int].encode(bar)
    )).map(JsonCodec[Boolean].decode)
  
  def doSomethingElse(baz: Double): Future[Unit] =
    rawRpc.invoke("doSomethingElse", Map(
      "baz" -> JsonCodec[Double].encode(baz)
    )).map(JsonCodec[Unit].decode)
}

Conversely, an invocation of RawRpc.asRaw for MyRpc would generate something like this:

new RawRpc {
  def invoke(methodName: String, parameters: Map[String, Json]): Future[Json] = methodName match {
    case "handleStuff" => realRpc.handleStuff(
      JsonCodec[String].decode(parameters("foo")), 
      JsonCodec[Int].decode(parameters("bar"))
    ).map(JsonCodec[Boolean].encode)
    
    case "doSomethingElse" => realRpc.doSomethingElse(
      JsonCodec[Double].decode(parameters("baz"))
    ).map(JsonCodec[Unit].encode)
  }
}

Of course, the MyRpc is an arbitrary user trait. Also, our framework is more generic than the example above, because:

• RawRpc may take multiple forms so the macro may need to do reflection on RawRpc too (not just on user's RealRpc)
• some methods and parameters of RealRpc may be treated specially based on their signature, annotations, etc.

[nicolasstucki]

Unfortunately, it is not something we can realistically have by 3.0. We aim to have this in 3.1.

Meanwhile, we could see if there is a possible workaround. It looks like we could use a java.lang.reflect.Proxy to handle the calls to handleStuff/doSomethingElse. The code of this proxy would be the code you would generate for the bodies of those methods with some extra wrapping logic.

[ghik]

I see, thank you.
Unfortunately java.lang.reflect.Proxy isn't a solution for us because we need this to be working in Scala.js. The best thing now seems to compile these traits with Scala 2 (macro invocations will be limited to their companion objects) and consume (invoke & implement) them in Scala 3.