wrong assumption about `invokespecial`

[lrytz]

I went again through the invokespecial spec. One thing i didn't realize before is that the invoked method may be defined in a different class than the one used as receiver in the invocation instruction.

The compiler assumes that invokespecial A.m will always invoke method m defined in class A, which is not the case: it may invoke an overriding member in a subclass of A.

Related / affected issues:

• inline new mixin methods #86: the optimizer considers invokespecial as statically resolved, this may not hold under separate compilation
• new trait encoding: skip unnecessary mixin methods #98
• https://issues.scala-lang.org/browse/SI-4989

Java example:

public class A {
  public int m() { return 1; }
}
public class B extends A {
  // public int m() { return 2; }
}
public class C extends B {
  public int m() { return super.m(); }

  public static void main(String[] args) {
    System.out.println((new C()).m());
  }
}

Method C.m has the invocation INVOKESPECIAL B.m ()I, even though there's no method m in B (it's commented out). According to the spec, the method to be invoked is selected by searching for a matching member (name and signature), starting at the superclass of C, continuing with further superclasses, and finally interfaces (default methods). So it will invoke A.m and print 1

If we un-comment the definition of B.m and only re-compile B.java, the INVOKESPECIAL will now invoke B.m and print 2.

Here's an example where we get things wrong:

class A {
  def m = 1
}

class B extends A {
  override def m = 2
}

trait T extends A

class C extends B with T {
  override def m = super[T].m
}

object Test {
  def main(args: Array[String]): Unit = {
    println((new C).m)
  }
}

According to the spec this should print 1 (right?)

C.super[T].x [...] is called a static super reference. In this case, the reference is to the type or method of x in the parent trait of C whose simple name is T.

In M4 we get

public class C extends B  implements T  {
  public m()I
    ALOAD 0
    INVOKESPECIAL T.m ()I
    IRETURN
}
public abstract interface T {
  // no method m
}

Which is incorrect, running the example gives java.lang.NoSuchMethodError: T.m()I.

In current 2.12.x we get this - the change is probably due to my recent PR scala/scala#5096:

public class C extends B  implements T  {
  public m()I
    ALOAD 0
    INVOKESPECIAL A.m ()I
    IRETURN
}

Running the example gives 2. What happens is that method selection (see the invokespecial spec) starts at the superclass of C, which is B, and looks for a member matching m()I. Selection does NOT start at A.

We make the same mistake in 2.11: the body of C.m contains INVOKESPECIAL A.m ()I, so running the example also gives 2.

[lrytz]

I added a lot of labels to this bug to make it look more important 🤓

[sjrd]

From the linked spec:

If all of the following are true, let C be the direct superclass of the current class:

• The resolved method is not an instance initialization method (§2.9).
• If the symbolic reference names a class (not an interface), then that class is a superclass of the current class.
• The ACC_SUPER flag is set for the class file (§4.1).

Otherwise, let C be the class or interface named by the symbolic reference.

Note the last sentence. Apparently we could achieve what we initially want simply by not setting ACC_SUPER on Scala-generated class files. I have no idea what else is impacted by this, though.

[sjrd]

Also, FTR, the corresponding Scala.js IR node (ApplyStatically) behaves as was previously assumed, i.e., the lookup does start at A.

[sjrd]

Uh re ACC_SUPER, my comment is wrong:

In Java SE 8 and above, the Java Virtual Machine considers the ACC_SUPER flag to be set in every class file, regardless of the actual value of the flag in the class file and the version of the class file.

(from https://docs.oracle.com/javase/specs/jvms/se8/html/jvms-4.html#jvms-4.1)

[lrytz]

yeah, ACC_SUPER is really a historical artifact, we should not use it, see http://stackoverflow.com/questions/8949933/what-is-the-purpose-of-the-acc-super-access-flag-on-java-class-files

[retronym]

Eek. If only we could do in regular bytecode as https://docs.oracle.com/javase/7/docs/api/java/lang/invoke/MethodHandles.Lookup.html unreflectSpecial is able to do reflectively.

[retronym]

Static methods (paired with non static forwarders) are looking more and more like a way out of our invokespecial blues.

[lrytz]

Shockingly, even INVOKESTATIC B.m ()I does not necessarily mean that B.m is executed. Java example:

public class A {
  public static int m() { return 1; }
}
public class B extends A {
  public static int m() { return 2; }
}
public class C extends B {
  public static void main(String[] args) {
    System.out.println(B.m());
  }
}

This produces INVOKESTATIC B.m ()I, running C prints 2. Now delete the definition of m in B and re-compile only B.java. Method resolution does not fail, but instead returns A.m, so running C now prints 1.

[retronym]

It sure is hard to disclaim one's inheritance.

In addition to the separate compilation example, it is legal in Java source to write B.m in the absence of the method in B if it exists in A. If A is inaccessible, this can be the only way to call it.

[lrytz]

Ah, right! Anyway this issue is less problematic because we can statically resolve the method (in the scala compiler / optimizer), assuming the correct classes are on the classpath, which is a core assumption of the optimizer anyway.