Static methods in objects extending an abstract class, or classes with covariant-overriding mxins, get inconsistent java generics signature

[scabug]

See this [http://scala-programming-language.1934581.n4.nabble.com/valid-code-throws-NoSuchMethodError-at-runtime-td2222685.html#a2222685 thread]

Compiling this with scalac (tested 2.8.0.RC2):

abstract class BulkSearch {
       type R   <: Row
       type Rel <: Relation [R]
       type Corr <: Correspondence[R]

       def searchFor(input: Rel): Mapping[Corr] = null
}

object BulkSearchInstance extends BulkSearch {
       type R   = UpRow
       type Rel = UpRelation
       type Corr = UpCorrespondence
}

class Row
class UpRow extends Row

class Relation [R <: Row]
class UpRelation extends Relation [UpRow]

class Correspondence [R <: Row]
class UpCorrespondence extends Correspondence [UpRow]

class Mapping[MC <: Correspondence[_]]

and compiling this with javac:

public class JavaApp {
       public static void main(String[] args) {
               BulkSearchInstance.searchFor(new UpRelation());
       }
}

produces no compile-time errors but a runtime error later on:

Exception in thread "main" java.lang.NoSuchMethodError: bugs.BulkSearchInstance.searchFor(Lbugs/UpRelation;)Lbugs/Mapping;
       at bugs.JavaApp.main(JavaApp.java:15)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at com.intellij.rt.execution.application.AppMain.main(AppMain.java:110)

After some analysis, I found out that is probably due to an inconsistent generic signature being generated for the static method delegate in BulkSearchInstance.class.

Scalac generates these signatures for BulkSearchInstance.searchFor:

method_info.descriptor (see � 4.3 JVM spec): (LRelation;)LMapping;
attribute 'Signature' (see JSR 014): (LUpRelation;)LMapping<LUpCorrespondence;>;

When javac sees that, it produces something strange (i.e. not matching the descriptor found in BulkSearchInstance.class):

invokestatic    SI-4; //Method BulkSearchInstance.searchFor:(LUpRelation;)LMapping;

which looks like the erasure of the generic signature. However, JVM's linker looks for the exact descriptor when looking for methods. I think the solution should be

1. Always generate consistent descriptor and generic signature
2. At least, in the case of the static delegator method in the object class, generate the most specific type possible. The method can't be overridden anyways.

Workaround: use a trait instead of an abstract class

[scabug]

Imported From: https://issues.scala-lang.org/browse/SI-3452?orig=1
Reporter: @jrudolph
See #1459, #8293
Duplicates #7374

[scabug]

@dragos said:
(In r22043) Closes #3452, inconsistency between method descriptors and java signature for static forwarders. Review by extempore.

[scabug]

@paulp said:
Well, I guess I'm 10 months behind on my reviews. (More even.) There was no test case checked in, and as far as I can tell the behavior never changed. The example given in this ticket fails at runtime with scala 2.8.0 and r24603.

[scabug]

@jrudolph said:
I've further simplified the example to

trait Search[M] {
    def search(input: M): C[Int] = null
}

object StringSearch extends Search[String]

trait C[T]

and

public class JavaApp {
       public static void main(String[] args) {
               StringSearch.search("test");
       }
}

In a strange way the error depends on the return value being an applied type (strange because the error appears in the parameter position). The problem is actually catched using the -Ycheck:genjvm flag:

src/main/scala/Test.scala:5: warning: compiler bug: created generic signature for method search in StringSearch that does not conform to its erasure
signature: (Ljava/lang/String;)LC<Ljava/lang/Object;>;
original type: (input: String)C[Int]
normalized type: (input: String)C[Object]
erasure type: (input: Object)C
if this is reproducible, please report bug at http://lampsvn.epfl.ch/trac/scala
object StringSearch extends Search[String]
       ^
one warning found

[scabug]

DaveScala (davescala) said:
Another report of this bug
see:
http://groups.google.com/group/scala-user/browse_thread/thread/35fa02126594bfa4

package types
trait IStringPair[T] { 
  def a : String 
  def b : String 
  def build(a : String, b : String) : T 
  def cat(that : IStringPair[T]) = build(this.a + that.a, this.b + that.b) 
  override def toString = a + b 
} 

class StringPair(val a : String, val b : String) extends IStringPair[StringPair] { 
  def build(a : String, b : String) = new StringPair(a, b) 
  def len = a.length + b.length 
} 

C:\scala-2.10.0-M2\examples\testjava>scalac -Ycheck:genjvm types.scala
types.scala:10: warning: compiler bug: created generic signature for method cat
in types.StringPair that does not conform to its erasure
signature: (Ltypes/IStringPair<Ltypes/StringPair;>;)Ltypes/StringPair;
original type: (that: types.IStringPair)types.StringPair
normalized type: (that: types.IStringPair)types.StringPair
erasure type: (that: types.IStringPair)Object
if this is reproducible, please report bug at http://lampsvn.epfl.ch/trac/scala
class StringPair(val a : String, val b : String) extends IStringPair[StringPair]
 {
      ^
one warning found

[scabug]

@paulp said:
What version of scala is that? I can't reproduce it with any version.

[scabug]

@paulp said:
OK, I can reproduce the breakage (which is the important part) as outlined at that thread, but I don't get the warning.

[scabug]

DaveScala (davescala) said:
I could reproduce it in scala-2.10.0-M2 with compiler option -Ycheck:genjvm which generates the warning and seems to fix the signature erasure too.

[scabug]

@jrudolph said:
Yes, you need the -Ycheck:genjvm to generate the warning. Looking at the code there's a return after generating the warning which circumvents the generation of the broken signature. This would explain why using the flag makes the runtime error disappear.

It may be sensible to enable this flag/code path by default if this can't be fixed soon or is still likely to be broken in some cases. That would mean that a) you can use these methods at all from Java even if the compiler still has a bug when generating them b) you are loosing generic type info at the Java side because the generic type is missing. Compare that to the current situation where although Java can compile against the generic type, the call fails at runtime.

[scabug]

@paulp said:
I spent most of the last two days on this bug. It's a multiway battle of a bunch of tests where one passing leads to another failing. Still unresolved.

BTW I don't see the warning regardless of -Ycheck, I don't know why. I know it's generating the bad signatures.

[scabug]

@jrudolph said:
I see. For the original case one simple (naive?) solution would have been to use the checking code to generate the method descriptor from the generic signature. This should work for the static forwarders for objects.

For the new case Dave gave this is obviously not working because the important part is that the method descriptor actually implements the Java interface correctly. So, the generation of the generic signature has to be fixed in this case to match the descriptor. Maybe in this case we should just copy the overwritten method's signature over to the method implementing the method.

It's strange that you're not seeing the warning, it's there in the code, isn't it?. Dave seemed to be working with Scala 2.10.0-M2, right? In my unorderly ways I'm often on some version, where in this case it was f987afe which is master from two weeks ago.

[scabug]

@paulp said:
Oh, wonderful. I wasn't seeing the warning because the check looking for the string "genjvm" which is not actually the name of the phase. I, having catlike reflexes with respect to the phase names, was entering it as "-Ycheck:jvm" which is what it is supposed to be. You guys were smart enough to enter the string it was looking for.

• NOTICE! I'M FIXING IT! **

That means it's -Ycheck:jvm, and -Ycheck:genjvm refers to a non-existent phase. The options which take a phase argument should probably fail on mystery strings, but right now they silently ignore. Still, you know if it checked by examining the output: it will always say "[Now checking: jvm]" if it understood you, and it will be silent if it did not.

[scabug]

@paulp said:
The reason I only warn rather than fixing it at the signature generation point is that detecting it depends on sun internal classes and I don't want to add one more mystery by having it work some places and not others. It would not be rocket science to write our own signature parser, but it's not something I have time for at the moment (even though I've already done it, more than once in fact...) if anyone wants to bring up to date some of the parsers in here, it'd be so appreciated.

https://github.com/paulp/awesome

Look how beautiful:

https://github.com/paulp/awesome/blob/master/src/main/scala/awesome/jvm/SignatureParser.scala

This one is especially compelling given that that's all the necessary logic:

https://github.com/paulp/awesome/blob/master/src/main/scala/awesome/jvm/DescriptorParser.scala

[scabug]

@Blaisorblade said:
Paul, I wanted to start hacking on the compiler by addressing this bug, since writing a signature verifier sounded easy enough. However, the task seems to have become irrelevant by now, and maybe to have always been irrelevant for this bug. Hence I collected the pieces together to reintroduce support for -Xverify and to enable the proposed workaround.

More in detail:
(1) the signature parser does not seem to be needed for this warning - in fact, it is currently not used at all, but was used when you wrote this message* for a different warning. You removed this part of the implementation of -Xverify 4 days ago (scala/scala@317a105#diff-18).
(2) I made the extra checking unconditional, since the only obstacle you mentioned (dependency on Sun classes) does not apply.
(3) By now, an ASM-based signature parsing is integrated and used by the GenASM backend - I understand that now ASM is already included in the compiler. This warning has no dependencies on the rest of the ASM backend. Therefore, I readded an implementation of that warning based on ASM, by intelligently merging the old code with the GenASM code. The warning code seems to run correctly (reporting no warning here), but I'd like to have some testcase.

In a moment I'll create a pull request which will reference this comment.

• I chose some commit around the date of your last post, 30 Mar 2012 (commit 88bc33301a1d1994a7a60f91ff7603c8e902b6a8 from the scala/scala tree). At that time there are two independent warnings, and one relies on Sun's SigParser (triggered by -Xverify) while the other is the one triggered here (triggered by -Ycheck:jvm).

[scabug]

@Blaisorblade said:
I just submitted pull request scala/scala#584. I'm running the test suite now, and I'll run 'ant dist' later to be totally sure I didn't break anything (since ant is managing incremental compilation, that looks required).