Replace ANTLR 2.x with updated ANTLR 4

[benfry]

We're using a very old version of ANTLR. @fjenett did some work to move us to ANTLR 4, but there hasn't been enough testing (or someone committed to finishing that work) to include it in 3.x yet.

See also #3054, which is about updating the grammar (the java.g file) to support newer Java language features.

[benfry]

Link to Florian's earlier work: https://github.com/fjenett/processing-preprocessor-antlr4

Note that because this is so fundamental to what the PDE does, it needs a lot of testing and a committed maintainer (or at least longer than "hey this is working / incorporate it / bye"). We can't incorporate a "mostly working" set of code.

[JakubValtar]

I just happen to have a working ANTLR4 prototype branch based on Florian's work with complementary Android branch I worked on in August... it was more of a proof of concept work, but tests are green and I can run Java and Android sketches from PDE. Needs update to current HEAD, new error reporting, better tests. API for other modes will need some thought.

https://github.com/JakubValtar/processing/tree/feature-antlr4
https://github.com/JakubValtar/processing-android/tree/feature-antlr4-android

It's a great responsibility and I have to consider if I will be able to do good enough job and maintain it. However, I can at least make it work in current version in the meantime.

[benfry]

Cool, thanks for looking into it. Keep us posted on how it evolves, and/or someone may also be able to help build on your work as you've done w/ Florian's.

Here's an excerpt from an email from Florian:

As a question remains how we should handle errors from the parser. ANTLR-4 comes with a nice listener interface that let's one subscribe to the parser. This would make error handling a lot cleaner and easier for us … but would mean that JavaBuild would need to implement the listener interface:
https://github.com/fjenett/processing/blob/antlr4-preproc/app/src/processing/mode/java/preproc/PdePreprocessor.java#L127
The compilation and runtime error handling needs some rethinking too. The way i generate the .java files leaves the headerOffset solution not functioning anymore as all in-between imports are moved to the beginning of the file. One way of solving this could be that i return a mapping between sketch and java lines so we could do a simple reverse lookup. That would make integration with PDE-X easier too i assume.
As a goodie i am using the parser to fish out the arguments from the size() call. Currently it is only printed to console but could easily be returned through the ProcessorResult …
Another one: if an api keyword is used as variable name an error will be generated. For example the "Function" sketch in the examples has "size" as a method parameter.
Empty sketches raise an error too. This is mainly because the parser can not handle empty sketches at the moment.

And some back-and-forth with Florian responding to Manindra about incorporating ANTLR 4 with the PDE X code (which is now being merged into the main mode):

One way of solving this could be that i return a mapping between sketch and java lines so we could do a simple reverse lookup. That would make integration with PDE-X easier too i assume.
This has been a long standing problem with PDE X. I've recently been able to achieve not just line matching, but also precise offset matching (character by character). It isn't 100% accurate, but works 95% of the time.

The ANTLR parser generates a full parse tree including line numbers and char-position-in-line information for each token. To generate Java from this i update the parse tree itself and then call getText() on it. So the updated parse tree does have the information we need to map Java back to the sketch code.

In fact, PDE X does a lot of preprocessing itself. PDE X takes the sketch code, converts it into compilable Java code(so that it can be checked for compilation errors) by making the following changes:

• Removal of import statements and moving them to the top of the java source file (using regex)
• Conversion of int(), char(), etc to PApplet.parseInt(), etc. (using regex)
• Replacing '#' with 0xff for color representation (using regex)
• Converts all 'color' datatypes to int (using regex)
• Appends class declaration statement after determining the mode the sketch is in - ACTIVE or STATIC
• Convert all decimal literals (4.5, 3.14, etc) by appending a f. (4.5f, 3.14f, etc) (using Eclipse JDT ASTVisitor api)
• Add 'public' access specifier to all methods which don't have an access specifier. (using Eclipse JDT ASTVisitor api)
  Let me know if any of this could be of use to you.

My preprocessor does all these through a parse tree visitor. Let me finish adding it to the old Java mode and then we can have a look at how useful it is to the PDE-X. From my point of view it is much cleaner than the regex based parsing and having only one preprocessor in the PDE would be favourable too. Ideally we find a way to produce the JDT-AST directly from the ANTLR parse tree without having to re-parse the generated Java code. Or?

[JakubValtar]

Yeah, that's also a possibility to update the code and hand it to someone else.

I think I already fixed some of the issues... I have to revisit my branch and make a list, it's been four months since I touched it. Also, it would be great to talk about the whole system later, what is PDE X doing and what should this preprocessor exactly do and what not.

[JakubValtar]

It's code time! https://github.com/JakubValtar/processing/tree/feature-antlr-4.5

Merged with current master. Ant build works, tests are green, size() is parsed into sketchXxxx() methods, you can run sketches from PDE. No error reporting yet.

Edit: just found Florian's other repository. Will update my branch with improvements he made.

[benfry]

Cool, thanks for the update.

[benfry]

@JakubValtar what's your status on this?

[JakubValtar]

@benfry Sorry about lack of updates, I was really busy recently. I'm finishing multiple projects this week, will have time to work on this next week.

[benfry]

Thanks; let me know how it goes.

[JakubValtar]

@benfry Hi, several things got delayed and sadly I'm not able to work on this one before my graduation mid-June. It would be probably best to tackle this during the summer.

[clankill3r]

@JakubValtar Did you happen to find time yet?