Extending Coffee using annotations or comments?

[StanAngeloff]

I've been thinking about this for a while. A lot of people (incl. me) expect a lot of Coffee. Some of us are happy with what the core has to offer, others want more. Instead of adding new features to the core, why not make it extensible? Unfortunately re-generating the parser is expensive and as attractive as it may seem, I don't think it's possible. Many languages, both static and dynamic, have found a way around this by introducing annotations/attributes. A classical example would be enforcing type checking on function parameters (let's just go with it):

sayHello: (person) ->
    throw new Error 'ArgumentError: person not a Person' if not (person instanceof Person)
    puts "Hello ${name.getName()}!"

[Type name: 'person', enforce: Person]
sayHello: (person) ->
    puts "Hello ${name.getName()}!"

Coffee will completely ignore these in the generated code, unless there is a registered hook that can handle the generated AST. In the above example, Coffee would attempt to load a file 'type.coffee' and attach a processor from it to the sayHello node. At compile time, the processor can decide what to do.

I realise this may be accomplished using comments, so if we can think about a way of letting outside extensions play nice with Coffee that would be great. Perhaps another command line option to register a bunch of files which will respond at compile time:

# MyType name: 'person', enforce: Person
sayHello: (person) ->
    puts "Hello ${name.getName()}!"

> bin/coffee -c --register-addon addons/my_type.coffee input.coffee

I recall Jeremy already added a method to traverse the tree, so why not make it pluggable as above?

/rant almost over

EDIT: rant on: Having thought more about this, the easiest way to allow extensions would just be to include them and pass in the AST generated at compile time and let them do whatever they want with it. I would think some people may want certain extensions enabled permanently so perhaps Coffee can look in the users' home directory for installed and enabled exts?

rant over

[jashkenas]

I would love to make CoffeeScript an extensible language. I think we're already a good part of the way there, simply due to the fact that the implementation is written in CoffeeScript.

I don't think that we should limit language extensions to a particular syntax -- be it your proposed brackets, or Python's @annotation, or fancy comments. You can already fake annotations with higher-order functions, like so:

type: (klass, func) ->
  (arg) ->
    if not arg instanceof klass
      throw new Error "ArgumentError: $arg not a $klass"
    func arg

And then use it to enforce types like this:

sayHello: type Person, (person) ->
  puts "Hello ${person.getName()}"

So I think that with language extensions, we should be aiming to enable radical changes to the syntax. This means exposing hooks for the lexer -- to recognize new tokens, new hooks for adding rewrite rules, and hooks for the addition of new Node classes to the syntax tree.

The wrinkle, as you say, is the parser. You can't change it on the fly because it has to be compiled by Jison in advance. We might have to just sneak around it by creating an "Extension" token, whose contents are passed straight through the parser untouched (as an Expression), and dispatched to your Node at the other end, where you can process them as you see fit.

Does anyone have experience with extensions in other languages? (Aside from Lisp macros, which don't change any syntax.)

[jashkenas]

Alright, Stan. I've pushed a commit that allows you perform basic syntax extensions to the language. Anything you can lex, and that can serve as a valid Expression, is good to go. We're not exposing a special api for it, not really, just making the CoffeeScript internals accessible.

It works like so: You extend CoffeeScript with a new lexing function. It works the same as all of the other tokenizing methods on Lexer (return true if you match, increment @i, push a token), and runs with the highest priority. As you create tokens, you can either generate existing CoffeeScript tokens in the form you like, or you can tag an EXTENSION token. Extension tokens are passed through the parser verbatim, and their value is used as a Node during code generation. So you can subclass BaseNode and do anything that the rest of the Nodes can do, if you choose.

Here's the test case, which demonstrates an extension that turns --wordgoeshere-- into a literal that means "split this into individual characters".

http://gist.github.com/327529

(By the way, I talked to zaach a bit about modifying the parser on the fly, and he agrees that there's no obvious way to do it.)

Let me know if this scheme works for your plans.

[StanAngeloff]

Great! This should open up the door for anyone interested in extending Coffee to add their own flavour to the language. I think that's what we should call extensions -- flavours! Lol I'll have a proper look at them soon.

I have another crazy idea about using Coffee to produce PHP, but I've still not convinced myself I can do it in my spare time. Flavours would be very handy when/if I get to doing it.

[grayrest]

Seems like this is most of the way there to a full macro system. If you compare Dylan's to your gist borrow some ideas and interpolate, it almost looks like you could write your gist like:

macro { '--' ?identifier:str '--' } ->
     '${str}'.split('')

Which is a messed up jumble of dylan+coffee syntax but the idea is that the bit between the {} is whitespace separated tokens and the indented part is essentially an indention-delimited string, so the interpolation is just string interpolation. It'd essentially work out to be the contents of your gist but the syntax is shorter. Of course, I'm plastering over all the complexities involved, but it at least looks like it could work in the short amount of time I've spent thinking about it.

Edit: I'm not arguing that macros SHOULD be implemented, just pointing out that this looks awfully close and linking to Dylan, which I haven't used but is the only language I know of that has first class language extensions but not s-exprs.

[henrikh]

I really like that approach grayrest. I also think it would completely redefine what CoffeeScript can do.

[matehat]

As pointed out in #257, we need an clear way to hook a given extension to CoffeeScript though.

[jashkenas]

grayrest: Thanks for the link. That's some very interesting reading. Here's more documentation:

http://www.opendylan.org/books/drm/Macros_Overview

And also, an alternate perspective:

http://www.fun-principles.info/slot/site/dylan/alpha/openpoints/Dylanmacrosareakludge..html

To summarize, Dylans' macros are limited to token matching, similar to a rule in our Rewriter that's constrained to only matching a literal sequence of tokens. To quote:

Some implementations and a future version of the Dylan language specification might
allow macro expansions to be produced by compile-time computation using the full
Dylan language and an object-oriented representation for programs. Such a procedural
macro facility is not part of Dylan at this time.

With lexing rules, you can actually create arbitrary new syntax, not just macro expansion for tokens that CoffeeScript already knows how to recognize. It's a little more work, but I think that language extension should be reserved for the cases where you have a major syntactic improvement to offer -- for most other cases passing closures should suffice.

I'm all ears for a nicer API for language extensions, both for traverse-based rewriting of the AST, and for lexer/rewriter rules. I think it should be driven by real examples though. Let's let the actual proposed extensions define what the interface should look like for expressing them, in future tickets.

Closing this one.

[StanAngeloff]

I am thinking of taking a stab at implementing Boo-like macros in Coffee. More in issue 313.

[GeoffreyBooth]

Relates to #4540 and #4572.