import declaration should not include semicolon or require line end

[aeschli]

See microsoft/vscode#30527

The coffeescript grammar includes the JavaScript grammar.
With the following example:

`import Ember from 'ember'`
# foo bar baz
if true then console.log('qnx')

the JavaScript grammar is used to tokenize what's inside the backward quotes. Problem is that it does not return to the initial state after processing 'ember'. It expects a semicolon or the end of the line. Because of that, the JavaScript grammar scope remains and the rest of the file is colorized also with the JavaScipt grammar.

For supporting embedded cases like this, it would be great if the import declaration rule could end to end with the string literal or any non-expected symbol.

[sheetalkamat]

The import rule currently ends with either semicolon, end of the line or new line. Shouldn't the rule any ways end at lookahead of ` since the rule needs to be applied only to whats inside of template string shown in the issue? I think we are doing the right thing here.

[aeschli]

The problem is that the JavaScript grammar gets to evaluate its rules on the full line, including the ending `. It's not just the content of the template string.
The import statement is matched, and the engine stays in the 'import' rule until it sees the line end.
Only then the coffee end rule for ` is active again, but ` has been consumed already. So the end rule will not match until the next occurrence of a `.

TextMate grammars lack a way of expressing embedded content. They just include rules from other grammars, but the problem is that these grammars don't know about the stop tokens.

Clearly, the embedded grammar (JavaScript, in this case), can't be blamed for that. But to play along, best is to exit rules as soon as possible.
microsoft/vscode#34224 is a similar problem.

[sheetalkamat]

@aeschli That doesnt seem right though... Since then any begin end rule would need to account for where it gets embedded and what to look for in the end rule? Eg. writing any thing inside say template string would need to account } as end of content and that just doesn't seem right.. There are other grammars that use end of line as end rule as well.. how will that work?