TypeScript extensibility

[billti]

TypeScript Extensibility

This is an umbrella issue for tracking the design of an extensibility model for TypeScript.

There have already been a number of issues opened with regards to supporting Angular2 via such a model, (especially for richer template editing support), so this will be the initial focus. This not only ensures rich support for a key TypeScript scenario, but by solving real problems, it also ensures we are not designing in a vacuum.

High level problems

This section captures an overview of the problems to be solved. These are not of equal priority, and not all features below may be implemented, but are captured here for completeness, and to be considered in the broader picture.

• Syntax highlighting: Template strings inside a TypeScript file currently show as simple string literals, yet ideally would be colored in a semantically meaninful way.
• Additional file-types and structure: Angular templates may live in a separate file, containing the HTML-like template contents. For React, TypeScript added a new file extension (.tsx) and matching grammar/parser changes for this file type. Ideally this would be an extensibility point, not baked into TypeScript.
• Syntactic and semantic diagnostics: Both the compiler and the language service should be able to surface domain specific errors in areas such as syntax, type usage, deprecation notices, etc. Such errors should map to original source locations, not intermediate artifacts.
• Rich IDE features: Statement and member completion, signature help, find all references, go to defintion, rename, etc. should work within the template contents as expected.
• Custom commands: Some plugins may provide editors additional functionality not exposed by the existing language service API (e.g. an editor visualization for the component hierarchy). A plugin should be able to expose additional commands for such usage.

Challenges

This section captures non-trivial challenges to be addressed, either with the problem domain in general or the current TypeScript architecture.

• Syntax highlighting: Several editors, (e.g. Sublime Text, VSCode, etc.), use a TextMate bundle for syntax highlighting. This is generally a static file describing the grammar for a file type, usually detected by file extension. Expanding the grammar via a plugin is a challenge. Even if statically augmenting the grammar, detecting when a string literal is an Angular template or not, or this .html file is an Angular template or regular .html file, is also challenging to do at parse time (and often depends on type information for inline template, or path resolution for external templates, to determine definitively).
• Syntax highlighting: TextMate aside, other editors (e.g. Visual Studio) do syntax highlighting via other means (such as calling the TypeScript classifier), which would need to be augmented somehow also.
• Program structure: In order to provide some of the above functionality, it is desirable to generate an intermediate representation (e.g. containing a "compiled template") and ask TypeScript questions about this code, then map the results back to the original source. However, the representation of the program that the compiler or language service has is immutable. It is not possible to receive a request about a source location, generate an intermediate representation of it, update the program with the generated code mid-request, then ask TypeScript questions about it before responding.
• Loading: TypeScript runs in environments besides Node.js (e.g. in-process in Visual Studio, or the tsc.exe command-line compiler), thus a plugin can not make assumptions such as being able to call require or take dependencies on other Node.js packages.
• Performance: As the program is immutable, a new program is created on every edit. Most language service operations require that the program is bound and type-checked, which is often done on-demand when a question is asked. For acceptable performance, (faster than 50ms response time), processing should be kept to a minimum and done entirely in memory where possible (e.g. avoid serializing/reading generated artifacts on disk such as temporary code, source map files, etc.).

Angular specific challenges

• Ambiguity: What if two different components have a templateUrl that resolves to the same file on disk? In which context should TypeScript evaluate that file?
• Determination: If a user opens an HTML file, how can the editor determine if this is an Angular template rather than just a regular HTML file without loading the full TypeScript program and resolving all the templateUrl properties?
• Path resolution: What if the path on disk doesn't match the path at runtime? Is some type of baseUrl or config object needed to map paths from the TypeScript source to the template files?
• Dynamic content: TypeScript needs to be able to statically analyze code that may be dynamically generated at runtime. For example, how to handle an inline template that reads <div>${getStart()}<foo [prop]='expr'/>${getEnd()}</div>, or analyze the component with the directives given as directives: getMyDirectives()?
• Name resolution: Within an expression in a template, scoping is quite different to normal. All instance members are available as top level names (i.e. no this. needed), and the usual global members aren't available (i.e. can't reference Date.now, parseInt, or similar). Thus resolving names and providing completion lists requires quite different logic to the usual TypeScript behavior.
• Micro-syntaxes: Within Angular templates, expressions are mostly JavaScript expressions, but not quite. For example, certain operators have different meanings, arguments are passed differently when using pipes, etc. See https://angular.io/docs/ts/latest/guide/template-syntax.html for more info.

Current work

The below is currently experimental work to spike various approaches.

• TextMate grammar to provide syntax highlighting for inline templates: https://github.com/billti/TypeScript-TmLanguage/tree/ngml
  • Note: This current uses a heuristic of determining if a template string is a value for an object literal property named template, or is preceded with a /** @html */ comment. See https://github.com/billti/TypeScript-TmLanguage/blob/ngml/TypeScript.YAML-tmLanguage#L507
• A fork of TypeScript with a rudimentary plugin model (plugin source lives within the codebase currently) and some initial feature support: https://github.com/billti/TypeScript/tree/ngml
  • TODO: Update with the latest TypeScript and Angular2 releases.
• Usage in VSCode:
  1. Replace the typescript.tmLanguage file under the install location at Contents/Resources/app/extensions/typescript/syntaxes with the version from the TypeScript-TmLanguage repo above (from the ngml branch).
  2. Fork the TypeScript repo above, checkout the ngml branch, and build.
  3. Open the VSCode global settings, and set the typescript.tsdk property to the build location in the prior step (e.g. /src/typescript/built/local).
  4. To get completions within the template on characters such as <, [, etc. open typescriptMain.ts from the Contents/Resources/app/extensions/typescript folder under the VSCode install location, and change the line that calls registerCompletionItemProvider to read vscode_1.languages.registerCompletionItemProvider(modeID, completionItemProvider, '.', '<', '(', '[', '\'');
  5. To make changes edit the local TypeScript repo, rebuild, and relaunch VSCode.
• Limitations:
  • Currently there is no dynamic plugin model. The plugin is compiled into the TypeScript service. To minimize build changes, all plugin code is lumped into one file currently (src/services/plugin-ngml.ts), as are some basic unit tests.
  • It currently uses its own rudimentary template parser. I haven't spent the time to figure out how to reuse the code from angular2/src/compiler (if possible)
  • All attribute values should be in single quotes. Unquoted or double quoted doesn't parse properly in the textMate grammar yet.
  • Interpolation inside attribute values is not wired up yet, i.e. <foo name='Hi {{name}}'></foo> won't work yet.
  • Attribute bindings (i.e. dotting off of attr) don't work yet, i.e. <td [attr.colspan]='expr'>
  • Only expressions where the first token is a member bind correctly, i.e. mem1(mem2) + mem3 will only bind mem1 to the class instance member correctly.
  • Components within components do not work yet (in fact, none of the directives values are resolved currently).
  • Special directives, such as *ngFor='#item of items', aren't implemented yet.
  • Angular specific syntax, such as pipes or the Elvis operator don't work yet, i.e. mem?.value | mypipe:"arg1"
  • Naming conversion still uses snake-case, and needs to be updated for the Beta release changes to attribute name mapping.
  • Rename isn't working yet (but is simple since the Beta change to simplify name mapping and is nearly done)
  • Still to figure out work for features such as outlets, etc.
• Current features
  • Completions and errors on HTML tags (using a basic list of HTML elements)
  • Completions on member expressions
  • Syntactic and type errors on member expressions
  • Signature help and tool-tips on expressions
  • Data and event binding using the [prop] or (event) syntax
  • Introduction of scoped template locals using the #name attribute syntax
  • Goto definition on identifers within template expressions
  • The below shows an example of the plugin in action

• TODO
  • Detail the architecture and which of the above challenges informed which choices
  • Figure out how to plug in to the compiler as well as the language service for the errors as build time as well as design time

Existing related issues:

• [typescript checking frontend template files #5151] typescript checking frontend template files
• [[tsserver] custom command (ex: to collect Angular2 @Component/selector) #5730] [tsserver] custom command (ex: to collect Angular2 @Component/selector)
• [Design Meeting Notes, 11/20/2015 #5740] Allow evaluation of a detached buffer in a specified context (angular2 templates)
• [Language Serivice support in Angular 2 inline templates #6477] [tsx] Angular 2 inline templates

[angelozerr]

@billti your demo is very impressive!

I tell me if your work could take care of HTML file too. You provide HTML, Angular completion for @Component/template, but what about @Component/templateUrl ? I mean:

• person.html

<h1[style. // here Ctrl+Space shows FontFamily

• person.ts:

@Component({
  templateUrl: " // here Ctrl+Space shows person.html
})

[billti]

This is partly the first bullet under challenges above, namely that we can't just assume any .html file is an Angular template, and you don't want to load the TypeScript language service and scan all source to see if it is anytime someone opens an HTML file. If it had a unique extension you could identify the grammar and language service based of this (e.g. *.ngml or similar).

With a unique extension that we know definitely is an Angular2 template, then editors could automatically apply the correct grammar (i.e. TextMate bundle in the case of Sublime or VSCode), and load the correct language service (and provide intellisense in the areas you outline above).

[angelozerr]

@billti if I have understood your comments, the challenge is to detect if an HTML file *.html is an angular template or not? For me this information is about the project nature. This nature could be detect :

• with an user settings (ex : for Eclipse you can set a nature in the .project)
• or could be detect:
  • search depdendencies in the package.json
  • or search if the project contains a node_modules/angular2 folder
  • or some other thing (ex : tsconfig.json, etc)

With *.ngml it fixes the problem with project nature, but not the full scan of typescript files to retrieve the well class which defines @Component/templateUrl

I'm a little afraid with this case for performance problem to have to scan the full typescript files. Or perhaps I have not understood something?

[ericmdantas]

That'd be awesome!

[billti]

@angelozerr That's half the battle. The challenge is more of a catch-22, and is something like "how do I detect if I need to load the language service, without loading the language service to do the detection logic".

For performance reasons many editors do not load the language service unless needed. Currently, an .html document never needs the TypeScript language service, thus it is rarely - if ever - loaded when opening an HTML document. Assuming the above logic would live in the TypeScript (or one of its plugin's) code, then you would need to load the TypeScript language service on every HTML document you open to detect if it is an Angular template. Does that make sense?

Maybe that's a trade off we'll have to make. It's just not one to take lightly.

[LPGhatguy]

@billti How much of the detection code needs to be baked into the full-fledged language service? Would it be possible to build a service whose sole purpose is to detect what a file is, but not do anything with it?

It could share source with the language service, but it would be a smaller process with an otherwise smaller source base.