Why a locate hook is unnecessary

[guybedford]

In creating the upgrade path in SystemJS for permitting URLs as module identifiers, it has turned out best to deprecate the locate hook. This may be over-explaining the obvious or dwelling on decisions already made, but coming to this conclusion has taken me a surprising amount of consideration so I'd like to describe the reasoning behind this here to retain some reference for the decision and attempt to leave somewhat reasoned feedback.

The question that started this was whether we should re-introduce the locate hook into the specification. Re-introducing the locate hook will enable normalize to normalize module names into a custom schema that can be defined by the loader implementation and form the string names that are stored in the module registry. Locate then handles the final resolution into URLs that can be fetched.

The justification for considering this was to retain compatibility with AMD-style module loading where we have a baseURL-schema. In this schema, modules names are stored in the registry always as plain names relative to some baseURL. jspm also uses its own schema in the registry to refer to modules such as npm:module@x.y.z.

There is a draw to having the sense of storing these universal schema names inside of the module registry as a portable naming system but I'd argue this lure is mostly one of elegance as opposed to practicality.

I've implemented the baseURL-schema normalization in the current SystemJS, and have been experimenting recently with at least three different complete implementations of normalization of a custom schema alongside URLs (the new requirements of the spec, which completely make sense).

In the end, trying to make a custom schema work alongside URLs in the same registry space, ends up causing more issues, for no practical gain.

Dot Normalization

As soon as we allow both AMD-style module IDs relative to some baseURL alongside URLs, the first issue we hit is the need to define "dot normalization". This basically means that relative normalization needs to be defined for the subset of both URLs and non-URLs.

It's not a lot of code, but it is the first sign here that we're duplicating work.

Non-uniqueness

The next issue we have is the non-uniqueness of our schema. This issue here is that import '/local/path.js' is now distinct and separate to the module at import 'local/path.js' in the scenario where baseURL='/' (one resolves as a name and the other as a URL). This will cause confusion as we are allowing the same unique module to be referred to by two different possible names breaking a key principle of the registry being unique.

Having two ways to refer to the same module is a bug waiting to happen, causing problems for configuration (which variation do we configure?), creating the possibility of a module being executed twice, and interfering with bundling workflows.

Expecting the user to know that they should write import('x') instead of import('./x') arbitrarily is a hard ask.

This leads down a road of trying to catch these uniqueness issues in the normalization pipeline itself, which then ends up becoming URL normalization, followed by a reverse normalization into the schema.

All schemas have the non-uniques problem

Schema non-uniqueness with URLs applies to any custom schema chosen that maps to URLs, not just the baseURL system. Even if we come up with the perfect custom naming schema, as soon as we want that schema to co-exist alongside URL requests we hit these issues.

In order to retain unique identification and configuration of modules, one ends up normalizing from schema space into URL space, and then reverse-normalizing back into schema space at the end of normalize, before resolving back into URLs from the schema in locate, just in order to have our perfect schema names stored in the registry.

Add to this the idea of a configuration space consisting of both schema and URL identifiers as well, and this compounds the problem even further.

One ends up swapping between spaces in such a way that URLs become the primary space anyway, and we're just pretending that the schema is the primary space.

Beyond the baseURL-schema

Another common issue with baseURLs is that when back-tracking below the baseURL, we end up with "normalized" paths looking like "../../module.js", which is really not acceptable for a naming system either.

If we return to the question of what AMD's baseURL schema is really trying to accomplish, the core principle is one of portability of modules, which is completely in agreement with what we should be aiming for. URLs are obviously not a portable naming system for modules (modules can move between environments and hence change URL), so the question is simply how to maintain portability of modules in spite of using URLs?

URLs are the schema

It turns out to be very simple to do this - normalization is seen as the process of converting a "portable module name" into an "environment-specific name". And the most environment-specific name is the URL which we store in the module registry.

The concept that we need to have a registry based on our perfect portable schema is flawed. We still keep our schema if we like - which we can bundle into just the same:

System.register('custom:portable/schema', ...);

Where the name above name is normalized into a resolved name of http://www.site.com/packages/custom/portable/schema.js by the loader when being processed and stored in the registry (bundle names are now treated as unnormalized).

There is no big loss that the registry now contains this value under an environment-specific URL instead of the schema. One can just accept that any lookup into the registry must pass through a normalization phase first:

// lookup a module by its schema name by passing through a simple URL-normalization first
Reflect.loader.lookup(Reflect.loader.schemaToURL('custom:portable/schema'));

If an implementor really wants to use a custom schema, make the schema URL-based and add the implementation to the fetch hook so everything works out well anyway:

import 'custom:///portable/schema'

The other consequence of using URLs is that configuration then always goes through a normalization phase itself:

Reflect.loader.configure({
  module: {
    './some/local/module.js': {
      moduleFormat: 'CommonJS'
    }
  }
});

The above would normalize the above configuration into http://site.com/local/path/some/local/module.js.

The benefit of this is that users don't need to understand the special naming schema - they can just reference modules as URLs exactly as they expect and correctly configure things without needing to have studied the system in detail.

One implication here for implementors is that build systems wanting to use portable naming system schemas need to reverse-map the schemas at build time from URLs in the registry, but that is a very minimal cost and a straightforward 1-1 mapping.

I've yet to hear a single use case that is lost by enforcing that the registry is only to store URLs - the justifications for allowing the registry to store a custom schema seem to cling to dated models due to history, while there are many benefits as described to both implementors and users in enforcing URLs as the schema and keeping the locate hook deprecated.

[guybedford]

One caveat to this - we still allow special modules to be stored in the registry like @empty or @loader. The way we handle this in the latest SystemJS is to not URL-normalize a module if it is a plain name and already in the registry (which can be determined by a simple lookup). This way special environment-independent system modules can still be defined and used. What makes this an exception though is that these modules never get used as a parentAddress to normalize against due to their nature of being set in the registry and not loaded into the registry.

Also to address the plugin argument - http://www.site.com/plugin.js!http://www.site.com/module.js can be fine as a module name - we can handle this by stripping the plugin part in the fetch hook, or when doing relative normalization to this as a parent.

[matthewp]

This is all predicated on something that I don't understand:

In creating the upgrade path in SystemJS for permitting URLs as module identifiers,

Where was it discussed that this was needed or desirable? Is there a link you can provide? I must have missed this discussion. I'll refrain from commenting on it until I read the arguments.

---

It sounds like a "custom schema" is a module name and schemaToURL is a synchronous normalize that doesn't take parentName or parentAddress.

Look-up becomes much less useful without module names. How can I lookup jquery, for example? It's key might be http://example.com/myapp/node_modules/some_framework/node_modules/jquery/dist/jquery.js.

I'm going to have to always run resolve if I use lookup now it seems. That works fine for jquery or something else that is likely in the sites table, but what if the parentName matters to how it resolves?

For example, in some situations you want to create distinct modules based on who is importing them. With module names I can create a pattern like:

import register from 'register';

and this normalizes to register/childName. So you could have (in the registry):

register/foo
register/bar
register/something/else

Where foo and bar and something/else are modules that are all importing register.

Your schemaToURL won't work here because I have to do 2 asynchronous resolves before I can lookup the module. I have to resolve foo and then "register/" + fooURL.

No doubt the loss of normalize can be worked around but the question is why should it be given there is already a specced solution in place.

The assumption of dropping normalize is that modules always map directly to files, but that is not the case. You cannot do something like:

import data from 'package.json!json';
import npmData from 'package.json!npm';

These are distinct modules that have different purposes, but happen to be locating the same file. This means you have to create a custom schema (a module name format) to handle this. You'll have to do this any time you have modules that don't map directly to files; you have to hook both resolve and fetch.

How are you going to handle fetch, with your plugin example? You're going to have to write something like:

var fetch = loader.fetch;
loader.fetch = function(key){
  // http://www.site.com/plugin.js!http://www.site.com/module.js
  var url = key.split("!")[1];
  return fetch.call(this, url);
};

Except now all fetch hooks will be receiving the key "http://www.site.com/module.js" which is not the module's actual key. Hopefully those hooks aren't expecting to have the real key (to store in localstorage, or call lookup later or something) because they don't have it.

[matthewp]

I want to reiterate that last point so that it doesn't get lost:

If there is a need for keys that do not map directly to the url that is fetched it means that any fetch hook cannot trust that the key it is receiving is the module's actual key. This is a big problem. It highlights the need for separate keys and urls.

[guybedford]

@matthewp thanks for the quick response. I've got this work running on SystemJS master so feel free to play around with it there and see how it all works out too.

From what I've heard, it sounds like there is consensus that designing a loader for the web that does not support loading URLs is really not something that can be considered. URLs are fundamental to the web. I don't have any links, just heard notes from a meeting (quite some time ago) where it was decided that the automatic adding of a .js extension is untenable as a default system that does not support URLs, and that importing from URLs should be possible.

Using URLs does not inhibit contextual normalization at all - import 'register' is allowed to have different meaning depending on the parent context (parentURL). We just manage that context through the URL of the parent by mapping all configuration itself into URL space. (schemaToURL is basically just a sync normalize, and permits a parent second argument for contextual normalization).

I mentioned the plugin pattern in my second comment - plugin systems can still extend URLs to include extra syntax for plugins just fine.

I'm not sure I follow your final argument, which seems to rest on an assumption about the fetch hook that has never been formalized, what use case exactly is affected by this?

[matthewp]

From what I've heard, it sounds like there is consensus that designing a loader for the web that does not support loading URLs is really not something that can be considered. URLs are fundamental to the web. I don't have any links, just heard notes from a meeting (quite some time ago) where it was decided that the automatic adding of a .js extension is untenable as a default system that does not support URLs, and that importing from URLs should be possible.

Ah, this is probably the crux of the problem then. If you think about a module identifier as being a pointer to a file then I can see how this requirement would surface. Hopefully we can have this debate again now that the spec is being worked on in the open. I would suggest a mailing list instead of GitHub issues as it's easy to get drowned out by notifications and miss stuff here. Mailing lists are better for discussion and debate.

I mentioned the plugin pattern in my second comment - plugin systems can still extend URLs to include extra syntax for plugins just fine.

I'm not sure I follow your final argument, which seems to rest on an assumption about the fetch hook that has never been formalized, what use case exactly is affected by this?

Yes, my assumption is that the fetch hook does an xhr request for the key (which is a url). I can't see it being implemented any other way.

If you were to override the fetch hook because you have a custom url schema (such as with a plugin system) you will be calling super() with a url that is not the module key. This breaks an important assumption when overriding a hook; that you have access to the module's true key.

[MajorBreakfast]

• The URL approach makes sense for the web.
• I've a feeling that allowing async AND sync normalization (through schemaToURL()) might be problematic as there are use cases that can only be achieved asynchronously (e.g. worker talking to main thread, network request, etc.).
• System.register('custom:portable/schema', ...) (from above example) has an invalid URL. It should read custom://.... Edit: It's valid.
• Can you provide some build tool URL examples? I'm guessing you've already thought about this a lot because it's relevant for system-builder. I'm curious what things need to be considered to make the bundles portable.
• I personally prefer github over mailing lists (a lot)

[jrburke]

I can see a loader design that perhaps does not specify a locate hook, with fetch() taking care of that internally, and the loader just having a normalize or resolve. Although something like a .locate() on a module meta for a module has been useful in AMD systems, and I expect that utility to carry forward. But leave that on the side for now.

I still think it makes sense for the ID keys for modules internal to the loader to be module IDs instead of URLs though.

Modules are units of functionality, like functions. The module loader is like a scope object for those code units, and the identifiers for those code units should just be something like an identifier, not a URL.

For module bundling cases and some plugin loader resource IDs that are not URLs, the provider may not be at an actual URL. Best to keep the IDs separate from URLs to enforce that conceptual distinction.

---

As to the dot normalization for CJS/AMD IDs, this exists because objects should not name themselves in source form. So sub-pieces inside a package need to be able to reference other parts in that package. Instead of seeing the module ID parts as a file path/URL, see it as referencing different parts of an object with a nested property/object structure.

There is a package that provides button. There can be a button/dropdown inside of there. If button/dropdown needs to refer to functionality in button, the '../button' allows dropdown to refer to that functionality without knowing the global name.

Using '/' instead of '.' for the parts in a whole package object definition makes it easy to use '../' or './' without knowing the global object name or needing to to invent a new syntax that does not conflict with '.', if '.' was instead used for separating object parts (so button.dropdown vs button/dropdown.

It also turns out to be helpful for a simple ID-to-path conversion, if a file fetch is needed. That just helps reduce the number of indirection layers, so creates a simpler overall system, but not the primary goal of the IDs.

---

If the argument is that not being able to use URLs for IDs as something that is not fundamentally of the web, I think this is assuming too much on the JS module system related to the old world of script src tags. As illustrated above, modules IDs are a replacement for object.sub.value references that allow async tracing and resolution before triggering execution of the module body.

Perhaps the "URLs must be supported" comes from thinking that script tags should be usable in an ES module system, but I think that is not the best way to look at script src tags in a module system. Some notes on that:

Script src tags are separate primitives for loading script and evaluating it. Consider it an XHR call with an eval() in global script space, and extra semantics for blocking rendering. Dependencies are expressed via a linear global list of script src tags.

In contrast, modules are about units of code that specify their dependencies not with an implicit global identifier but with a local identifier tied to a string reference, where that string reference allows some level of indirection and resolution.

Sometimes that indirection means translating that ID to a URL to load, sometimes it means just grabbing that module definition by the normalized string from a registry populated by bundles modules. Modules are a tree structure that (to work in the browser) are asynchronously discovered.

That async discovery of layers in a tree model fundamentally does not work well with the linear, and sometime render-blocking, model of script src tags. There is a set of problems when using an AMD loader alongside manually coded script src tags that are just avoided by not using the script src tags at all. The conceptual models are just different, and there will be less confusion by keeping them separate.

[MajorBreakfast]

@jrburke Your last post is a bit verbose. Just say everything once, please.

Part 1

I can see a loader design that perhaps does not specify a locate hook, with fetch() taking care of that internally

Fetch does nothing more but loading.

code units should just be something like an identifier, not a URL

It is an identifier. (What else would it be?) Also, one of the the most frequently used ones.

the provider may not be at an actual URL

It might not be a web server. But URLs work for all kinds of sources (e.g. android content providers). Just use a custom schema.

Part 2
That is supported and yes relative module paths are useful.

Part 3
SystemJS can load modules that export a global; so @guybedford is definitely aware of these points. As you say script tags are something different to modules. But so are URLs and script tags. URLs are an essential building block of the web. So, I say that this won't cause confusion because we already use them for all kinds of things. However, "Having two ways to refer to the same module is a bug waiting to happen" (from @guybedford's post above).

[jrburke]

Part 1
It is an identifier. (What else would it be?) Also, one of the the most frequently used ones.

I meant more like the JS language concept of Identifier. Sorry for not making that clear.

It might not be a web server. But URLs work for all kinds of sources (e.g. android content providers). Just use a custom schema.

URLs are great for global systems, like addressing content across an OS, that need disambiguation, android content providers being one of those. However, module use is more about local context, and it is more useful to have names that represent contracts than actual URLs. My 'jquery' could actually be provided by 'zepto'.

It is good if a dependency indicates via a globally resolvable identifier (like a URL) in the package.json where to fetch a a local identifier dependency if the local project does not have an opinion, but also allow the local project to override that with a preferred local value.

There is a very strong analogy with custom elements in HTML, where they are tag names, but the resolution of that name to provider is separate from that name, and each tag may not be delivered by individual URLs.

Part 2
That is supported and yes relative module paths are useful.

This was in response to @guybedford's original post section on Dot Normalization. The duplication is not really about a path duplication, but about being able to reference different concept parts in a localized way, without knowing the global ID, and if it looks like duplication, it is more about flattening the number of transforms from ID to path (when a path is required for fetching) than being specifically a URL concept.

Part 3
URLs are an essential building block of the web.

They are, because they help disambiguate across a global space, across all web sites. It does not follow though that it is therefore what a module loader needs to use as internal keys. And given the use cases of module bundling, loader plugins IDs that are not URLs, and even other examples in the web space, like custom elements, it is much more flexible to keep the URL concept solely related to when fetching is needed, when global disambiguation is needed (like, the browser cache is global across all web sites, so a URL is needed in that case).

[johnjbarton]

it is much more flexible to keep the URL concept solely related to when fetching is needed,

Is every module registered the result of fetch()? So that every entry in the registry corresponds to a URL used in fetch()? Not using this equivalence would seem to make implementation more confusing.

Doesn't the issue of "module ids" vs URLs relate to module specifiers rather than the registry keys?

[matthewp]

Is every module registered the result of fetch()? So that every entry in the registry corresponds to a URL used in fetch()? Not using this equivalence would seem to make implementation more confusing.

No, the loader.install (previously loader.set) API allows you to create entries arbitrarily from any object in memory. This is where URLs as keys really breaks down, modules are not all files.

[MajorBreakfast]

My 'jquery' could actually be provided by 'zepto'

@jrburke In this case jquery normalizes to the URL of zepto. That URL can then be used as key.

[johnjbarton]

This is where URLs as keys really breaks down, modules are not all files.

Well URLs do not identify files so why does "modules are not all files" relate?