We will focus on how a JavaScript module can be loaded, and get a gist of what all module loaders do.
If you are new to modules in JavaScript I would recommend to start with this article by Addy Osmani.
Do you use Browserify, WebPack, jspm, StealJS, or SystemJS? This article will provide a peak under the hood at the layer on top of which those libraries are implemented.
The nature of a human brain is that it cannot deal with a lot of objects at the same time (Miller’s Law). If you are going to build a large JavaScript application, you should stop, remember this limitation and start thinking in terms of modules.
Modules are a way to organize your application. Simply break your functionality into small pieces, focusing on how they will work with each other, and then assemble them together. A module could be seen as a black box with a clear, simple API. Modules commonly depend on other modules.
In today's modern browsers, there is support for not only writing and loading modules, but performing various low-level tasks around loading and executing the module. This article will explain the current standard for module loaders - their lifecycle and their hooks. A future article will show a detailed example for how to use these hooks.
Pop quiz! If you have modules written in CoffeeScript and CommonJS, is it possible to use them both within an ES6 application?
The answer is “Yes to both”. This article will explain how this is possible using Module Loaders.
Table of Contents:
1. Module Loaders
For modern web development, the following module standards are available:
- AMD - Asynchronous Module Definition, good for loading modules asynchronously (dynamic import).
- CommonJS is widely known for being used in NodeJS. It is good for synchronous module loading (static import) which works well for server-side scripting.
- ES6 - WHATWG’s module standard, is still a draft, will become the official standard for JavaScript modules. It allows both static and dynamic imports.
They have different but similar APIs and serve the following purposes:
- define a module (module syntax);
- load a module.
In this article we will focus on how a module is loaded and get a gist of what all module loaders do.
A module system aims to simplify your development: you can focus on your current module and only have to care about what modules you directly depend on. The module loader does all the heavy lifting:
- performs the loading task,
- acts as a dependency manager
- and maintains a Module Registry (an object that keeps track of all modules and stores theirs source code along with other meta data)
Let’s look at how WHATWG specification describes what a module loader should do:
The JavaScript Loader allows host environments, like Node.js and browsers, to fetch and load modules on demand. It provides a hookable pipeline, to allow front-end packaging solutions like Browserify, WebPack and jspm to hook into the loading process.
The loader is a system for loading and executing modules, and there is a way to participate in the process. There are several Loader hooks which are called at various points in the process of loading a module. The default hooks are implemented on the Loader.prototype, and thus could be overridden/extended.
2. Loader Pipeline
In the diagram you can see the different stages that the Loader passes through:
Note: WHATWG (ES6) module standard defines four stages: “Resolve” replaces “Normalize” and “Locate”.
Normalize Phase
During the Normalize phase the Loader converts the provided name into a Module Identifier that will be used as a key to store the module’s data in the Module Registry. The given name could be a relative path to the resource, it also could contain a shorthand mapping to a certain path, or any other logic that a particular Loader implementation provides.
Locate Phase
The Locate phase serves to determine the final resource address that the Loader will use to fetch the resource from. It is either a URL (if the host is the browser), or a path (if the host is a NodeJS server).
Fetch Phase
During the Fetch phase Loader fetches the resource by provided address. It could be that module’s body is provided to the Loader directly, in which case this phase will be skipped. The result of this phase is a string with the source code of the module.
Translate Phase
The Translate phase is probably the most interesting, because pure JavaScript is not the only way to program for web. There are a lot of popular options: TypeScript, CoffeeScript (with all its dialects), Elm, Flow, next generation JS standards, etc. Technically, there is no limit for what could be used. You can use any programming language if you can provide a JS translator that will compile your code into JavaScript.
Instantiate Phase
During the Instantiate phase module’s dependencies are loaded and linked together, then the module gets evaluated.
3. Loading hooks
Now let’s see how the process could be customized. For each of the stages there is a hook, which is a method that will be called with certain arguments. A hook can either return an immediate result or a promise.
When you override the loader’s hook method you can also call the original method. In this case you will have to pass it the parameters as defined by the hook’s signature. Alternatively, you can just return the expected result.
For an example we will look at how module my.js imports module math.js. Both are saved in the same directory called "utils" (look here for ES6 module syntax):
Normalize: (name, referrerName, referrerAddress) → normalizedModuleName
The Module Loader calls this hook by passing three arguments: name, referrerName (the normalized name of the module that initiated the import), referrerAddress. The result of the call should be a string, which is a normalized module name. It is usually a path to the module file or folder from the root of the project. This way it uniquely identifies a module within the project.
Locate: loadRequest → loadRequest
This hook receives the loadRequest object, in which the name property is a normalized module name. It adds the address property to the object, which represents the resource address. It is called immediately after normalize unless the module is already loaded or loading (the same applies to the rest of the hooks).
Fetch: loadRequest → sourceCodeString
Receives the loadRequest object with address property, and returns a string containing the source code of the module.
Translate: loadRequest → ecmaCompliantSourceCodeString
Receives the loadRequest object with a source property, which is a result of the previous step. The purpose of this hook is to translate the source code into ECMAScript. If the code is in another language (CoffeeScript for example), this is when the transpiling would happen.
Instantiate: loadRequest → instantiationRequest
In this hook, the translated source is instantiated. It receives loadRequest with the source property as a translated source. It returns an instantiationRequest object, which has two required properties. The value of the deps property is an array of strings. Each string is the name of module dependencies. The value of the execute property is a function which the loader will use to create the module.
A module is evaluated during the linking process. First, all of the modules it depends upon are linked and evaluated, and then passed to the execute function. Then the resulting module is linked with the downstream dependencies.
Finale
It is worth mentioning that the current draft of ECMA-262 does not include the specification for module loaders since it was removed in 2014. You can still find it in the archive. It is a very useful resource - SystemJS' and StealJS' implementation was based on that draft. The new draft is now being developed by WHATWG and is not completed yet.
To recap, we looked at what a module system is, what standards are available for modern web development, then we dove into the loader pipeline and saw how it could be extended. In the next post we will write a simple loader’s plugin for translating CoffeeScript on the fly (no need to precompile, and you can even debug in browser against the original source)!
development
