Skip to main content

News

Topic: CommonJS modules (Read 2801 times) previous topic - next topic

CommonJS modules
This has been discussed on github for sphere 2.0: https://github.com/sphere-group/pegasus/issues/9

I think all the current engines should move to implement require as a built-in as soon as possible, and Sphere libraries should begin to be shipped as simple npm-like packages that can be used with require. With a forked `npm` using a custom package index we can make Sphere libraries easy to install and use, with proper dependency management. In short, modern.

We can still shim support without built-in require for backwards compatibility - writing a module loader isn't that complicated, and we're only going to be loading JavaScript.

I have been working with minisphere and Duktape has half of it built in - it performs path resolution and you have to supply it with the module. I wrote a loader that enables use of require('module') that will look for four things:

1. module.js
2. module.json
3. module/index.js
4. module/package.json

The final check is for packages. We open the package.json and find out what the main file is called, then load that. We also have a convenience loader for JSON files.

It will look for files in your scripts directory, then in the modules directory. This is the directory that Sphere libraries would be installed to by the package manager.

Code: [Select]

(function() {
  var extensions, fileExists, jsHandler, jsonHandler, packageHandler, paths,
    indexOf = [].indexOf || function(item) { for (var i = 0, l = this.length; i < l; i++) { if (i in this && this[i] === item) return i; } return -1; };

  jsHandler = function(filePath, file) {
    return file;
  };

  jsonHandler = function(filePath, file, exports) {
    var json, key, value;
    json = JSON.parse(file);
    for (key in json) {
      value = json[key];
      exports[key] = value;
    }
  };

  packageHandler = function(filePath, file) {
    var json, script;
    json = JSON.parse(file);
    file = OpenRawFile(filePath.substr(0, filePath.length - 12) + json.main);
    script = CreateStringFromByteArray(file.read(file.getSize()));
    return script;
  };

  fileExists = function(path) {
    var check, checked, entries, parts, testPath;
    if (path.substr(0, 3) === '../') {
      path = path.substr(3);
    }
    parts = path.split('/');
    checked = [];
    while (true) {
      testPath = checked.join('/');
      entries = GetDirectoryList(testPath);
      entries = entries.concat(GetFileList(testPath));
      check = parts.shift();
      if (indexOf.call(entries, check) >= 0) {
        checked.push(check);
        if (parts.length === 0) {
          return true;
        }
      } else {
        return false;
      }
    }
  };

  extensions = {
    '.js': jsHandler,
    '.json': jsonHandler,
    '/index.js': jsHandler,
    '/package.json': packageHandler
  };

  paths = ['../scripts', '../modules'];

  Duktape.modSearch = function(id, require, exports, module) {
    var extension, file, filePath, handler, i, len, path, script;
    for (i = 0, len = paths.length; i < len; i++) {
      path = paths[i];
      for (extension in extensions) {
        handler = extensions[extension];
        filePath = path + "/" + id + extension;
        if (fileExists(filePath)) {
          file = OpenRawFile(filePath);
          script = CreateStringFromByteArray(file.read(file.getSize()));
          return handler(filePath, script, exports);
        }
      }
    }
    throw new Error("Cannot find module " + id);
  };

}).call(this);


Duktape is currently limited and assigning module.exports doesn't work, only properties on exports. It also has limitations that mean we can't load package files from a subdirectory in the package. A replacement is being considered to make it more flexible, but this works otherwise. We can still shim instead of using Duktape until then.

Let's make this the first step to Sphere 2.0. A real module loader and a real package manager.

  • Fat Cerberus
  • [*][*][*][*][*]
  • Global Moderator
  • miniSphere Developer
Re: CommonJS modules
Reply #1
I'm still confused as to what modules actually are.  There's a cloud of jargon surrounding them that makes them very unapproachable.  "Packages", "modules", "npm"... honestly, most JavaScript technologies are like this.  This is why I had no clue what Ajax was at all for years, and to this day I still don't fully understand the concept.
miniSphere 5.0.1 - Cell compiler - SSj debugger - thread | on GitHub
For the sake of our continued health I very much hope that Fat Cerberus does not become skilled enough at whatever arcane art it would require to cause computers to spawn enourmous man eating pigs ~Rhuan

Re: CommonJS modules
Reply #2
I'm not against this idea, but I also don't think it solves a real problem that we have. I also consider it a complex solution to a simple problem, but it is the most common solution unfortunately.

Re: CommonJS modules
Reply #3
Lord English, modules are pretty simple - essentially a module works exactly the same as a script included with EvaluateScript, with a couple of differences:
   
   * The module is run in a new scope, and any variables defined do not leak into the global scope.
   * To export an API, a module has a 'magic' variable called exports. This is what is returned when you require a module.
   * A module can be a directory of scripts instead of just one script. In this case, the system reads package.json in the directory to find out what the main script is and includes that, or it defaults to index.js.
   
   A package is a distributable module that can be installed by a package manager like npm. A package includes a package.json file that describes the package and what dependencies it has. Using this information, the package manager can automatically install anything else necessary for the package to work.
   
   So, the benefits as I see them:
   
   1. Modules prevent naming conflicts

   Including a library currently requires creating your objects/other variables in the global scope so they can be accessed by game scripts. This can cause conflicts with other libraries or stuff in your own game.
   e.g. if I want to use Link.js, I have to make sure I don't have my own variable called 'Link'. I also have to make sure every other library I'm using also doesn't define 'Link'. If they do, I would have to modify those libraries and can no longer cleanly update them when a new version comes out.
   Instead of having 'Link' be a global, you can just require('link') wherever you need it. Any libraries that have their own 'Link' variable are unaffected.
   
   2. Packages allow better dependency management
   
   Without packages, if you include a library that depends on another library, you have to know that the dependency exists, and you have to know which version of the other library is expected. Alternatively, the library can include its dependencies inline, which causes conflicts if you have more than one library with the same dependency.
   With a package system, the dependencies are specified in the package.json and can be installed automatically. If two modules depend on the same package, only one copy can be installed and they will both use it.
   
   3. Packages allow easy updating
   
   All your installed packages are specified in your projects own package.json as dependencies. To update something, you just bump the version number and run the package manager to install it again.
   
   4. A package index makes finding libraries easier
   
   Instead of searching the internet, forums, etc, to find everything on different sites, we can create a single package index with every sphere library that's easily searchable. We also make installing libraries easy with a package manager: just run "spm install link" and now you can require('link') in your script.
   
   5. We can re-use packages from other JS package managers
   
   In my own project, I have included the lodash node package. Because it supports CommonJS, it works out of the box. (I only had to define a 'global' global to get it to think it was running on node).
   
   
   A lot of the problems are not *big* problems with Sphere games because of the lack of different libraries - generally you're going to be OK and not have many conflicts. But if Sphere is going to gain wide usage, they are problems that will need to be solved, and I think it's better to do it sooner rather than later.
   
   Note that backwards compatibility for libraries shouldn't be a problem - you can check for the presence of "require" in your library and set globals if it's not there. Your library will continue to work in the normal environment, but also be usable as a module.
   
   We're still writing scripts like its the 90s - but the state of JS development has moved on significantly since Sphere was released. These are features that most modern JS developers are going to expect as standard.
  • Last Edit: April 16, 2015, 06:42:23 am by casiotone

Re: CommonJS modules
Reply #4
1. Modules prevent naming conflicts

I consider this the least compelling argument. A well designed package will not have this issue the way we do things now. A badly made package will likely have much more concerning issues to deal with.

2. Packages allow better dependency management

This is pretty neutral to me. We haven't really had too many scripts/libs that have others as requirements, but it likely will become a concern in the future.

   3. Packages allow easy updating

Again, this will likely be more useful with a more complex ecosystem. Even then I think, it's mostly a nice-to-have, but it is nice to have.

   4. A package index makes finding libraries easier

This is a very good point, and I think it's probably the most notable advantage we would see immediately.

   5. We can re-use packages from other JS package managers

This I have my doubts about. Most Node-based packages I have seen depend on things inherent to Node.js, or at the very least are necessarily done different in Sphere. Many packages would need fairly significant changes, and I suspect would likely benefit from being wholly imported.

   We're still writing scripts like its the 90s - but the state of JS development has moved on significantly since Sphere was released. These are features that most modern JS developers are going to expect as standard.

  • Fat Cerberus
  • [*][*][*][*][*]
  • Global Moderator
  • miniSphere Developer
Re: CommonJS modules
Reply #5
I will say the dependency management would be useful.  This is actually what's stopped me from releasing the Specs threader as a standalone thing, it depends on Link and I didn't want script users to have to deal with the extra complexity of having an additional dependency (even if it's just one file), particularly since it's a dependency I don't maintain.

It's certainly something worth considering for inclusion in the Pegasus spec.
miniSphere 5.0.1 - Cell compiler - SSj debugger - thread | on GitHub
For the sake of our continued health I very much hope that Fat Cerberus does not become skilled enough at whatever arcane art it would require to cause computers to spawn enourmous man eating pigs ~Rhuan

Re: CommonJS modules
Reply #6

I will say the dependency management would be useful.  This is actually what's stopped me from releasing the Specs threader as a standalone thing, it depends on Link and I didn't want script users to have to deal with the extra complexity of having an additional dependency (even if it's just one file), particularly since it's a dependency I don't maintain.

It's certainly something worth considering for inclusion in the Pegasus spec.

Well, it's already in the spec. But I'd like to see it now!  :)

Re: CommonJS modules
Reply #7
This weekend I'm going to look into setting up a bower fork and registry for Sphere.

Bower's model should be a much better fit than npm as it dedupes modules and puts everything in one directory instead of npm's method of nesting dependencies, since loading multiple versions of the same library is likely to break in Sphere.

We can use this straight away for existing code without moving to CommonJS - it should be cool.

Re: CommonJS modules
Reply #8
I do like the look of bower better than npm. Being set up for a browser rather than Node.js is probably better for Sphere-like engines.

  • Fat Cerberus
  • [*][*][*][*][*]
  • Global Moderator
  • miniSphere Developer
Re: CommonJS modules
Reply #9
Just going to bump this thread as minisphere 1.2 has been released and includes CommonJS support and require() built in.  module.exports is not supported yet, but the next release of Duktape will include it.

As a word of caution, 1.2's implementation is a bit barebones.  There is no package support as in casiotone's solution, it simply loads a JS file from ~/cjs_modules and executes it as a module (via a C-side Duktape.modSearch() function).  Any advanced package-type stuff will have to wait.
  • Last Edit: June 06, 2015, 03:03:50 am by Lord English
miniSphere 5.0.1 - Cell compiler - SSj debugger - thread | on GitHub
For the sake of our continued health I very much hope that Fat Cerberus does not become skilled enough at whatever arcane art it would require to cause computers to spawn enourmous man eating pigs ~Rhuan

Re: CommonJS modules
Reply #10

Just going to bump this thread as minisphere 1.2 has been released and includes CommonJS support and require() built in.  module.exports is not supported yet, but the next release of Duktape will include it.

As a word of caution, 1.2's implementation is a bit barebones.  There is no package support as in casiotone's solution, it simply loads a JS file from ~/cjs_modules and executes it as a module (via a C-side Duktape.modSearch() function).  Any advanced package-type stuff will have to wait.


Awesome!

You could implement something like described in this comment: https://github.com/svaarala/duktape/issues/60#issuecomment-60973567

i.e. provide a way to register a modResolve function that takes the calling module id and the id passed to require and returns the path the engine should load from, then any advanced resolution can just be done in JS.