So it finally struck me the other day, the beauty of the Pegasus design and its obsession with modules.  For a long time I thought it just added pointless complexity, but, well... I don't think that anymore.  It's now the direction I want to pursue for minisphere 4.0 and forwards.

The thing about CommonJS modules is, module import is language-agnostic: As long as the underlying platform (minisphere in this case) knows how to load a module given only its name and returns a set of bindings back to the JS environment, the JS code using those bindings doesn't care whether they came from JS code, loaded from a DLL, etc.  This is in stark contrast to, say, RequireScript which can only deal with JS scripts and expects a concrete SphereFS filename.  Module names by contrast are abstract.

What this means for us is that a plugin subsystem can easily be layered on top of the module system:

• https://github.com/fatcerberus/minisphere/issues/63

For instance a graphics plugin might register a module for itself, say "screen".  When a script says require("screen"), only then is the plugin loaded and a render window created.  And despite the lazy initialization, there's no risk of a race condition because the require() is also what gives you the bindings.

In this way plugin references are implicit, avoiding the need for a plugin manager GUI or anything of that sort; a game only loads plugins it uses, loads them on-demand (pre-loading, if needed, could be implemented via a Cell build step which scanned your scripts for require() calls), and any plugins which are never referenced never need to be included (in contrast to a plugin manager which can create unintended explicit references).

For implementing the plugin system and maybe cleaning up the socket/networking system, is taking a look at Node.JS's source useful in any way?

Hm, that's tough to say.  Node.js uses require() and has a package manager, that's true, but its API is set up quite a bit differently from Sphere's.  It's designed around asynchronous operations and callbacks (usually in the form of promises), whereas game logic tends to work better with a Sphere-like synchronous design.  Of course networking could still benefit from async though (= network lag shouldn't cause game lag), so I'll see what happens.  The good part is that minisphere already includes a promise library out of the box, as part of miniRT.

For Turbo, I switched to allowing each game having to explicitly specify which plugins it wants. I don't particularly see the benefit from unifying scripts and native libraries, though. It's true you could, in theory, write a native plugin and an alternative JS script, but I'm not sure the benefit.

From native, you rather need to write to your embedded engine. This is avoidable (something in the same vein as Swig, I toyed with this concept for years in TurboSphere), but you tend end up with terrible APIs in most languages, like how Unity is for any language but C#.

If you were to add the ability to call into, say, Python scripts from JS, I see some big benefits. But that implies some pretty epic changes--for one, requiring Python For two, though, then you would need to write bindings. When I thought that through for TurboSphere, I figured I could just embed Python into a plugin, since I already have a binding API (the plugin API + V8, later SM's API), and the ability to embed any language I want just by having native plugins.
But at that point, I've defeated the purpose of a unified module system, since I'm back to using the native interface to define a script API.

I suppose at that layer it matters how much you want to put into the plugins, and how much you want to put in the engine. I moved to an extremely lean engine very early, so that easily dictated which way I went.

The main benefit as I see it is to game developers.  The developer doesn't have to care whether the module is implemented in JS, Python, native or what-have-you, it just has to know the name of the module.  With a more traditional plugin system you have two distinct but interrelated systems - one to pull in JS bindings (RequireScript) and another to define plugins, and it may not always be clear which plugin provides which bindings.  C# uses a system like this, where you define assembly references for the project, and it can get confusing - especially when the name of an assembly doesn't exactly correspond to the namespaces it exports (the Framework itself is guilty of this at times).

While it's true that you have to get the bindings out of the plugin into the JS environment somehow, and do this in a way that's as generalized as possible, that's the case even with a more traditional setup.  I wouldn't want all my plugins to have to depend on Duktape, for example.

I guess the big divide to me is two-fold:

1. You need to actually provide dependencies differently for script and for native. This can be mitigated with a plugin manager, though, so it's less important.

2. All scripts are cross-platform, certainly not all plugins are.

Not having plugins depend on your JS engine is a larger problem than you (or at least I, for one) originally thought.
At one point I toyed with adding Python support to TurboSphere. Not mixing JS with Python, but allowing plugins to provide APIs for both languages at once. It became apparent quite quickly that this required some pretty major work to smooth out the differences between cpython and V8, even though the languages have pretty similar capabilities, and I was not even worrying about inheritance/prototype abstraction yet. You would have an easier time not worrying about other language bindings, but I'm not sure how much easier it would be.

I also abandoned the idea of a JS-wrapping API (beyond an optional header of templates that worked well for how the Sphere APIs are written), since my JS engine exposes this already. Whether my plugins depend on libturbo or on SpiderMonkey, it doesn't really matter. There is a dependency either way, and SM already has a decent and reasonably stable API to write against without me abstracting it.

I want to be clear I'm not trying to discourage you, I'm just sharing what I ran into working on similar systems.

To be clear, I'm less concerned with what plugins are actually capable of and more with shoring up the abstraction itself.  That is, whether bindings are provided by a native DLL or a JS script shouldn't matter to the calling code, which can import either through a unified interface.  This is the way the Node.js module loader works, and it's a worthwhile abstraction I think.

If it's impossible (or at least prohibitively difficult) to make a plugin to, say, add "aftermarket" Python support, so be it.  It is what it is.  But if I have a map engine implemented in JS and I decide to make that into a native plugin to improve performance, doing so shouldn't be a breaking change (at least not for that reason alone).

Awesome, you understand why I wanted modules so bad.

I want to propose a couple of things:

One. Use a very node-ish module system. You can view their code and base the system of theirs. Do it, it is stable, it works well, is used by millions. You can then add on top of it: first search for a system (compiled) module, then search for files in system, then search for local files. (also see my comment on GitHub). If you make sure that the compiled module exports according to the spec, this could be a nice speedup for your system.
Node also supports native modules. They are compiled using Node-Gyp. It allows for example for a native protocol to Redis.

Two. Always make sure there is a full set of system modules available, either native or non-native.

Three: You can indeed replace JS modules with native modules at will. Also, I think adding "aftermarket" Python support is very possible. But we need to think about how. Will we want to exchange sphere-python.dll between TS and MS?

EDIT:
8: I just saw a lot of Futue of Pegasus topics. Ill just make mine.


Greetings!