I just want to say that being able to run this:
https://github.com/fatcerberus/kh2bar

...as-is, with no transpilers or shims whatsoever, is an awesome feeling.

Progress report!

Here's what works so far:

• All Sphere v1 and v2 APIs
• Cell
• ES6 syntax, no transpilation required
• ES6 built-ins, no polyfill required
• ES6 modules!
• ES2017 async functions (!!)
• require() can load ES modules
• STILL backward compatible with Sphere 1.x games

Here's what doesn't:

• File and line information on the error screen.  SpheRun does show a stack trace after a crash, however.
• SSj and SSj Blue
• Linux support; it crashes under VirtualBox at least

Getting require() to be able to load modules was important, as without that ability, transpiled code doesn't work, and backward compatibility is lost.  Both Babel and TypeScript transpile import statements to require() calls.

Good work, is it worth me checking out the latest version of the chakra branch and trying to get it going on Mac?

Also I love how you almost seem to have taken over the issues page on CC GitHub repository.

Yeah, it should be fairly stable at this point.

I can't understand the benefit of an ES17 async functions - possibly as I've never written web javascript.

I similarly don't understand the benefit of promises, unless they're to do with working with working in a browser event loop?

Unrelated to previous post: how does new Texture(width, height, pixels); handle alpha values?
This is used in my rmp loading code to load the tileset, and there are some tiles that are meant to be partially transparent that just don't come out right.

Another unrelated point: the linux failure, could it be related to ICU? the library that CC uses for handling unicode - bizarrely when I first built CC statically it still expected to link to a dynamic copy of ICU (which is not installed by default on macos and I imagine isn't in various linux builds) building CC with command --embed-icu makes the build script download icu and link to it statically. (I couldn't see an obvious way to link statically other than using that without editing a make file myself which with CC scared me slightly)

The Texture constructor, when passing it a buffer, should just lock the texture and copy over the pixel data (RGBA) wholesale, overwriting whatever is there.  What kind of issue are you having?  Admittedly that facility doesn't get tested very often, so there could be a bug.

Note that the pixel data should be tightly packed RGBA, i.e. stride = width.  At least I think it's RGBA...

And it might be ICU, I'll have to look into it.  The .so binary I put in dep/ is the official x64 Linux binary from the CC GitHub page.  It would be nice if we could static-link CC even for Linux, since there are no official packages yet that I could find.

OK so I got the CC branch to build on macOs but a few problems and it doesn't run
Issues to make it build:
1. Your functions asprintf and vasprintf conflict with the names of some functions in the macos version of stdio.h which gets included somewhere.
2. I had to specifically link icu as well as chakra
3. I also had to link libc++ (though this is installed as standard on macos so ok to use the system dylib)

Run time crash (on loading sphere by double clicking the app, i.e. it should in theory be trying to launch the start up game):

Code: [Select]

Thread 12 Crashed:
0   libsystem_c.dylib             	0x00007fffb5df5b52 strlen + 18
1   spheredev.miniSphere          	0x0000000106c6aaf7 jsal_push_new_error_va + 16 (jsal.c:1000)
2   spheredev.miniSphere          	0x0000000106c5b048 jsal_error_blame + 160 (api.c:308)
3   spheredev.miniSphere          	0x0000000106c8a39d js_RequireScript + 77 (vanilla.c:5054)
4   spheredev.miniSphere          	0x0000000106c6bbcc do_native_call + 447 (jsal.c:1820)
5   spheredev.miniSphere          	0x00000001073d01c7 Js::JavascriptExternalFunction::StdCallExternalFunctionThunk(Js::RecyclableObject*, Js::CallInfo, ...) + 199
6   spheredev.miniSphere          	0x00000001075257ae amd64_CallFunction + 78
7   spheredev.miniSphere          	0x00000001072eb130 void Js::InterpreterStackFrame::OP_CallCommon<Js::OpLayoutT_CallIFlags<Js::LayoutSizePolicy<(Js::LayoutSize)0> > >(Js::OpLayoutT_CallIFlags<Js::LayoutSizePolicy<(Js::LayoutSize)0> > const*, Js::RecyclableObject*, unsigned int, Js::AuxArray<unsigned int> const*) + 304
8   spheredev.miniSphere          	0x000000010728e58d Js::InterpreterStackFrame::ProcessUnprofiled() + 13565
9   spheredev.miniSphere          	0x000000010727f28f Js::InterpreterStackFrame::Process() + 463
10  spheredev.miniSphere          	0x000000010727e912 Js::InterpreterStackFrame::InterpreterHelper(Js::ScriptFunction*, Js::ArgumentReader, void*, void*, Js::InterpreterStackFrame::AsmJsReturnStruct*) + 1138
11  spheredev.miniSphere          	0x000000010727e48d Js::InterpreterStackFrame::InterpreterThunk(Js::JavascriptCallStackLayout*) + 93
12  ???                           	0x000000010f740132 0 + 4554228018
13  spheredev.miniSphere          	0x00000001075257ae amd64_CallFunction + 78
14  spheredev.miniSphere          	0x00000001073d2c1b Js::JavascriptFunction::CallRootFunction(Js::Arguments, Js::ScriptContext*, bool) + 27
15  spheredev.miniSphere          	0x0000000106eb674f JsCallFunction + 735
16  spheredev.miniSphere          	0x0000000106c6a4ca jsal_call_method + 166 (jsal.c:199)
17  spheredev.miniSphere          	0x0000000106c6a39a jsal_call + 53 (jsal.c:178)
18  spheredev.miniSphere          	0x0000000106c6cb9e jsal_try_call + 74 (jsal.c:1525)
19  spheredev.miniSphere          	0x0000000106c7c764 script_eval + 215 (script.c:99)
20  spheredev.miniSphere          	0x0000000106c6d82a _al_mangled_main + 2505 (main.c:274)
21  spheredev.miniSphere          	0x0000000106de4326 +[AllegroAppDelegate app_main:] + 23
22  com.apple.Foundation          	0x00007fffa20628ad __NSThread__start__ + 1243
23  libsystem_pthread.dylib       	0x00007fffb5fd893b _pthread_body + 180
24  libsystem_pthread.dylib       	0x00007fffb5fd8887 _pthread_start + 286
25  libsystem_pthread.dylib       	0x00007fffb5fd808d thread_start + 13

I may investigate further tomorrow, just posting it for now in case there's anything that jumps out at you.


Edit, point on Pixel data from previous post
Issue I'm getting is certain tiles that should be partially transparent being 100% transparent. (Image data I'm loading is pulled from a sphere v1 tileset inside an rmp)

Edit... one more point
With Chakra included this Sphere is no longer mini, the app I've just built is 42mb. (I note it's probably possible to optimise it down a bit for size but I don't know how much and I wonder if the CC enabled version may end up needing a different name...)

The problem is that there's seemingly no access to the module's exports from the native side--so there's nothing to latch onto.

There's no way to do it natively via some kind of shim? This seems like it would be a definite deal breaker, unfortunately.

With Chakra included this Sphere is no longer mini, the app I've just built is 42mb. (I note it's probably possible to optimise it down a bit for size but I don't know how much and I wonder if the CC enabled version may end up needing a different name...)

How about bigMiniSphere?

There's no way to do it natively via some kind of shim? This seems like it would be a definite deal breaker

That's exactly what I did end up doing, did you see my second reply?

edit: If you missed it, what I did was to have miniSphere evaluate a fake top-level module:

Code: (javascript) [Select]

import * as Module from "<module filename>"
global.___exports = Module;

...then return the value of ___exports from require().  The same technique is used to load the main class from the startup module.

Ah, sorry, I didn't notice that.

Also, I built minisphere in Debian, and I'm getting a segmentation fault when I exit, and I get this message in gdb (after rebuilding with debugging symbols)

Code: [Select]

4205	malloc.c: No such file or directory.
(gdb) backtrace
#0  malloc_consolidate (av=av@entry=0x7ffff64a7b00 <main_arena>) at malloc.c:4205
#1  0x00007ffff6186dca in _int_malloc (av=av@entry=0x7ffff64a7b00 <main_arena>, bytes=bytes@entry=4096) at malloc.c:3488
#2  0x00007ffff6188f34 in __GI___libc_malloc (bytes=4096) at malloc.c:2928
#3  0x00007ffff6c9d5d1 in ?? () from /usr/lib/x86_64-linux-gnu/liballegro.so.5.2
#4  0x00007ffff6c9e212 in al_create_bitmap () from /usr/lib/x86_64-linux-gnu/liballegro.so.5.2
#5  0x00007ffff6c9eb35 in al_clone_bitmap () from /usr/lib/x86_64-linux-gnu/liballegro.so.5.2
#6  0x00007ffff6ca2cc8 in al_convert_bitmap () from /usr/lib/x86_64-linux-gnu/liballegro.so.5.2
#7  0x00007ffff6ca321b in _al_convert_to_memory_bitmap () from /usr/lib/x86_64-linux-gnu/liballegro.so.5.2
#8  0x00007ffff6d08c80 in ?? () from /usr/lib/x86_64-linux-gnu/liballegro.so.5.2
#9  0x00005555555d969e in screen_free (it=0x555555913ea0) at src/minisphere/screen.c:146
#10 0x0000555555568327 in shutdown_engine () at src/minisphere/main.c:484
#11 0x000055555555e4c8 in main (argc=<optimized out>, argv=<optimized out>) at src/minisphere/main.c:167

If I try to change the resolution of the startup game, I get this as well

Code: [Select]

*** Error in `/usr/bin/minisphere': corrupted size vs. prev_size: 0x00005555564aec5f ***

This is a 4.8.4 build or...?

Also, I think I can use a similar module shim to make things work the other way around too, i.e. importing CommonJS modules.  Instead of giving the source code of the CJS module directly to the module loader (which would cause a syntax error), I instead replace it with this shim:

Code: (JavaScript) [Select]

export default const Module = require("<filename>");

I can't understand the benefit of an ES17 async functions - possibly as I've never written web javascript.

I similarly don't understand the benefit of promises, unless they're to do with working with working in a browser event loop?

They're not too useful in Sphere--yet.  They can be very useful if you have APIs that do stuff in the background, though, for example downloading files.  Suppose you have an operation that needs to download a file but can't do anything until the file is complete, however you want your game to continue running as normal.  You could implement that as an async function:

Code: (javascript) [Select]

async function downloadHugeFile(url)
{
    SSj.log(`starting to download ${url}...`);
    await downloadFile(url);  // yields to event loop
    SSj.log("download complete!");
}

The hypothetical downloadFile() function would return a promise that it resolves when the download is complete, after which point execution continues right after the await.

Notes:

• async () => 812 is shorthand for: () => Promise.resolve(812);
• await x() is shorthand for: x().then(retval => { /*everything after the await*/ })

Sphere has an event loop as well; in Sphere v1 you invoke it manually by calling FlipScreen().  Sphere v2 has a real, Node-like event loop that runs as long as there's something in the Dispatch queue.

I can't understand the benefit of an ES17 async functions - possibly as I've never written web javascript.

I similarly don't understand the benefit of promises, unless they're to do with working with working in a browser event loop?

They're not too useful in Sphere--yet.  They can be very useful if you have APIs that do stuff in the background, though, for example downloading files.  Suppose you have an operation that needs to download a file but can't do anything until the file is complete, however you want your game to continue running as normal.  You could implement that as an async function:

Code: (javascript) [Select]

async function downloadHugeFile(url)
{
    SSj.log(`starting to download ${url}...`);
    await downloadFile(url);  // yields to event loop
    SSj.log("download complete!");
}

The hypothetical downloadFile() function would return a promise that it resolves when the download is complete, after which point execution continues right after the await.

Notes:

• async () => 812 is shorthand for: () => Promise.resolve(812);
• await x() is shorthand for: x().then(retval => { /*everything after the await*/ })

Sphere has an event loop as well; in Sphere v1 you invoke it manually by calling FlipScreen().  Sphere v2 has a real, Node-like event loop that runs as long as there's something in the Dispatch queue.

If I understand this correctly, the downloadFile function would be run in a seperate thread and the second SSj.log call would take place in the event loop when the second thread reported that the file was downloaded?

In other words this allows you to make non-blocking api calls and have specific functions pause until the api call is complete then resume BUT other JS execution continues during the pause?

I can't really think of anything in the Sphere v1 or v2 apis that would benefit from being called in that way, if I could call another JS function with await in front of it rather than an api funciton that could be amazing but based on my reading of it it doesn't work that way.

I've tried running CC miniSphere with verbosity on to see all of the logs and find the crash:

I got the following:

Code: [Select]

initializing Allegro 5.2.2.0
initializing Dyad 0.2.0
initializing JavaScript
initializing dispatch manager
initializing Galileo subsystem
initializing audio subsystem
initializing input subsystem
opening kevfile #0 `#/miniSphere.conf`
    `#/miniSphere.conf` doesn't exist
reading key `keymap_Player1_MENU` from kevfile #0
SNIP - removed numerous lines about loading keys
reading key `keymap_Player4_Y` from kevfile #0
disposing kevfile #0 no longer in use
initializing sockets subsystem
    Dyad.c 0.2.0
initializing spriteset manager
initializing map engine subsystem
creating new mixer #1 at 44 kHz
    format: 2ch 44100 Hz, 16-bit
initializing JS script manager
initializing legacy support module
opening kevfile #1 `#/system.ini`
reading key `Font` from kevfile #1
loading font #0 as `#/system.rfn`
creating image #0 at 128x192
creating image #1 as 8x12 subimage of image #0
SNIP - removed numerous lines about creating subimages of image #0
creating image #256 as 8x12 subimage of image #0
reading key `WindowStyle` from kevfile #1
reading 16x16 image #257 from open file
reading 16x16 image #258 from open file
reading 16x16 image #259 from open file
reading 16x16 image #260 from open file
reading 16x16 image #261 from open file
reading 16x16 image #262 from open file
reading 16x16 image #263 from open file
reading 16x16 image #264 from open file
reading 16x16 image #265 from open file
reading key `Arrow` from kevfile #1
loading image #266 as `#/pointer.png`
reading key `UpArrow` from kevfile #1
loading image #267 as `#/up_arrow.png`
reading key `DownArrow` from kevfile #1
loading image #268 as `#/down_arrow.png`
disposing kevfile #1 no longer in use
setting up jump points for longjmp
searching for a game to launch
opening `/Xdevelopment/startup/game.sgm` from game #1
opening package #1 `/Xdevelopment/startup/game.sgm`
failed to open package #1
opening kevfile #2 `@/game.sgm`
parsing SGM manifest for game #1
reading key `name` from kevfile #2
reading key `author` from kevfile #2
reading key `description` from kevfile #2
reading key `screen_width` from kevfile #2
reading key `screen_height` from kevfile #2
reading key `script` from kevfile #2
disposing kevfile #2 no longer in use
         title: Startup
        author: tunginobi
    resolution: 320x240
       save ID: (null)
loading image #269 as `@/icon.png`
    failed to load image #269
loading image #270 as `#/icon.png`
initializing render context at 320x240
creating image #271 at 320x240
cloning image #272 from source image #270
2017-08-31 18:17:17.628 minisphere[22798:1702920] void * _Nullable NSMapGet(NSMapTable * _Nonnull, const void * _Nullable): map table argument is NULL
activating legacy shaders
opening kevfile #3 `@/keymap.conf`
    `@/keymap.conf` doesn't exist
reading key `keymap_Player1_MENU` from kevfile #3
SNIP - removed numerous lines about reading keys
reading key `keymap_Player4_Y` from kevfile #3
disposing kevfile #3 no longer in use
initializing Sphere v1 API (v2.0)
creating new mixer #2 at 44 kHz
    format: 2ch 44100 Hz, 16-bit
initializing Sphere v2 L0 API
creating new mixer #3 at 44 kHz
    format: 2ch 44100 Hz, 16-bit
loading system default font
GAME CRASH: uncaught JavaScript exception.
)

Note the game it's trying to load is just the startup game that comes with sphere, I have not edited it.

Interestingly having turned on verbosity it's stopped segfaulting, which seems utterly bizarre.

If I understand this correctly, the downloadFile function would be run in a seperate thread and the second SSj.log call would take place in the event loop when the second thread reported that the file was downloaded?

In other words this allows you to make non-blocking api calls and have specific functions pause until the api call is complete then resume BUT other JS execution continues during the pause?

I can't really think of anything in the Sphere v1 or v2 apis that would benefit from being called in that way, if I could call another JS function with await in front of it rather than an api funciton that could be amazing but based on my reading of it it doesn't work that way.

...I think I'm going to bring the Pact class back.

You can await on anything that returns a promise.  In the case of a native API call, yeah, the download would probably run in a separate thread.  But it doesn't have to.  For example, you could do it in JavaScript using a socket.  Here's a sloppy pseudocode implementation:

Code: (javascript) [Select]

import { Pact } from 'sphere-runtime';

var pact  = new Pact();
var downloads = [];

function downloadFile()
{
    let socket = new Socket(url, 80);
    let promise = pact.promise();
    downloads.push({ socket, promise });
    return promise;
}

Dispatch.onUpdate(() => {
    for (let dl of downloads) {
        allData += dl.socket.read(dl.socket.bytesPending);
        if (downloadIsFinished) {  // whatever criteria for when a download is done
            pact.resolve(dl.promise, allData);
        }
    }
});

Then you could just do:

Code: (javascript) [Select]

async function loadInBackground()
{
    SSj.log("loading...");
    let downloadedData = await downloadFile("whatever");
    SSj.log("finished loading!");
}

A promise is basically just "I'll call you back later when I'm in a better position to give you an answer".  You don't need real threads to do that, although they do help.