As another stepping stone in eventually making a JS-based map engine, I've added the ability to construct an image from an ArrayBuffer, which is needed to allow tilesets and such to be loaded in script.  miniRT will also be aware of this functionality, so it can be used in schemas, like this one here to load RTS tilesets:

Code: (javascript) [Select]

/**
 *  Sphere 1.x tileset format schema CommonJS module
 *  a schema for reading Sphere 1.x tileset files using miniRT/binary
 *  (c) 2015-2016 Fat Cerberus
**/

module.exports =
[
	{ id: 'signature', type: 'fstring', size: 4, regex: "\\.rts" },
	{ id: 'version', type: 'uintLE', size: 2, range: [ 1, 1 ] },
	{ id: 'numTiles', type: 'uintLE', size: 2 },
	{ id: 'tileWidth', type: 'uintLE', size: 2 },
	{ id: 'tileHeight', type: 'uintLE', size: 2 },
	{ id: 'bpp', type: 'uintLE', size: 2, values: [ 32 ] },
	{ id: 'compressed', type: 'bool', values: [ false ] },
	{ id: 'hasObstructions', type: 'bool' },
	{ type: 'reserved', size: 240 },
	{ id: 'images', type: 'array', count: '@numTiles', subtype: { type: 'image', width: '@tileWidth', height: '@tileHeight' } },
	{ id: 'tiles', type: 'array', count: '@numTiles', subtype: { type: 'object', schema: [
		{ type: 'reserved', size: 1 },
		{ id: 'animated', type: 'bool' },
		{ id: 'nextTile', type: 'uintLE', size: 2 },
		{ id: 'delay', type: 'uintLE', size: 2 },
		{ type: 'reserved', size: 1 },
		{ id: 'obstructionType', type: 'uintLE', size: 1 },
		{ id: 'numSegments', type: 'uintLE', size: 2 },
		{ id: 'nameLength', type: 'uintLE', size: 2 },
		{ id: 'terraformed', type: 'bool' },
		{ type: 'reserved', size: 19 },
		{ id: 'name', type: 'fstring', size: '@nameLength' },
		{ id: 'obsmap', type: 'switch', field: 'obstructionType', cases: [
			{ value: 0, type: 'reserved', size: 0 },
			{ value: 2, type: 'array', count: '@numSegments', subtype: { type: 'object', schema: [
				{ id: 'x1', type: 'uintLE', size: 2 },
				{ id: 'y1', type: 'uintLE', size: 2 },
				{ id: 'x2', type: 'uintLE', size: 2 },
				{ id: 'y2', type: 'uintLE', size: 2 },
			]}},
		]},
	]}},
];

The Image() constructor takes width, height, and an ArrayBuffer as arguments and populates the image with the pixels in the buffer.  The buffer must be large enough to represent the entire image, and the pixels are assumed to be in 32-bit RGBA order, which seems idiomatic for Sphere.  Here's an example of using the feature from my WIP binary file reader:

Code: (javascript) [Select]

			case 'image':
				var width = _fixup(fieldType.width, data);
				var height = _fixup(fieldType.height, data);
				if (typeof width != 'number' || typeof height != 'number')
					throw new Error("missing or invalid width/height for `image` field");
				var pixelData = stream.read(width * height * 4);
				value = new Image(width, height, pixelData);
				break;

This also, as a bonus, would allow you to do software surfaces in script (in minisphere, CreateSurface() makes hardware surfaces)--although I'm not sure what the performance of that would be.  In any case it's something FJ mentioned I should have early on in the project's life, and now I finally implemented it.

Loading Surfaces straight from ArrayBuffers was also something I needed/wanted for the Turbo map engine. I wouldn't say it's strictly necessary, but it made loading tilesets, fonts, and spritesets MUCH faster.

I worried a little about the performance as well. Regardless, I think the solution that would work best would be to expose a memcpy API to JS, to allow copying between ArrayBuffers. In fact, it would be useful to have versions of many C functions like memcpy, memset, etc. in JS, and it would also be easy to make script equivalents to allow any scripts that used them to work on systems without native functionality.

It's notable that SDL2's blit is actually faster than memcpy'ing, since it can make use of RLE data to skip unnecessary reads, but this would still give you performance quite close to a native implementation.

That's also a part of why I'm going in much farther on ArrayBuffers in Turbo. I think they are a really useful thing for JS, they just need some really basic features added on top (like a string conversion, and memcpy) to really be used most effectively. Unfortunately, given the philosophy of the browser developers, these things will likely stay out of JS proper for the forseeable future.

I definitely agree, I've found that using only stock ArrayBuffers isn't really adequate for most of the use cases they are ostensibly meant to solve, you need to layer some extra stuff on top of them first.  Part of the issue is that an ArrayBuffer object by itself is pretty much a black box, like a void pointer that you can't cast.  You can construct a typed array or DataView on top of it to manipulate its contents, but copying data around in bulk between buffers like with memcpy in C?  Yeah, not going to happen.  And it's also rare that you see an ArrayBuffer used as input, from what I've seen they are typically created on-demand whenever some data is generated (I'm guilty of this too, FileStream.read() returns a new one for each call...), which honestly defeats the purpose.

I'll take that memcpy API idea into consideration as minisphere 3.1 develops.

By the way the miniRT binary file loader is turning out to be one of the best ideas I ever had.  Loading a tileset:

Code: (javascript) [Select]

	var binary = require('miniRT/binary');
	var schema = require('miniRT/schema/tileset');
	var tileset = binary.load('maps/TestvilleTiles.rts', schema);
	while (true) {
		tileset.images[1].blit(0, 0);
		FlipScreen();
	}

And to load a map instead, it's as easy as substituting the correct schema:

Code: (javascript) [Select]

	var schema = require('miniRT/schema/map');
	var map = binary.load('maps/Testville.rts', schema);

The tricky part is to make the schema format flexible enough to represent entire file formats while still remaining JSON serializable.  So far this hasn't been an issue, but we'll see what happens when I get to the spriteset loader (.rss).  I still remember how hard that was to get right when I implemented it in C, so much so that it managed to earn itself a "HERE BE DRAGONS!" comment.  Which is NOT a compliment--the only other function in the codebase that managed to earn that honor was main() (I'm slowly cleaning it up though).

I'll take that memcpy API idea into consideration as minisphere 3.1 develops.

I should clarify, I really meant memcpy on TypedArrays rather than directly on raw ArrayBuffers, and I expect such an API to operate on the element sizes, not always byte sizes. But given how both SM and V8 work, it would result in a small amount of arithmetic and then a call to memcpy uner the hood.

Code: (javascript) [Select]

var xml = require('miniRT/xml');
// ...fetch some XML from a file or something...
var dom = xml.parse(xmlText);

Which makes decoding XML as easy as JSON!

There's an ulterior motive, of course.  I eventually want to support the Tiled map editor, which uses XML as its default save format.

Tiled? Yesss! That's something I had eventually hoped for, but didn't ask about anymore because XML can be a bitch to parse. This is excellent news and would bring Sphere that much closer to being cross-platform friendly.

I should clarify, I really meant memcpy on TypedArrays rather than directly on raw ArrayBuffers, and I expect such an API to operate on the element sizes, not always byte sizes. But given how both SM and V8 work, it would result in a small amount of arithmetic and then a call to memcpy uner the hood.

That makes sense, but here's something interesting: In Duktape's implementation of ArrayBuffer, they are secretly byte arrays - you can index them using [] without constructing a Uint8Array on top of them.  I tend to avoid taking advantage of it though because it's not standards compliant.

I did it - I got miniRT to load RSS spritesets in one pass (any version).  I thought it was impossible, but I just needed to make a small change to the schema handling and it all came together.

The only tweak I needed to make was, when processing a sub-schema (i.e. an 'object' field), to construct the sub-object using the parent object as its prototype.  This allows the individual frame headers to reference properties from the main RSS header, and everything works.

Code: (javascript) [Select]

/**
 *  Sphere 1.x tileset format schema CommonJS module
 *  a schema for reading Sphere 1.x tileset files using miniRT/binary
 *  (c) 2015-2016 Fat Cerberus
**/

module.exports =
[
	{ id: 'signature', type: 'fstring', size: 4, regex: "\\.rss" },
	{ id: 'version', type: 'uintLE', size: 2, range: [ 1, 3 ] },
	{ id: 'numImages', type: 'uintLE', size: 2 },
	{ id: 'frameWidth', type: 'uintLE', size: 2 },
	{ id: 'frameHeight', type: 'uintLE', size: 2 },
	{ id: 'numPoses', type: 'uintLE', size: 2 },
	{ id: 'base', type: 'object', schema: [
		{ id: 'x1', type: 'uintLE', size: 2 },
		{ id: 'y1', type: 'uintLE', size: 2 },
		{ id: 'x2', type: 'uintLE', size: 2 },
		{ id: 'y2', type: 'uintLE', size: 2 },
	]},
	{ type: 'reserved', size: 106 },
	{ id: 'data', type: 'switch', field: 'version', cases: [
		{ value: 1, type: 'object', schema: [
			{ id: 'images', type: 'array', count: 64, subtype: { type: 'image', width: '@frameWidth', height: '@frameHeight' } },
		]},
		{ value: 2, type: 'object', schema: [
			{ id: 'poses', type: 'array', count: '@numPoses', subtype: { type: 'object', schema: [
				{ id: 'numFrames', type: 'uintLE', size: 2 },
				{ type: 'reserved', size: 62 },
				{ id: 'frames', type: 'array', count: '@numFrames', subtype: { type: 'object', schema: [
					{ id: 'width', type: 'uintLE', size: 2 },
					{ id: 'height', type: 'uintLE', size: 2 },
					{ id: 'delay', type: 'uintLE', size: 2 },
					{ type: 'reserved', size: 26 },
					{ id: 'image', type: 'image', width: '@width', height: '@height' },
				]}},
			]}},
		]},
		{ value: 3, type: 'object', schema: [
			{ id: 'images', type: 'array', count: '@numImages', subtype: { type: 'image', width: '@frameWidth', height: '@frameHeight' } },
			{ id: 'poses', type: 'array', count: '@numPoses', subtype: { type: 'object', schema: [
				{ id: 'numFrames', type: 'uintLE', size: 2 },
				{ type: 'reserved', size: 6 },
				{ id: 'name', type: 'pstringLE', size: 2 },
				{ id: 'frames', type: 'array', count: '@numFrames', subtype: { type: 'object', schema: [
					{ id: 'imageIndex', type: 'uintLE', size: 2 },
					{ id: 'delay', type: 'uintLE', size: 2 },
					{ type: 'reserved', size: 4 },
				]}},
			]}},
		]},
	]},
];

Having written these schemas, it occurs to me that the Sphere 1.x file formats are ridiculously complex.  They definitely suffer from backwards-compatibility baggage, especially the spriteset format (that said, version 3 isn't terrible, but it's not really optimal either).  I'm thinking it might be high time we deprecated them and replaced with something a bit more modern and easier to work with.  For starters, a more modular format would be welcome.  JSON, perhaps?

Well, I don't think using a single schema to represent the RSS format is a good idea, unless you only plan on supporting V3.

I actually rather like some of the formats, in particular the font format is nice and to the point. But I'll agree, I think that RTS, RSS, RMP are a little long in the tooth.

Well, I don't think using a single schema to represent the RSS format is a good idea, unless you only plan on supporting V3.

Nonetheless, I managed to pull it off.   I posted the full schema I used above.  I used a bit of conditional magic, switching on the version field to read different structures depending on which version number is in the header.  There's also some basic validation built in too, as you can see on the version and signature fields.

On an unrelated note, I'm currently experimenting with rendering maps using the Galileo API instead of legacy blitting.  I already make an atlas for tilesets on load, so the entire map could be represented as a single Shape and thus rendered with a single GPU call, with the vertices cached on the GPU (Allegro 5.2 supports vertex buffers).

The one thing I'm having trouble with is figuring out the vertex order.  I know I have to make a triangle strip, but I'm unsure how to set up the vertices and texture UV to keep the map as one continuous strip.  @FJ, you wrote the Turbo map engine to take advantage of Galileo didn't you?  Any tips you could offer?

By the time I wrote the Turbo runtime, all TurboSphere had for graphics was Galileo

I had to issue an extra vertex for each tile to reset the positioning.

https://github.com/FlyingJester/TurboSphere/blob/master/bin/system/scripts/turbo/map.js#L200

(At least I think that that's the line.)

I never found a better way around this, but it's not like this is an unusual problem with OpenGL-style rendering. NVidia even created an extension just to deal with this kind of thing, called the Primitive Reset, which would take the place of this extra vertex. In more complex scenes that would be more desirable, but in this case it really makes no difference compared with burning an extra vertex here.

I think I understand the theory at least, you throw in an extra vertex in order to get the texturing to work properly.  What I DON'T understand is:

* Why are there 4 vertices per tile?  My understanding was that a tri-strip only requires 2+n vertices (where 'n' is the number of triangles).  A tile is two triangles, so after the first tile, you should only need 2 vertices for each one.
* Why does the extra vertex use tex_coords[3] of the next tile?  Shouldn't that be tex_coords[0]?

It will only need 2+n when you are not trying to use non-adjacent points in the texture on adjacent triangles like we need to. If you find a way around this, I'd love to know, but I'm pretty sure this is just how it is.

As far as the texture coordinates of the burned vertex, I think it does not matter since the triangle that the burned vertex appears in is totally flat across the bottom of the tile. The next triangle is formed up the right-hand side of the triangle (again with zero width), and then finally the first triangle of the new tile is draw. I think (it's been a while since I looked at this code) that the only non-zero-width triangle this burned vertex appears in is overdrawn, and it only appears in a single non-zero-width triangle (which spans the bottom left of the new tile, the top left, and then the top right, and is immediately overdrawn by issuing the third vertex of the new tile).

I would also like to note that it was while writing this code that I changed the default drawing mode of Galileo Shapes at or above 4 vertices to be triangle strips instead of fans. This is precisely the situation I said could not be done with a single Shape using triangle fans.

Yeah, I default to strips too for 5+ vertices.  Specifically, my defaults are:

Code: (c) [Select]

draw_mode = shape->num_vertices == 1 ? ALLEGRO_PRIM_POINT_LIST
	: shape->num_vertices == 2 ? ALLEGRO_PRIM_LINE_LIST
	: shape->num_vertices == 4 ? ALLEGRO_PRIM_TRIANGLE_FAN
	: ALLEGRO_PRIM_TRIANGLE_STRIP;

I use a fan for == 4 vertices so that the vertices for an isolated quad (the common case for emulating a blit) can be specified clockwise, which is more intuitive than the zigzag pattern needed for a strip.  It's worth noting though that minisphere's Shape() constructor has an additional optional parameter which allows you to specify the primitive type.  That allows Galileo to also replace legacy APIs such as LineSeries(), PointSeries(), etc.