Spherical forums

User-Made => Projects => Topic started by: Rhuan on July 04, 2017, 01:15:05 pm

Title: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on July 04, 2017, 01:15:05 pm
So, picking up from some brief discussion in the miniSphere topic I'm planning on building Javascript based replacements for sphere's map engine and spriteset handling to work with the Sphere v2 api.

This is at an early stage (I've not written any code yet) but I've previously written a close to fully functionally engine for sphere v1 which I intend to use as a model as it supported most of the features we'll want and had a logical structure which should map to Sphere v2 - though there will be several key changes and it needs to be more flexible and work for other people not just me - I'm also hoping that using Galileo properly will enable the whole system to run a lot faster.

My hope is that this can become part of the standard minisphere distribution if I get it right...

Anyway that said I've written out my current design thoughts below, I'd appreciate comments, feedback and suggestions:

Planned features


Choices to make:



Other thought patterns
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on July 05, 2017, 08:45:14 am
Quote
Custom format with a creation function that takes any of the above 3 as inputs -> current preference


For spritesets, that's my preference too.  There's no reason for the engine itself to need to natively use user-editable formats when we have a fully programmable official Sphere v2 compiler in the form of Cell. :)

I'll give all this a better look either tonight or tomorrow, I didn't get much sleep last night so I'm not alert enough to take all this in. :)
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on July 06, 2017, 01:32:10 pm
I still have written any code for this yet, planning to start tomorrow or saturday now.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: DaVince on July 06, 2017, 04:21:10 pm
This is some solid stuff, whatever you do I think Tiled compatibility would be a huge positive addition.

Also, mjs seems to make more sense since that's how ES6 modules are imported anyway, right? Gives you more flexibility too.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on July 06, 2017, 04:32:13 pm
Agreed, .mjs makes most sense for a brand new map engine.  That lets you export stuff by name without polluting the program with global variables and is a lot cleaner syntax-wise than CommonJS-style modules.

Full disclosure here: Cell transpiles down all .mjs scripts to .js right now because Duktape doesn't yet support ES6 modules.  But in the future, yeah, I'd definitely prefer map scripts to be mjs.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on July 06, 2017, 05:27:06 pm

Full disclosure here: Cell transpiles down all .mjs scripts to .js right now because Duktape doesn't yet support ES6 modules.  But in the future, yeah, I'd definitely prefer map scripts to be mjs.
How does this work with the mini RT or sphere run time aka the system scripts you made for minispgere, aren't they .mjs?
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on July 06, 2017, 06:35:39 pm
Sphere Runtime modules are .js.  I actually can't use any ES6 syntax in them or they won't work, which gets frustrating at times.  Since the runtime is part of the engine, Cell can't transpile them.

As for how everything still works with that setup, the nice thing about TypeScript and Babel is that this:
Code: (javascript) [Select]

import { Thread, Console } from 'sphere-runtime';


Gets transpiled to something like:
Code: (javascript) [Select]

var req1 = require('sphere-runtime'),
    Console = req1.Console,
    Thread = req1.Thread;


Which means the modules work with both Node-style require as well as ES6 import.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on July 06, 2017, 07:03:04 pm
Hmm, ok I'd missed that, I had in my mind that  using var xyz = require("something") meant "something" was an mjs file.

Anyway I can decide that point later, I'm going to try and get started on the core logic tomorrow.

Any feedback on my other decision points would be appreciated.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on July 07, 2017, 11:57:39 am
require() is a Node.js thing and actually predates modules being a standard part of JS.  When executing a require()'d script, the engine wraps the script's source code in a function which it then calls, which allows you to put variables and functions at the top level without polluting the global scope.  In ES6 modules there's no need for that function hack, since the JS engine handles the scoping directly.

Anyway, regarding the sprite engine: Do you plan to allow sprite objects to be used independently from the map engine?  I think that could be useful for some types of games that can't use traditional tilemaps but still use sprites, e.g. a top-down shooter.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on July 07, 2017, 01:48:59 pm

require() is a Node.js thing and actually predates modules being a standard part of JS.  When executing a require()'d script, the engine wraps the script's source code in a function which it then calls, which allows you to put variables and functions at the top level without polluting the global scope.  In ES6 modules there's no need for that function hack, since the JS engine handles the scoping directly.
Ah right, makes sense.

Quote
Anyway, regarding the sprite engine: Do you plan to allow sprite objects to be used independently from the map engine?  I think that could be useful for some types of games that can't use traditional tilemaps but still use sprites, e.g. a top-down shooter.
Yes, see point 1.xi above.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on July 07, 2017, 02:03:48 pm
Hmm... collision detection seems like it might be an interesting challenge to implement outside the context of a map engine.  In all the map engines I've written, this was always the difficulty I had when trying to make entities be a self-contained class, and is even seen in the miniSphere codebase: There's a circular dependence between maps and persons because the map engine needs to track active entities and the entities themselves have to know about the map they're on in order for collision to work properly.  It's not really possible to fully isolate them in my experience.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on July 07, 2017, 07:36:41 pm

Hmm... collision detection seems like it might be an interesting challenge to implement outside the context of a map engine.  In all the map engines I've written, this was always the difficulty I had when trying to make entities be a self-contained class, and is even seen in the miniSphere codebase: There's a circular dependence between maps and persons because the map engine needs to track active entities and the entities themselves have to know about the map they're on in order for collision to work properly.  It's not really possible to fully isolate them in my experience.
I have an idea, will report back after I've given it a go.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on July 08, 2017, 10:05:11 am
So update on structural plans, now planning this to be 3 scripts:

mEngine.js: i.e. Map Engine
sEngine.js: i.e. Sprite Engine
cEngine.js: i.e. Collision Engine

The scripts will have "hooks" to enable them to work together though each will function alone, obviously certain features will be disabled when a script is not used e.g. all collision detection will be handled by cEngine so if you don't use it the Sprite Engine's path finding will not check for the appropriateness of a route.

I note I've decided to set the scripts up as CommonJS style modules for now using require, similar to the Sphere RT modules; this is partly because I haven't yet compiled cell for Mac and partly because I'm hoping that these can eventually be added to the miniSphere distribution in someway maybe included in the SphereRT folder and hence not being part of a project directory and therefore out of scope for cell.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on July 10, 2017, 07:13:16 pm
Brief update.

I've begun with the sprite engine/SEngine, been having a busy time with the rest of life so progress is slow but it's under way...

Key thing I'm considering at the moment is the balance of the interface and how many methods is too many methods, particularly with entity objects and the like where there may be a lot of them. (NOTE: "entity" is the name I am using instead of the Sphere v1's "Person")

e.g. should there be:

SEngine.prototype.queueMove(entity_id, direction, frames);

OR should it be:

Entity.prototype.queueMove(direction, frames);

The latter seems unnecessary to me, it was my first thought of how to do it BUT, I'm not sure why it was.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: mezzoEmrys on July 10, 2017, 08:37:43 pm
One chip I'm throwing in for discussion - could cEngine and sEngine allow you to include an arbitrary number of hitboxes relative on a single entity? One of the things I've found difficult to work with is that sprites are limited to a single hitbox square for native implementation. As an example, trying to implement something like a semi-circle sword swing could do with a bit of accuracy by changing one large hitbox into two or three smaller ones, without needing to split it up into that many sprites and make sure they are all coordinated.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on July 10, 2017, 08:51:37 pm

One chip I'm throwing in for discussion - could cEngine and sEngine allow you to include an arbitrary number of hitboxes relative on a single entity? One of the things I've found difficult to work with is that sprites are limited to a single hitbox square for native implementation. As an example, trying to implement something like a semi-circle sword swing could do with a bit of accuracy by changing one large hitbox into two or three smaller ones, without needing to split it up into that many sprites and make sure they are all coordinated.
I was planning on any given sprite having a collision shape that would be any polygon you chose to define.

I assumed that there would be no need to have multiple boxes if each box you have can be any shape you like?

Though with the current piece I've written the polygon would be tied to the sprite, so if you had two entities on screen with the same sprite and changed one of their polygon's it would change the other one too, I'm guessing you don't want this? I'll change it so you can change the poly for an entity.

In fact what may work well for a hack and slash is being able to have specific frames implement polygons that go with them, I could do that.

I note that you'd potentially still need to split the person from their sword otherwise you won't know whether the player bumped into someone or hit them with a sword - cEngine will be able to tell you which entities are hitting which other entities but if the person and their sword were one entity it would not be able to distinguish them.

I suppose I should add a simple way of hooking two or more entities together to enable systems like that to work smoothly.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on July 10, 2017, 08:55:30 pm
If you look at a game like Smash Bros., there are some attacks characters can do that have multiple rectangular hitboxes, and you can't represent those as a single polygon because they may come out at different times and/or do differing amounts of damage.  So I definitely see where mezzo is coming from here.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on July 10, 2017, 09:00:10 pm
Hmm so instead of a singe hit-poly include an array of them and then for collision detection return -1 for not collided or an id for the collided poly if collided?
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: mezzoEmrys on July 11, 2017, 01:42:41 am
Originally I was only considering hitboxes and not hit-polys for speed, since that's how I'm used to things working with other engines. Having hit-polys would fix that, but the later conversation about having individual polygons for individual frames is even better. Having the hitpoly id(s?) returned on collide would save the trouble of having two individual entities, since one entity can, at that point, do all the important things I can think of that a composite entity would, since you can have one script manipulate them both.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on July 11, 2017, 03:21:53 am

Originally I was only considering hitboxes and not hit-polys for speed, since that's how I'm used to things working with other engines. Having hit-polys would fix that, but the later conversation about having individual polygons for individual frames is even better. Having the hitpoly id(s?) returned on collide would save the trouble of having two individual entities, since one entity can, at that point, do all the important things I can think of that a composite entity would, since you can have one script manipulate them both.
Well, let's see if I can make this run at a decent speed....
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on July 11, 2017, 07:40:22 pm
Interface indecision...

So for queuing movement commands what should the interface be?

There will be a function that will pathfind to a specific location and then queue movement to follow the path.

But what about an interface for directly queuing specific moves?

My current method is this:
Code: [Select]
SEngine.entity_Id["Jeff"].queue.push({direction:"west",ticks:10});


Is that sort of approach good or should there be a function of some kind - I just can't see what value the function would add.

Edit: Generalising the above point I'm thinking about how much store I should set by making the engine have nice function interfaces, e.g. I added the following earlier:
Code: [Select]
SEngine.prototype.Idle = function()
{
  return (this.waiting == this.entities.length);
}

Which is a function some people would probably find helpful BUT - if I explain some of the systems internal variables that sort of check can be in-lined very easily without the function overhead, thoughts?
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on July 13, 2017, 05:46:48 pm
Did some performance testing on what I've got so far; my script can process movement for about 1,600 entities before dropping below 60FPS - only had about half on screen at a time rest were updating off screen.

I added some timing code - the update code processing where everything is was using about half the time and the rendering code the other half; hopefully the galileo speed improvements in 4.7 will make this a bit better, but also hopefully most games shouldn't need to process over 1,000 entities at once.

I also note that my update script currently calls an array.sort once per frame which may be a key cause of slow down that I could cut.

(I note that I don't currently have any collision detection in this so it will get slower)
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: DaVince on July 14, 2017, 05:43:45 am
If your test ran at ~60 FPS with 1000 entities on screen, then that's quite a bit better than what we currently have already. Sir Boingers started slowing down on my previous second-gen i7 when I added more than ~50 cupcake sprites in a single map.

Speaking of collision detection and performance; would such an element be toggleable so that entities that don't need that feature won't waste their time on it? (And the same for some other components, perhaps?)
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on July 14, 2017, 01:45:46 pm

If your test ran at ~60 FPS with 1000 entities on screen, then that's quite a bit better than what we currently have already. Sir Boingers started slowing down on my previous second-gen i7 when I added more than ~50 cupcake sprites in a single map.

Speaking of collision detection and performance; would such an element be toggleable so that entities that don't need that feature won't waste their time on it? (And the same for some other components, perhaps?)
i am using a fairly powerful MacBook Pro so it may not be a fair test but seriously slowdown from 50 entities is bad.

Everything should only happen if needed but you still have to loop through the whole list of sprites and st least check if it's needed or not or do you.... I may have just thought of another optimisation. :)
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on July 14, 2017, 07:26:51 pm
So today I have:
1. Written a short piece of code to convert an rss file into what this system needs to work with. It's unfortunately v1 code - as being able to a) use LoadSpriteset and b) use surface_object#save is almost invaluable - note the v1 functions are used to read and convert the rss it's then v2 only to load the converted data and work with it.

2. Written a potentially unnecessary optimisation (double indexed fixed length movement queues instead of using array#push and array#shift) - it's not really fixed length in that it will extend if it runs out of room, but that shouldn't happen often.

Couple more points to consider for optimisation:
1. maintain an array of refs to currently moving entities to avoid needing to loop down the whole list and check for the ones that are moving -> optimisation for situations where there are a large number of stationary entities.

2. Consider replacing array#sort - this is currently called once whenever the update script runs, this is to allow for overlapping entities e.g. if a sprite is two tiles tall; the sort orders the render array so that that entities are drawn in order of increasing y value (i.e. lower down the screen = drawn later) (may need to allow for alternate sorting criteria later) -> to optimise could I replace the sort with a check only when an entity's y value changes that moves that one item forward or backward in the render array. This would be faster for situations where not many items move above/below each other BUT it could be slower for situations where large numbers of sprites move above then below. I note also that a negative feature of array#sort is that it's not stable - this means if you have 2 sprites with the same value they keep swapping order in the draw queue - if they're overlapping this leads to a flicker.


Next order of business:
I need a break from SEngine, time to start MEngine. Now, should a map be an array of shapes, each one representing one tile so I can clip the ones that don't need to be on screen OR should it be a single shape which is just the whole map - with this method I'd load the map by drawing it all out onto a surface then convert that to a texture and texture a single shape with it - I could use my shader to have the shape only be the size of the render window and move the texture around as needed. The only problem with this would be if there is a limit to the size of a surface or a texture (or is there is a size above which they become unworkable).
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: DaVince on July 15, 2017, 01:43:07 pm
Nice work on this! :D I don't think anyone minds your use of v1 code for the conversion process, since it's compatible after all.

Quote
Everything should only happen if needed but you still have to loop through the whole list of sprites and st least check if it's needed or not or do you.... I may have just thought of another optimisation. :)

Well, to be fair, every single entity was sending out COMMAND_ANINATE commands every single frame, but yeah, the frame rate got diminished quite quicky. It would be kinda nice if auto-animation could be part of the map engine itself, I suppose.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on July 15, 2017, 03:52:04 pm
Well, to be fair, every single entity was sending out COMMAND_ANINATE commands every single frame, but yeah, the frame rate got diminished quite quicky. It would be kinda nice if auto-animation could be part of the map engine itself, I suppose.
At the moment with my system the best you can do is queue an action but say to do it 10,000 times or something rather than re-queuing over and over which would get slow fast, I'm intending to add a feature to auto-repopulate empty queues though I'm not sure how best to do it yet - the most obvious method would be to add a function to an entity to be called when a queue is empty though that could be slow if I don't do it carefully - if I have it call just once when the queue runs out that would probably be ok; whereas if it's called every update when the queue is empty that could be a lot of extra function calls.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on July 16, 2017, 02:50:36 pm
Put in a feature to queue an action forever - set an action to happen 0 times or (any negative number) and it will repeat until/unless you manually remove it. (previously actions were cancelled when they had <1 times to go, I've changed it to 0 times to go specifically)

Also set up the basics of the map engine, it needs a lot of work still:
- no current support for animated tiles (I probably want to make it generate entities for the sprite engine to animate but be able to do it based on data in the map file)
- render function currently draws all layers - should change to drawing a range (so you can control which ones you want pre-sprites and which ones after)
- need to add support for moving layers

Other points:

- Need a collision engine...
- need to build a standard input mechanism
- should build an rmp loader
- should come up with file formats to use (the needed data for a sprite is a png spritesheet and then a "template" object, the needed data for a map is currently a Tiled JSON file and a tileset png)
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on July 25, 2017, 01:48:10 pm
Just a note, I've had a lot on and am going to be away for a bit, my plan is to have version 1 of this ready around the 1st September, sorry that it's taking a while, real life has got in the way...

I note that if this works as well as I'm planning it should be able to be a drop in replacement for the Sphere v1 map engine (though you would have to replace all related function calls, I could make a helper script to edit any function calls within persons in the map files to make that process easier).

The hope is that transitioning to this new system will be relatively painless and add flexibility and better control to a project as well as better v2 integration.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on August 11, 2017, 03:38:24 pm
So... Considering that miniSphere's Sphere 1.x compatibility has now reached to the point that I'm emulating bugs, I definitely think it's time for a new map engine. ;) :P
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: DaVince on August 13, 2017, 02:42:30 am
Since I'm very very busy with real life stuff anyway, I'm also going to wait on the New And Improved Map Engine before developing anything. It sounds very promising so far!
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 16, 2017, 10:45:30 am
Well I'm back and making progress.

After making a total mess of it for no clear reason I've now got the coordinate system for the map engine and the sprite engine working together properly, incorporating the ability to apply a transformation to the entire drawn map (including all sprites) and separately a facility to draw the map zoomed in or out.

I've been doing a few speed tests, I'd like the code to be a bit faster if possible, currently drawing 1200 sprites the update script takes approximately 7.5 milliseconds and the renderscript takes 6.9 milliseconds. (This includes processing movements and applying zoom and any other transformation).
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 19, 2017, 06:53:59 am
Just a brief apology that I haven't got a product to show yet I keep going down the rabbit hole of refactoring and micro-optimising what I've already written rather than making progress towards having a usable release; I'll hopefully be there soon though, most of the infrastructure is in place for the three scripts just need to get collision logic set up properly and then clean up the interface used for initiating everything.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 19, 2017, 12:26:46 pm
Well some progress made:
1. I can move around the map with the camera following me - (though I can't manage to make it always stay centred if I zoom in and out too much)

2. Collisions with map scenery work

3. Collisions with other sprites on the map did work earlier they're not working now and I'm not sure why, I will fix this...

4. An equivalent of FF6 style Mode 7 works with no slow down (thanks to Fat Cerberus)
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on August 19, 2017, 12:36:34 pm
4. An equivalent of FF6 style Mode 7 works with no slow down (thanks to Fat Cerberus)

I guess that means you figured out the projection matrices. :)
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 19, 2017, 12:50:34 pm
4. An equivalent of FF6 style Mode 7 works with no slow down (thanks to Fat Cerberus)

I guess that means you figured out the projection matrices. :)
In the end I copied and pasted your screen transformation code and edited a few numbers; I was fiddling around with z coordinates which actually appear to be irrelevant unless I'm missing something, the working version I'm using has all the z coordinates set to 0.

In other news, tile based collisions all work now. Polygon based collisions is the next target, then an RMP loader.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on August 19, 2017, 01:20:33 pm
If you're doing a 3D projection then the Z coordinate affects how close or far away the vertex is from the screen, extending from zero (the camera "lens") to negative-infinity.  Anything outside of the Z clipping distances (defined by the near and far parameters of project3D) won't be rendered.  That's why there's a Z translation just before the projection in my example.

Note: Anything with positive Z will be clipped by definition because it lies "behind" the camera.

In any case, good that you got it working at least.  3D is hard. :P
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 19, 2017, 02:02:53 pm
If you're doing a 3D projection then the Z coordinate affects how close or far away the vertex is from the screen, extending from zero (the camera "lens") to negative-infinity.  Anything outside of the Z clipping distances (defined by the near and far parameters of project3D) won't be rendered.  That's why there's a Z translation just before the projection in my example.

Note: Anything with positive Z will be clipped by definition because it lies "behind" the camera.

In any case, good that you got it working at least.  3D is hard. :P
I still can't draw shapes with z-values but I can transform the screen which works sufficiently well.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 20, 2017, 08:07:30 pm
Not finished yet (obviously) but I think these are really starting to take shape.

I'd be interested what people think of the code style and features etc, files are on dropbox so people can take a quick look: https://www.dropbox.com/sh/9rug8p7bjll4e8l/AAAbiYncMBRbYd8k7Wps5CpHa?dl=0

3 system scripts + 2 shader files needed for them to function + main.js shows using them - this isn't a full demo yet as they're not finished (so I haven't included the resources (map and sprites) that the main.js script is looking for.

Quick note on licensing etc. When finished I will release these under the 3 clause BSD license in line with miniSphere - for now I've posted them for comment but ask they are not used in any public projects as they're incomplete.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on August 22, 2017, 12:40:13 am
Just skimming real quick, I came across this bit of ugly Sphere v1 code:

Code: (javascript) [Select]
var file = OpenRawFile("output.txt",true);
var avgs = [0,0,0,0];
for(var i = 0; i < m_render_times.length; ++i)
{
avgs[0] += (m_render_times[i] / m_render_times.length);
avgs[1] += (s_render_times[i] / s_render_times.length);
avgs[2] += (m_update_times[i] / m_update_times.length);
avgs[3] += (s_update_times[i] / s_update_times.length);
}
file.write(CreateByteArrayFromString("average m_r = " + avgs[0] + "\n average s_r = " + avgs[1] + "\n average m_u = " + avgs[2] + "\n average s_u = " + avgs[3]));
file.close();

You could change that to (Sphere v2):
Code: (javascript) [Select]
// [snip: calculate averages]
FS.writeFile("~/output.txt", "average m_r = " + avgs[0] + "\n average s_r = " + avgs[1] + "\n average m_u = " + avgs[2] + "\n average s_u = " + avgs[3]);

I haven't actually tried it out yet, just skimming over the code for now.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 22, 2017, 02:00:33 am
Just skimming real quick, I came across this bit of ugly Sphere v1 code:
Trust you to spot that... :P I like being able to write to the directory I'm working in so I can have less open... That's just some temporary code for testing speeds of the different key functions though it won't remain in anything final.

Also main.js isn't really part of the project it's just a playground for trying it out, there isn't any v1 code in MEngine, SEngine or CEngine.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on August 22, 2017, 02:06:12 am
I figured it was temporary debugging code, I just used it as an opportunity to point out that Sphere v2 has a function for writing a string into a file in one shot so you don't have to mess with byte arrays.  Maybe you already knew about that function though :)
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 22, 2017, 02:14:30 am
Please could someone suggest some really massive/complicated .rmp files I can test with?

Whilst I'm aiming at supporting Tiled maps as well as .rmp ones I want to support .rmp fully so would be good to have some horrible test cases.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on August 22, 2017, 02:21:33 am
FBnil's Puff-Puff has pretty big .rmp files in the ~80k range.  They have embedded tilesets and lots of entities, so they'd be a good test I think.  I'd be interested to hear if my RMP-loading code from oh-so-long-ago works to load all (or even most) RMP maps in the wild.  It's a pretty gnarly format, topped only by RSS that has 3 versions all of which are totally different from each other! :o
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: DaVince on August 22, 2017, 06:47:52 am
Albrook from Kefka's Revenge might be a good test map too. Lots of layers and scripts everywhere.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 22, 2017, 07:24:06 am
Albrook from Kefka's Revenge might be a good test map too. Lots of layers and scripts everywhere.
Map and entity scripts are scaring me slightly, roughly I'm planning on using the function constructor to parse them and I can set up execution conditions easily enough my concern is I'd like to make v1 maps work which means I need to implement a shim over the top of all the v1 map/entity related functions AND provide that shim to all the map and entity scripts, I can do it but I haven't worked out an efficient way to do it yet (if you've read my code you'll know what I think of inefficiencies...)

I also need a mechanism for passing other functions and variables to those scripts as by default in v2 they'll be run with no access to anything global.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on August 22, 2017, 09:25:57 am
If you're striving for full Sphere v1 compatibility, you actually want to run map/entity scripts with `EvaluateScript()`, making them into functions (they get their own scope) or using `eval()` (uses scope of caller in ES5+) will change the behavior.

The difference if you're curious is that Sphere 1.x runs all scripts as top-level program code, the same way browsers handle multiple scripts.

This, incidentally, is the reason I made SetUpdateScript() etc. be able to accept a function instead of a string, since it let's you access local variables in your update/render/person scripts that way.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 22, 2017, 01:01:57 pm
If you're striving for full Sphere v1 compatibility, you actually want to run map/entity scripts with `EvaluateScript()`, making them into functions (they get their own scope) or using `eval()` (uses scope of caller in ES5+) will change the behavior.

The difference if you're curious is that Sphere 1.x runs all scripts as top-level program code, the same way browsers handle multiple scripts.

This, incidentally, is the reason I made SetUpdateScript() etc. be able to accept a function instead of a string, since it let's you access local variables in your update/render/person scripts that way.
i'm trying to avoid using any of the v1 api, I want all of this stuff to work if you build sphere without vanilla.c

I'm not looking to emulate v1 the idea of the compatability shim I'm thinking of is to make it do enough so that anyone with a v1 project who wants to switch to v2 can use their existing maps without going through editing them all; their main script files will obviously need some updates.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 22, 2017, 04:04:30 pm
Why can rts tiles have individual obstructed pixels?
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on August 22, 2017, 04:33:48 pm
No idea, I never implemented it in miniSphere and haven't had any compatibility issues, I think even the Sphere 1.5 editor only supports line segment-based obstruction.  The RTS spec says:
Code: [Select]
byte blocked;       // 0 = no obstruction data, 1 = old obstruction data, 2 = new obstruction data

So I just assumed it was deprecated.

edit: Yep, Sphere 1.x ignores it too:
https://github.com/sphere-group/sphere/blob/master/sphere/source/common/Tileset.cpp#L513-L517
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 22, 2017, 04:36:46 pm
No idea, I never implemented it in miniSphere and haven't had any compatibility issues, I think even the Sphere 1.5 editor only supports line segment-based obstruction.  The RTS spec says:
Code: [Select]
byte blocked;       // 0 = no obstruction data, 1 = old obstruction data, 2 = new obstruction data

So I just assumed it was deprecated.

edit: Yep, Sphere 1.x ignores it too:
https://github.com/sphere-group/sphere/blob/master/sphere/source/common/Tileset.cpp#L513-L517
The very first example map I tried has pixel based obstruction data...

And opening it in the 1.5 editor allows me to edit it and keep it.

If you make a new tileset you don't it get it but if you use an old one it's kept. And I want to get this to work now...
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on August 22, 2017, 04:49:40 pm
Sphere 1.x also ignores it, though, as you can see from my GitHub link.  Kind of weird that the editor still supports it but the engine doesn't...

Although that tileset code is in the Common folder so that means the same code is shared with the editor... maybe 1.5 still supported pixel-based collision and they removed it later.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 22, 2017, 05:50:03 pm
Oh I'm wrong - they're not pixels, it's just it lets you have segments of 1 pixel only.
And I found something not in tileset.rts.txt, a tile in an rts file can optionally have a name, the tileset I was trying to read had a named tile in it which was throwing off my reader. I found this by looking at tileset.cpp.

tileset.rts.txt states that the tile information block ends with 22 reserved bytes, conversely tileset.cpp has a 2 byte name_length then a byte for "terraformed" (seemingly not used) then 19 bytes reserved. THEN after that it looks for a name of length: name_length. (the length can be 0 in which case it looks for nothing).

tileset.rts.txt says nothing of this name.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on August 22, 2017, 05:55:17 pm
Hmm... not sure where I picked up the information if it's not in tileset.rts.txt, since miniSphere has code to load the tile names:
https://github.com/fatcerberus/minisphere/blob/v4.8.4/src/minisphere/tileset.c#L154
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 22, 2017, 05:58:51 pm
Hmm... not sure where I picked up the information if it's not in tileset.rts.txt, since miniSphere has code to load the tile names:
https://github.com/fatcerberus/minisphere/blob/v4.8.4/src/minisphere/tileset.c#L154
Test map and tileset I was using are from spectacles so I'm guessing you either saw it in tileset.cpp or you had the same problem and fixed it when you were first working on miniSphere.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 22, 2017, 08:29:24 pm
I'm not sure if the effects suit the map, but for testing purposes here is South Figaro from Kefka's revenue being rendered by MEngine with a Scale2x filter and a small mode7 effect.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on August 22, 2017, 08:34:56 pm
Looks really nice, although your aspect ratio seems off (the tiles are too tall).  I think this might be the cause:

Code: (javascript) [Select]
var wh = screen.width / 2;
var hh = screen.height / 2;
screen.transform = new Transform()
    .translate(-wh, -hh)
    .scale(3 / wh, 3 / hh)  // <-- HERE
    .rotate(0.6, 1.0, 0.0, 0.0)
    .translate(0, 0, -1.0)
    .project3D(120, wh / hh, 0.1, 2.0);

The factors for both axes should be the same, you want to divide by either wh or hh but not both (I tend to prefer the former).  That was my mistake when I originally wrote the code.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 22, 2017, 08:45:24 pm
Albrook from Kefka's Revenge might be a good test map too. Lots of layers and scripts everywhere.
So I tried loading Albrook and just got a black screen, I also tried deleting the nightsky layer in case it was to do with handling alpha wrongly but that had no effect. :( something to fix I guess but hey I'm not done yet.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on August 22, 2017, 08:46:53 pm
Albrook is a hell of a map, I even remember having trouble getting miniSphere to load that one properly. :o

Kefka's Revenge in general is a monster of a game.  The fact that miniSphere runs it nearly flawlessly at this point is a testament to my dedication to backwards compatibility. :P
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 22, 2017, 08:48:05 pm
The factors for both axes should be the same, you want to divide by either wh or hh but not both (I tend to prefer the former).  That was my mistake when I originally wrote the code.
In the original you divided both by wh, I assumed it was a mistake and changed it...
Albrook is a hell of a map, I even remember having trouble getting miniSphere to load that one properly. :o
All I'm processing at the moment is the tile data, I'm ignoring scripts and entities, so I'd hoped it would just work.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 23, 2017, 12:54:15 pm
Albrook is a hell of a map, I even remember having trouble getting miniSphere to load that one properly. :o
So my current theory is that my turning the tileset into a single strip, 32,480 pixels wide and 16 pixels tall is the problem.

@Fat Cerberus do you know of a hard limit on sizes of surfaces that can be handled by miniSphere?

(I construct the map by loading all of the tile images out of the data file and then drawing them on to a surface which I convert to a texture then use as an atlas with which I texutre one shape the size of a tile which then draws onto a large surface the size of the whole map.)

But I get nothing but black, I checked the images coming in and they seem ok, I tried converting the surface to a texture and blitting it with prim and I get nothing, if I do the same with South Figaro (and it's sensibly sized tileset) the blit draws the tileset onto the screen fine.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 23, 2017, 01:10:28 pm
UPDATE... see attached.

I dropped the combining the tileset step and the load time went way up (retexturing the tile shape once per tile to draw per layer (i.e. a bazillion times...) takes a while) but now it actually works.

So I think I'm right on the surface size limitation :( I guess I could tile them in a rectangle rather than a straight line - a straight line just made the sums easier.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on August 23, 2017, 01:17:09 pm
miniSphere doesn't impose a hard limit on texture size, but not all GPUs like huge textures.  What you want to do is construct a square atlas by taking the ceiling of the square root of the number of images (that's a mouthful! :P) and use that value for both the X and Y sizes:
https://github.com/fatcerberus/minisphere/blob/v4.8.4/src/minisphere/atlas.c#L59

The correct U/V for a tile is then easily calculated by simple modular division:
https://github.com/fatcerberus/minisphere/blob/v4.8.4/src/minisphere/atlas.c#L95-L109
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 23, 2017, 01:47:57 pm
miniSphere doesn't impose a hard limit on texture size, but not all GPUs like huge textures.  What you want to do is construct a square atlas by taking the ceiling of the square root of the number of images (that's a mouthful! :P) and use that value for both the X and Y sizes:
https://github.com/fatcerberus/minisphere/blob/v4.8.4/src/minisphere/atlas.c#L59

The correct U/V for a tile is then easily calculated by simple modular division:
https://github.com/fatcerberus/minisphere/blob/v4.8.4/src/minisphere/atlas.c#L95-L109
Thanks for the suggestion, I was already planning something like that - doing it now, I only did the simple maths version because I coded the original implementation of this in the middle of the night.

I don't think I would have thought of using modulo though, I would have done divisions and extra floor-ing; modulo is definitely more elegant.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on August 23, 2017, 01:52:57 pm
Thanks for the suggestion, I was already planning something like that - doing it now, I only did the simple maths version because I coded the original implementation of this in the middle of the night.

I sympathize 100%!  This is pretty much the entire story of miniSphere 1.0 development. :P
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 23, 2017, 02:55:18 pm
OK after messing with several typos that now works but for some reason the night sky layer isn't drawing, I get all the other ones though...

Going to take a pause from compatibility and re-do how I draw maps, using the map as an atlas to texture a shape the size of the screen seemed like a good idea but I think it may be causing problems.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 23, 2017, 03:51:43 pm
Going to take a pause from compatibility and re-do how I draw maps, using the map as an atlas to texture a shape the size of the screen seemed like a good idea but I think it may be causing problems.
Ok, this is done and a graphical anomaly I was seeing before has gone which is nice - I was seeing ripples on the screen from time to time under the previous drawing method - assumedly some side effect of constantly moving such a massive texture around on a shape, drawing the whole map and just moving it so only part is on screen seems to have removed this issue.

EDIT: :( spoke too soon, the shimmer/ripple is still there, I wonder if it's an issue with how allegro interacts with my screen or something.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 23, 2017, 04:41:14 pm
Note sure if it's an issue with my computer or a miniSphere issue, could someone else test this and see if they get a shimmer on the screen as they walk: https://www.dropbox.com/s/lm3wsewvkaeuxss/sengine.zip?dl=0

License notes:
- sprite used for demo purposes made with https://github.com/jrconway3/Universal-LPC-spritesheet and are licensed CC-BY_SA, I'm not currently intending this to be distributed so not giving full credits
- map used in the test is pinched from Kefka's Revenge
- code is mostly mine with a few bits that are Fat Cerberus' + the Scale2x shader is based heavily on this: https://gist.github.com/singron/3161079 (no license is given), as with before this isn't really ready for use, pease test it and delete it for now

(Note obstructions are turned off in this test copy)
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on August 24, 2017, 10:22:44 am
Are you sure the ripples are not just screen tearing?  miniSphere doesn't attempt to vsync.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 24, 2017, 12:46:12 pm
Are you sure the ripples are not just screen tearing?  miniSphere doesn't attempt to vsync.
it could well be screen tearing not an issue I'd thought of, I will check later if dropping the FPS helps.

Is there a facility to turn on vSync or fake such from JS in minisphere?

If it doesn't already exist could we have a screen.naturalFR or something that gets the refresh rate of your screen so you can do screen.framerate = Math.min(screen.naturalFR,60) or something like that in order to turn on vSync?
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on August 24, 2017, 02:29:07 pm
I'm not sure if Allegro has any ability to use vsync, I would have to check.  It's not enough just to set screen.frameRate to match the monitor refresh, you also have to hold off flipping the backbuffer until you get a vsync signal, otherwise it eventually drifts out of sync.

Does your graphics driver have any ability to force vsync?  You could enable that and see if the ripples go away.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 24, 2017, 04:25:22 pm
Adding:
al_set_new_display_option(ALLEGRO_VSYNC, 1, ALLEGRO_REQUIRE);
As row 92 of screen.c does seem to have improved the ripples issue, there are still some shimmers.

I think they may be related to drawing a map scaled at what is likely a non-integer scale and moving it around fairly fast.(My test demo has zoom in and out keys which are probably just a bad idea :( )

There is no graphics setting on macOs to turn on vsync, in some areas macOs is great, in having simple access to some settings it really isn't :(.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 24, 2017, 04:31:32 pm
Hmm no, further testing with no zoom, movement speed decreased and vsync on (via the code above) still shows some shimmers. :(
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on August 25, 2017, 01:37:45 am
I see it now too, it's especially noticable on the sewer grates and staircases.  I think a lot of it is just an optical illusion due to the band-like patterns, I've even noticed similar artifacts in the actual KR when moving around.  I wouldn't worry about it too much.

I did get a glitch where on one launch it didn't draw anything, so it's not just you.  Haven't reproduced the segfault yet, but I'm pretty sure I know what causes it at least.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 25, 2017, 02:21:14 am
Hmm yeah, I've just tried playing KR with the screen res increased and such blurriness became common.

Though I also think it was worse before I put vSync on.

Anyway I'm going to put that to one side now and aim at having a usable release of this code within 1 week.

It won't have everything but it should have enough to be considered a full map and sprite engine, current plan for first release:

- map and sprite loading and displaying including full support for v1 formats and partial support for new formats (+facility and notes on how to add formats)

- collisions using tiles or polys, note, poly mode will only support rectangles and circles for now but will allow each object to have any number of either/both of those

- map and sprite scripts including scripts to activate in meeting certain conditions or colliding with something etc.

- movement and camera control
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: DaVince on August 25, 2017, 09:19:17 am
I haven't managed to test anything yet, but I'm quite certain that the shimmers are indeed an optical illusion. The original game suffered from this too, might be something with the pixel art.

Anyway, really looking forward to first release. You're making wonderful progress on this, nice work!  :D
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 26, 2017, 07:46:15 am
Trying to test polygon based collision detection using albrook, something seems horribly wrong with the collisions not sure what yet, I think I'm going to add a debug feature to draw all the collision polys on the map so I can see what's going on.

Other thought: I think I should revert to my previous method of map drawing, (where a shape is created the size of the screen and the image used as a texture atlas) this made it easy to do sliding transitions or suddenly have the map shrink to half of the screen or the like - whereas the current method which simply draws the whole map translated by appropriate x and y so the right part is on the screen means if you want the map to slide off or do anything else weird like that you'll need to bring in a surface and a surface.to Texture step which is super slow.

Side note, SSj.log may be the best function ever created.

Other side note: collisions in albrook don't work correctly running Kefka's Revenge in miniSphere, though they do appear to under Sphere 1.5; -> I think there's something wrong with miniSphere's handling of v1 map collisions.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: DaVince on August 26, 2017, 08:09:34 am
Yeah, collision is weird in Albrook on miniSphere but I never thought it was something with how collision is handled by the engine itself - KR breaks so easily I thought something very specific about the way it's coded made it not work well. Even in 1.5 KR likes to have other random issues. That game is one hell of a clusterfuck.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 26, 2017, 08:20:13 am
Yeah, collision is weird in Albrook on miniSphere but I never thought it was something with how collision is handled by the engine itself - KR breaks so easily I thought something very specific about the way it's coded made it not work well. Even in 1.5 KR likes to have other random issues. That game is one hell of a clusterfuck.
How could collision detection go wrong when the obstructions are set in the tiles?

Are you frequently changing layers or something weird like that?
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: DaVince on August 26, 2017, 08:22:47 am
I wasn't the one who coded the madness, but I do remember layer switching being present. I believe it's how it handles lower and upper floors, so you can be either on top of or under a bridge, for example. I think it's supposed to switch layers right between going from the dock into town too, but I'm not sure.

There has been collision funkiness or issues in random versions of Sphere between 1.10986 (the original Sphere version it came with) and 1.5 as well, but it hasn't been particularly predictable as far as I remember.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 26, 2017, 08:30:39 am
versions of Sphere between 1.10986... and 1.5 as well
1.10986... Good memories, I think I remember 1.10988 being the version I switched to after discovering spheredev and realising that 1.1 which I'd downloaded from sphere.sf.net wasn't the right version to be using.

I wasn't the one who coded the madness, but I do remember layer switching being present. I believe it's how it handles lower and upper floors, so you can be either on top of or under a bridge, for example. I think it's supposed to switch layers right between going from the dock into town too, but I'm not sure.
Hmm OK I'll take a look - may be that there's some odd feature used to decide which layer it's on that isn't always working.

Why is its code base such a nightmare?
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: DaVince on August 26, 2017, 08:38:44 am
tree (the coder) didn't know how to code when he started, learned as he went along, and he never seems to have refactored any of it, so it became what it became. It's almost kind of endearing though, with all the silly variable names and eval() hell everywhere.

There was one point where I wanted to try doing the scripts from scratch but there's actually so much to the game that's a huge undertaking all by itself. Refactoring anything would probably have broken more than it fixed. Unfortunately.

If I remember correctly, layer switching caused issues with the couch moving scene as well. Obstruction issues were plaguing that scene depending on what Sphere version you used and whether or not you were lucky enough to have all variables align just right.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 26, 2017, 09:01:45 am
And the problem wasn't albrook, the problem was that I'd drawn my sprite scaled at 50% but not scaled it's collision polygon.

Hmm, so I'm including debug features behind a this.DEBUG_MODE flag which is a property set at the top level of MEngine and SEngine, but there is aqutie a bit of code behind it - here's where I wish I could use cell and have a preprocessor macro equivalent set up as a tool to determine whether to include the debug code and possibly some other features - in general the code base is somewhat bloated by providing multiple ways of doing things when any given implementation will only use on of the paths - as another example there'a s lot of code to support a tile based system that's not used if SEngine.tile_movement is set to false.

And it all just feels like bloat - though when the mode is swapped suddenly it's needed and a lot of the code you were using before turns into bloat.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on August 26, 2017, 09:19:13 am
And the problem wasn't albrook, the problem was that I'd drawn my sprite scaled at 50% but not scaled it's collision polygon.

Fun fact: SetPersonScale() doesn't scale the sprite base in Sphere 1.5 but does in miniSphere.  This was a case where I felt fixing the bug was more important than perfect compatibility.

As for Albrook, a lot of the collision weirdness happens for me in the same places in Sphere 1.5.  For example at one point a pirate blocks you on the stairway... except you can walk right through him.  For a long time I thought this was a bug in miniSphere's obstruction code, but then found out the same issue is present in legacy Sphere too.

But hey, because KR is so terrible and prone to break at the slightest touch, it makes a really fun regression test!  And I don't mean "fun" sarcastically either, I genuinely enjoy trying to make it work.  The best compatibility hack I ever wrote was because of KR:
https://github.com/fatcerberus/minisphere/blob/v4.8.4/src/minisphere/vanilla.c#L1275-L1310

I'm still proud of that bit of clever hackery to this day. :D
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 26, 2017, 11:27:42 am
So currently working on sorting collision polygons into a series of segments so collision detection has less to check for each movement - trying to walk around albrook and checking every collision poly for every step is just a tad horrible (it manages between 20 and 45 FPS), all blocked tiles have several different collision polys so all in all it likely adds up to 1000s of polys to check every step.

Couple of questions/thoughts:

1. One thing I've half added but not finished is support for multiple instances of SEngine to interact with each other and collide with each other in one CEngine instance - is there any point in this? it would mean you could have sprites in distinct groups that could choose to draw one group only then suddenly change to drawing both groups AND these different groups would be able to collide with each other etc. BUT it will be a total pain to finish implementing this everywhere as it needs extra levels of checking and referencing in several places. Thoughts on value of such a feature?

2. Load times - for large maps (e.g. albrook) there are noticeable load times, (if you run the test you'll see this) - I'm going to try and optimise this by only require shape#draw call to render each layer whereas the current method requires one shape#draw per tile per layer - which I assume is why it's so horrendously slow. (Fat Cerberus planning to pinch the method I saw you discussing in the miniSphere thread years ago where you use 5 vertices per tile - I note that this will only be to initially create each layer, the humongous many vertices'd shape will be drawn to a surface and then the surface will be saved as a texture)

3. I keep wanting to stop and re-factor everything... BUT I must resist until I have a usable release done.... Urge to refactor rising... Ideally at some point I think I'd like to re-write it all in ES6 typescript with the same logic and same interface just with strict typing and a few optimisations, but yeah that must wait.

4. Are there any other features that are particularly wanted?
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on August 26, 2017, 05:40:24 pm
Hopefully you'll have a lot more performance headroom to do collision detection and such once I get CC working.  That said, you don't need to check every single polygon every frame, only the ones you know are in range.  See miniSphere's tile collision code:
https://github.com/fatcerberus/minisphere/blob/v4.8.4/src/minisphere/map_engine.c#L1284-L1304
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 26, 2017, 05:53:07 pm
Hopefully you'll have a lot more performance headroom to do collision detection and such once I get CC working.  That said, you don't need to check every single polygon every frame, only the ones you know are in range.  See miniSphere's tile collision code:
https://github.com/fatcerberus/minisphere/blob/v4.8.4/src/minisphere/map_engine.c#L1284-L1304
Interesting, if I'm reading that correctly you work out what tile you're on and then specifically check the the polys of the 9 tiles around you?

I haven't got the structure in place to do that, I could add it, but I want to allow for other types of obstructions not just the ones on the tiles without too much extra code.

So my optimisation plan (which I'm half way through implementing) is to divide the map into a series of segments then allocate all collision polygons to one or more segments (more if they overlap).

It will have a slightly longer loading time but should be super flexible and have speedy execution time anyway if I do it right.

EDIT: change of plan I am going to put in a tile based look up for collision polys that originated on tiles, as a tile based lookup is probably a feature that people will want exposed in the API anyway, and it will be even faster than the segment map I was going to implement (that said I will still do the segment map for other polys, anything not from a tile)
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on August 26, 2017, 06:29:05 pm
It figures out how many tiles to check based on the relative size of the tiles and the collision base.  If the collision base is smaller than a tile, then it only checks 4 tiles (2x2) max.  If the person is entirely inside a tile then it only needs to check one.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 26, 2017, 08:38:40 pm
OK, I've tried and discarded the idea of rendering the map based on making a massive shape with 5 vertices per tile - it's difficult to get it to draw correctly AND testing with drawing albrook showed it didn't give a speed benefit.

Time (in milliseconds) to render each layer of the map:

1. With each layer being one massive shape:
log: time to render 218
log: time to render 255
log: time to render 225
log: time to render 219
log: time to render 211
(note this includes time to create the shape as well as draw it, if I was to continue drawing with one of these methods this may be better - but the time taken to create these shapes is horrible)

2. With one shape being used but it's the size of one tile only - shader used to make it draw the correct tile for each location in turn, many many more draw commands but only 4 vertices (i.e. what I was already doing)
log: time to render 83
log: time to render 119
log: time to render 77
log: time to render 77
log: time to render 81

Almost 3 times the speed - I've discovered that the super slow part of my code was the debug stuff not rendering the map.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 26, 2017, 09:56:29 pm
And I've implemented a system similar to that used by miniSphere for collisions with obstructed tiles, all slow down has vanished :).

Currently only testing with one sprite though, need to implement collision segments for sprites then test with multiple people.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on August 27, 2017, 04:02:08 pm
@Rhuan: Sorry I haven't provided very much input here, you pointed out ChakraCore to me and the MIT license + pure C API + cross-platform-ness sealed the deal. :)
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 27, 2017, 06:07:59 pm
@Rhuan: Sorry I haven't provided very much input here, you pointed out ChakraCore to me and the MIT license + pure C API + cross-platform-ness sealed the deal. :)
No worries. Getting a decent modern JS engine into sphere seems like a pretty important thing to focus on.

I'm getting there with this stuff, just trying to make an engine that's crazily flexible whilst also as micro-optimised as I can manage isn't quick.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 27, 2017, 07:54:08 pm
Well I think I've just implemented a great algorithm for collisions between sprites using polygon based detection.

Except I've not included a handler for anyone who goes off the edge of the map - so insta crash :(

Fixed the crashes but there's still an error somewhere :(

EDIT: why does:
var end_x2 = (((x + max_w + d_x)/ t_size)|0) + 1;
give a different result to
var end_x2 = ((x + max_w + d_x)/ t_size)|0 + 1;
That's just odd.

I've become obsessed with bitwise operators instead of the Math library, aside from them scoring higher on most performance bench marks I just think they look cool.

But they do seem to have a few odd properties - adding that extra bracket seems to have got my sprite polygon collisions working.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on August 29, 2017, 11:52:55 am
It's because addition outranks bitwise-OR in operator precedence:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Operator_Precedence

In general, logical and bitwise operators are weird precedence-wise.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 29, 2017, 12:43:42 pm
It's because addition outranks bitwise-OR in operator precedence:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Operator_Precedence

In general, logical and bitwise operators are weird precedence-wise.
But the effect of a bitwise or with 0 on a number should be to it number down to the nearest integer - whether the 1 is added before or after shouldn't impact the result.

If the result of the first sum was 0.8 whever you do 0.8 rounded down = 0 then +1 = 1 OR 0.8 + 1 = 1.8 then round down = 1 it should be the same.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on August 29, 2017, 12:45:10 pm
Addition binds before bitwise or--so you were actually OR'ing with 1, not 0.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on August 29, 2017, 01:02:25 pm
Addition binds before bitwise or--so you were actually OR'ing with 1, not 0.
Oh, no wonder I was getting odd results.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on August 29, 2017, 01:08:04 pm
When dealing with operator precedence, I've found that it's more intuitive to think in terms of binding power, i.e. how "magnetic" each operator itself is, rather than trying to work out which subexpression will be evaluated first.  In your code, addition is more "magnetic" than bitwise OR, so the 0+1 gets combined before the OR is evaluated.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on August 29, 2017, 01:09:53 pm
Addition binds before bitwise or--so you were actually OR'ing with 1, not 0.
Oh, no wonder I was getting odd results.

...wait, was this a stealth pun?  Anything | 1 is always an odd number! :P
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on September 02, 2017, 06:33:14 am
I'm afraid I've had a busy week and I'm not ready to release a usable version of the engine :( Hopefully one more week.

Also got a question I can't work out how to answer at the moment - how do I process entity data read out of a Map file? Particularly when it's an RMP file and it's pointing at RSS files.

I need to add the entities to an instance of SEngine -but that's a different class from a different module - I could pass MEngine a reference to SEngine (I already pass SEngine a reference to MEngine...) it does start feeling a little awkward at that point - though I suppose I never iterate through the top level objects so the recursion shouldn't matter.

But there's another problem in my structure, at the moment my .RSS loader is a utility function sitting in my main script I didn't want it in SEngine as I thought it was horrible - but to avoid passing even more references I'll probably have to put it into SEngine :(.

Possible long term solution - RMP and RSS loaders get converted into a cellscript and removed from the core modules completely, for now though I think I'll have to add an RSS handling method to SEngine...

Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on September 02, 2017, 07:43:11 am
@DaVince I'd like an extensive test for my systems - just running a map on it's own doesn't really cut it, and I have no significant projects of my own to test it with as most of my projects are a few 1000 lines of code with no graphics that then gets abandoned.

So I'm thinking to test it I should re-factor Kefka's Revenge to use the v2 api and my systems for the map and sprites - this would be a great test BUT I obviously don't own KR, I can't see any license with KR it's just distributed freely, you were involved with its creation, what do you think of me using it for such a purpose? Particularly if I share the finished result (if there is such) as a proof of concept any objections?
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on September 02, 2017, 09:24:58 am
The RSS filename in a map file is relative to @/spritesets:
https://github.com/fatcerberus/minisphere/blob/v4.8.4/src/minisphere/map_engine.c#L1908

This kind of thing is why I designed SphereFS the way I did.  Paths are always relative to @/ in low-level APIs, so it's never ambiguous which file is referred to.  If some JS wants to use a base folder itself, it's made explicit by calling FS.fullPath() with a base directory name.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on September 02, 2017, 09:56:36 am
By the way, the pain points you're hitting here re: separation of concerns, this is exactly the problem I have any time I try to design an object-oriented map engine.  It's like designing a GUI framework, there's encapsulation in theory, but under the hood everything gets inextricably connected to everything else and there's usually no way around that.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on September 02, 2017, 10:21:52 am
By the way, the pain points you're hitting here re: separation of concerns, this is exactly the problem I have any time I try to design an object-oriented map engine.  It's like designing a GUI framework, there's encapsulation in theory, but under the hood everything gets inextricably connected to everything else and there's usually no way around that.
It'll be fine ... :P

But as it's going to force me to put an .rss loader into my SEngine.js script I'm currently working on a v2 .rss loader (I've snagged your  old rss schema and am currently tweaking to integrate with the datareader script I'm using)
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on September 02, 2017, 07:51:24 pm
By the way, the pain points you're hitting here re: separation of concerns, this is exactly the problem I have any time I try to design an object-oriented map engine.  It's like designing a GUI framework, there's encapsulation in theory, but under the hood everything gets inextricably connected to everything else and there's usually no way around that.
It'll be fine ... :P

But as it's going to force me to put an .rss loader into my SEngine.js script I'm currently working on a v2 .rss loader (I've snagged your  old rss schema and am currently tweaking to integrate with the datareader script I'm using)
OK RMP loader now tips the rss loader and then when a map is displayed entities appear all over it correctly.

Next step entity scripts.

(Note RMP is far from an ideal format I'm using it as a way of ensuring I support all the features we want, I will aim to implement extensive Tiled support once I have rmp handling finished)
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: DaVince on September 04, 2017, 04:48:31 am
@DaVince I'd like an extensive test for my systems - just running a map on it's own doesn't really cut it, and I have no significant projects of my own to test it with as most of my projects are a few 1000 lines of code with no graphics that then gets abandoned.

So I'm thinking to test it I should re-factor Kefka's Revenge to use the v2 api and my systems for the map and sprites - this would be a great test BUT I obviously don't own KR, I can't see any license with KR it's just distributed freely, you were involved with its creation, what do you think of me using it for such a purpose? Particularly if I share the finished result (if there is such) as a proof of concept any objections?
That sounds like a huge undertaking and refactoring/redoing a lot of messy code, are you sure about that? :smile:

I mostly did the music and contributed ideas to KR though, so it's really up to wiz (MusashiX09) and tree. I'll ask them about it.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on September 06, 2017, 06:24:17 pm
Brief update, I've had a busy couple of weeks at work which means progress has slowed almost to a stop. :(

That said not much is still needed to make this usable.
- the only major feature the engine still needs is person/map scripts which I will I'm aiming to add today/tomorrow (may drag until Friday/Saturday if things go wrong).
- I also need to fix the sprite drawing order for multiple layers - currently sprite layer is not respected when determining what order to draw them.

Then it will be a case of testing and building out usage examples/documentation for it to be in a usable and sharable state.

Other features I still want to add that may be for a version 2:

1. Custom formats that load faster + cell script to produce said format (note these formats will likely have a larger file size but we're not on dial up anymore)
2. Animated tiles, you can fake it with a sprite, but I'd like sphere v1 style tiles that animate to be able to be set in a map, my current rmp loader will ignore this information - it won't be hard to add a handler for it though.
3. Zones - again these are currently ignored - won't be too hard to implement though.
4. HUD/interface system - as far as I know sphere v2 has no api for .rws or any equivalent BUT further than that there should be system a system that lets you design an interface/HUD that then draws every frame if wanted, this system should set up the shapes and textures it needs (including pre-drawing any fixed text to a surface and turning that to a texture) then should just draw them every frame.
5. Extensive Tiled Support, my current Tiled loader is rather limited, it works for maps with tile based collisions only and doesn't load entities or scripts or custom properties
6. Repeating maps - currently the map scrolling locks nicely at the edge of the map, I need to write the algebra for an alternate mode where the map and the sprites and the collisions repeat
7. Re-factored ES6 codebase, I wrote all of this in ES5 as I wanted it to be includable at run time and miniSphere was on Duktape at the time, it would probably make sense to re-factor it with ES6 now miniSphere uses Chakra, but I'm going to put this at the bottom of the list for now.

Other Note
The current plan is to initially test/build out features by porting Kefka's Revenge - no guarantees on how successful this will be but :P
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: DaVince on September 07, 2017, 09:51:47 am
I think you've put forth an amazing effort with this in a short amount of time, great work! :D
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on September 16, 2017, 08:23:46 pm
Sorry for the slow down in progress/updates, work was busy for a while and then more recently I got somewhat addicted to speed profiling the new ChakraCore version of miniSphere, hopefully will get back on track with this now.

One key update today - courtesy of ChakraCore - I can now load rmp files and rss files significantly faster by loading the image data as Uint32Arrays and adjusting it into the right layout (tiles into layers, frames into spritesheets) by manipulating those Uint32Arrays without needing to use any graphics functions - this is a significant change to my previous method where everything was drawn onto surfaces with render functions to load it. (I note that the plan is to put these steps into a cellscript anyway which will reduce the relevance of the time they take - but the old method couldn't go in cell as Shapes/Shaders/Textures and Surfaces aren't available in cell - whereas this method can go into cell)

Uint32Arrays are used because I believe they provide the most efficient way to move 4 bytes arund at a time,

The two key functions for this new way of doing it are:
Code: [Select]
//convert a Uint32Array comprised of tile data where the tiles are written sequentially into an image
function tileBufferToTexture(buffer, t_width, t_height, width, height)
{
  var length = t_width * t_height * width * height;

  var output = new Uint32Array(length);
  for(var i = 0, x = 0, y = 0, t_x = 0, t_y = 0, c = 0, j = 0; i < length; ++i)
  {
    x   = i % t_width
    y   = Math.floor(i / t_width)  % t_height;
    t_x = Math.floor(i / (t_height * t_width)) % width;
    t_y = Math.floor(i / (t_width  * t_height  * width));
    j = (x + y * t_width * width + t_x * t_width + t_y * (t_width * t_height * width));
    output[j] = buffer[i]
  }

  return new Texture(width * t_width, height * t_height, output);
}


//write tile to position target in m_buffer
function setTileInBuffer(t_buffer, m_buffer, t_width, t_height, tile, target)
{
  var t_size = t_width * t_height;
  var m_start = t_size * target;
  var t_start = t_size * tile;
  //note uses a loop for moving the data rather than set and slice intentionally
  //due to large number of times this is called set and slice incur a larger overhead
  //due to related memory management issues
  for(var i = 0; i < t_size; ++ i)
  {
    m_buffer[m_start + i] = t_buffer[t_start + i];
  }
}
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on September 17, 2017, 10:15:37 am
After having given it some thought decided to update this whole project to ES6 style - it will make it a lot easier to understand and follow I think - shouldn't take too long either, lets see if I can have it ES6'd for tomorrow and will try and share a beta version at that point (may be more of an alpha, but we'll see).
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on September 18, 2017, 05:30:12 pm
Unfortunately my effort over the last two days to make a cell script to convert rmp files into the format I want has proved to be a total failure - the initial version of my planned format is totally useless, back to the drawing board with that, in the mean time I'll work on sprites.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on September 23, 2017, 06:22:13 pm
I'm sorry for the further delays - I've had a lot of important refactoring to do and also been testing miniSphere v5.

The attached shows a little test of 1400 sprites walking around with collisions disabled.

Multi-layer rendering is now working - the 700 Celes are on a higher layer than the 700 Arvis.

It also shows some windowstyles drawn with my new (draft) HUDSystem script - another piece to add to this package.

It allows you to assemble a HUD comprised of multiple shapes then move that HUD around as a unit and draw it with a single command - it also includes helper functions to simplify adding various shapes and object types to the HUD include windows using .rws files (and images and text etc). I'd like to add a few more helper functions then will be sharing it.

But for now the next goal has to be fixing the refactored map engine - I broke it when I converted it to ES6.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on September 28, 2017, 06:42:31 pm
OK updates:

1. Everything is written in ES6 style now pretty much
2. It works again
3. I've added handling for person scripts
4. I've added an extensible input handler

2 key things I still need before I can release a working demo now:
1. A method for loading person scripts from a file
2. A decent talk function to show how it works... :P

One further thing before I can publish this as ready for use:
- documentation
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on September 30, 2017, 04:25:00 pm
Alpha version of MapEngine and associated scripts available for download - I need to write some documentation and clean up a few things but I think this now counts as a usable system.

Note the graphical resources used to demo this are not mine and they are from Kefka's Revenge, please don't use them in any other public project without permission from their original authors.

The demo in this download allows you to walk around and talk to two people that is the extent of it - it's meant to demo a map engine and sprite system which I think it does.

Requires miniSphere 5 beta 2 at least also needs the latest version of the pact.js system module - this was updated today 30/09/17 so you maybe need to visit the miniSphere github for that.

edit by Fat Cerberus: Or just use beta 3 ;)

https://www.dropbox.com/s/y5r93nq23nftvfn/dist.zip?dl=0

EDIT: link updated - I fixed a bug where collision checks between sprites would sometimes fail.

EDIT2: the first fixed version wasn't actually fixed, should be fixed now

EDIT3: re-uploaded again.... Sorry about this, found and fixed more bugs, I guess I should have tested more before posting the first time.
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on November 03, 2017, 07:03:05 pm
The Map Engine should now be usable - more polishing to do and more features coming also more documentation is coming...

But the code is all on github so you can take a look, if anything's not obvious feel free to ask.
https://github.com/rhuanjl/SphereLibs
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on November 05, 2017, 12:38:05 pm
This has moved from a project to a usable library now I think, new topic:
http://forums.spheredev.org/index.php/topic,1529.msg11018.html#new
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: DaVince on November 07, 2017, 10:36:35 am
I've been using the map engine already and it's solid, fast and has a nice feature set! Good job on this, Rhuan. :)
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Fat Cerberus on November 07, 2017, 10:38:00 am
6/10 not enough pigs
Title: Re: Map Engine and Sprite Engine for Sphere v2 - Plan
Post by: Rhuan on November 08, 2017, 06:33:45 pm
6/10 not enough pigs
Should I make some kind of performance test involving a herd of pigs?