Spherical forums

Sphere Development => Engine Development => Topic started by: Fat Cerberus on June 13, 2015, 03:31:12 am

Title: Sphere 2.0 API: SphereFS
Post by: Fat Cerberus on June 13, 2015, 03:31:12 am
This is a rough draft of the SphereFS standard I'm working on.  It is designed to be backwards compatible with the Sphere 1.x sandbox but with much more flexibility while abstracting away whether the game is running locally or from a package, whatever.

Sphere 2.0: SphereFS
optional extensions:


SphereFS is a standard for sandboxing the file system exposed to a Sphere game, part of the Sphere 2.0 initiative ("Pegasus").

• The standard path separator is the forward slash (/).

• Unprefixed paths are relative and cannot subvert the sandbox.  Attempts to move upwards (via "../") past the directory containing game.sgm will fail and throw a sandbox violation error.

• Several built-in aliases are available for paths which may vary depending on the platform, engine installation location, etc.  Available aliases:

@/... (or bare relative path)
Specifies a path relative to the location of game.sgm on the running game's file system (either SPK or physical).  Not writable.

#/..
Specifies a path relative to the engine's System Asset Collection (SAC). Not writable.  The engine should declare the sphere_fs_system_alias extension if it supports this.

~/...
Specifies a path relative to the current user's save data directory.  This is the recommended location for storing game save data and is guaranteed to be writable (barring OS misconfiguration).  ~/ refers to the same location for all games, allowing save data to be shared between sequels, for example.

<Absolute Path>
A SphereFS implemention may allow absolute paths only if it declares the extension sphere_fs_absolute_path.
Title: Re: Sphere 2.0 API: SphereFS
Post by: Fat Cerberus on June 13, 2015, 09:52:04 am
Note that the semantics of ~/ will probably need to be different than ~sgm/ to maintain full compatibility, as existing games tend to assume the former is writable.  For example, it might be mapped to ~usr/ if the game is running from a non-writable medium like a package and someone opens a RawFile for write.
Title: Re: Sphere 2.0 API: SphereFS
Post by: DaVince on June 13, 2015, 12:06:17 pm
It's prepended with ~ to ensure it stays backwards compatible, I guess? It feels odd, but it's really not much different from % or $ placeholders. Maybe it's because ~ already conventionally means "home directory" on Linux.
Title: Re: Sphere 2.0 API: SphereFS
Post by: Fat Cerberus on June 13, 2015, 01:43:44 pm
Yeah, it's for backwards compatibility.  ~ means "location of game.sgm" in Sphere, so for consistency I followed the notation for the other escapes too.
Title: Re: Sphere 2.0 API: SphereFS
Post by: Flying Jester on June 13, 2015, 04:01:57 pm
Where does "/" go to in 1.5?
Title: Re: Sphere 2.0 API: SphereFS
Post by: Fat Cerberus on June 13, 2015, 07:22:55 pm
Nowhere.  It's a sandbox violation: "Invalid filename".
Title: Re: Sphere 2.0 API: SphereFS
Post by: Flying Jester on June 13, 2015, 07:34:44 pm
Then I'm curious why we need to specially qualify absolute pathnames with "~abs/". They would be invalid in 1.5 since the Unix root is invalid and Windows/DOS/Symbian filenames cannot have colons in them, and so no 1.5 game that works will use them.

A single / has the opportunity to be ambiguous, if you don't canonize properly. But since that's handled properly in 1.5, there will never be ambiguity between an absolute and a relative path.
Title: Re: Sphere 2.0 API: SphereFS
Post by: Fat Cerberus on June 13, 2015, 07:43:38 pm
The syntax is awkward on Windows though: /C:\path\file

But the abs notation has the same issue, of course.  Either way we have to maintain ~/ with its current semantics for backward compatibility, so for consistency's sake I kept it in the names of the new escapes as well.

Unless we make this a full emulated file system and abstract the underlying platform away entirely.  Then you'd have /usr, /sgm, etc. and something like /local where the entire physical FS is mounted.

Title: Re: Sphere 2.0 API: SphereFS
Post by: Flying Jester on June 13, 2015, 07:52:08 pm
I'm saying we just allow absolute paths the way you suggest they be handled, but with no special notation in paths just for them.

You'd just put C:\path\to\file, or /path/to/file depending on the environment. No need to specially qualify it, since it's not ambiguous, and the developer always knows how to specify it (unlike the other aliases, which could actually be anywhere in the fs tree).

That would also unify the notation, since then you allow valid pathnames (relative or absolute), but with several prefixes predefined.

I do think that this entire idea is much cleaner and generally a better idea than an entire emulated FS.
Title: Re: Sphere 2.0 API: SphereFS
Post by: Fat Cerberus on June 13, 2015, 07:58:14 pm
Hm, you're right. :)

I do still want to confine ".." to the sandbox though, even if the engine implements absolute paths.  Reason being, the game developer doesn't know where there game is going to be placed, so moving upwards relative to game.sgm is dangerous.  Absolute paths are generally provided by the user or temporary used by the developer during debugging, so this isn't an issue there.
Title: Re: Sphere 2.0 API: SphereFS
Post by: Flying Jester on June 13, 2015, 09:13:32 pm
Disallowing using a ".." out of an alias makes sense. I wouldn't say it's dangerous, but it is kind of a nonsense thing to do. From the perspective of the developer, that could lead anywhere. On a Unix system, it might not actually lead anywhere at all! Better to make it impossible than cause intermittent bugs.

A note about the sandboxing, and why it exists in Sphere the first place: I think it actually was a security measure. Sphere was designed in the '90s, to work with Windows 9x, Windows 2k, RHEL, and Debian. While the Linux distros had similar security to what we enjoy now (maybe even better, since they were more aligned with how Unix should work and not concerned with making raw device use easier for the desktop user), and Windows 2k had very almost all the security systems that Windows still has today (being Windows NT 6), in Windows 95 and 98 (the most common end-user OS at the time) not sandboxing the FS actually was kind of dangerous. You suddenly could write a virus in JS using Sphere under those circumstances.

I had experiences with the joy of the extremely fagile DOS-based fs in Windows 9x in my Visual Basic days. It's silly to think about it now, but after that occurred to me, it makes a lot of sense.
Title: Re: Sphere 2.0 API: SphereFS
Post by: DaVince on June 14, 2015, 05:39:22 am
I kinda like what N E O came up with in te other thread - prepending the path with something like a sphere:// protocol. Thoughts?
Title: Re: Sphere 2.0 API: SphereFS
Post by: Fat Cerberus on June 14, 2015, 08:39:37 am
I got the feeling that was more about a way to register the sphere protocol with the engine than anything internal - for example you could click a sphere:// link on a webpage and it would automatically download the game/run it off the internet.  Which would be awesome.

Using a protocol to specify filenames internally would be weird I think.  The ~ aliases feel more natural (read: Sphere-like), and I'm trying to avoid going down the Pegasus "new for the sake of it" route if at all possible.
Title: Re: Sphere 2.0 API: SphereFS
Post by: N E O on June 15, 2015, 08:02:48 pm

I got the feeling that was more about a way to register the sphere protocol with the engine than anything internal - for example you could click a sphere:// link on a webpage and it would automatically download the game/run it off the internet.  Which would be awesome.


Eventually this, yes. Also, a sphere:// link of some non-game file would open said file in whatever program is registered within Sphere to handle it, some thoughts follow:



So...yea.
Title: Re: Sphere 2.0 API: SphereFS
Post by: Fat Cerberus on September 11, 2015, 12:43:45 pm
I'm working on standardizing this for minisphere 2.0.  So far, to be SphereFS compliant, an API must:



I'll flesh these requirements out more as 2.0 development goes on.

Oh, and no worries: Classic functions such as LoadImage, LoadSound, etc. will retain the legacy behavior. ;)
Title: Re: Sphere 2.0 API: SphereFS
Post by: Fat Cerberus on September 17, 2015, 09:30:27 pm
I've also decided on the following requirement:

* A SphereFS compliant engine must report any path accessible through a SphereFS designator (~sys/ etc.) to be reported as such and never as an absolute path, even if the engine itself supports absolute paths.  This might seem onerous, but it greatly simplifies, e.g. debuggers by canonizing the relative path.

As a practical example of how this is useful: One of the things the Cell compiler will do when compiling a game is generate a source map which maps files in the compiled game to their locations in the original source folder.  The paths in the source map are relative, so if the engine always reports relative paths for its own scripts and other assets, the debugger can always locate the corresponding file in the project with little difficulty.
Title: Re: Sphere 2.0 API: SphereFS
Post by: Flying Jester on September 19, 2015, 05:59:18 pm
To me, there are only two examples of when absolute paths are useful to the engine:


The first is obvious, and necessary for most platforms (maybe not necessary for Windows, I am certain not for Haiku, or for XFCE on Unix).
The second is the only reason I really think absolute paths are important to script. This lets you, for instance, save a screenshot to a directory that the user specifies, or load an image to use for a skin from anywhere on the computer. In both those cases, the platform differences between paths also don't matter, since for the most part it will be up the libc on the platform (or the Mon runtime/CLR, etc), and it will know how to handle the platform's paths.
Title: Re: Sphere 2.0 API: SphereFS
Post by: Fat Cerberus on September 19, 2015, 06:30:39 pm
Yes, another thing I didn't mention above is that SphereFS standardizes on forward slashes as a separator.  / works on all major platforms, so it seemed the best choice.

Basically what my intent with the SphereFS standard is, to make so that a relative path is never ambiguous--"sounds/munch.wav" always refers to the same file regardless of which API is accepting the filename.  This has a few advantages:

* More flexibility for organizing assets without having to resort to ../ trickery
* No extra path processing needed to use a path constructed for one type of API with an unrelated function--opening sound files as rawfiles instead, for instance.
* As mentioned earlier, simplifies debugging by always using the relative path to refer to assets.  This is particularly important when the game being debugged is SPK-packaged.

It also makes package support a lot easier to implement if the relative path is canonical, for reasons I'd imagine are self-evident.

Absolute paths are unambiguous by definition, so supporting them alongside SphereFS is no problem.

Anyway, now that I've got this idea somewhat solidified I wonder if I should submit it as a pull request on the Pegasus repo.
Title: Re: Sphere 2.0 API: SphereFS
Post by: Rahkiin on April 29, 2016, 05:01:12 am
I have one issue.

Modules.

If I do "require('./file')", how does that work? Do we have two different systems, one being SphereFS for any file operation, and another syntax for require/resolve?

I think we should rethink this SphereFS to work with modules, for Pegasus. In Pegasus you really want relative paths for modules.

I do like having access to the  games user directory: a requirement.
And indeed, absolute paths are neccesary for game loading/resource loading by user.  Prefixing such is dangerous.

Also, I would not make this an extension but core. So you can always depend on it. It is annoying to do:

Code: [Select]

if (engine supports spherefs)
    use path '~usr/save1.json'
else
    use path 'C:.....'


When it is not neccessary, if we just enforce this kind of system.
Title: Re: Sphere 2.0 API: SphereFS
Post by: Fat Cerberus on April 29, 2016, 09:38:34 am
SphereFS itself is not intended to be an extension, the only thing I intended to be optional when I designed this is absolute paths since the default case is a sandboxed FS.  The prefixes like ~usr/ would always be available.

As for modules, relative IDs are resolved as described in CommonJS: relative to the calling module.  Logically the module namespace is separate from the file system because any other setup risks breaking CommonJS spec.

I know you're using Node.js as an exemplar when thinking of modules.  Node conflates the CommonJS namespace with the file system and makes for some really confusing semantics which deviate from CommonJS in a lot of cases.  I'll discuss that bit more on IRC.
Title: Re: Sphere 2.0 API: SphereFS
Post by: Rahkiin on April 30, 2016, 12:49:20 pm
kay, I think you need to explain indeed.

But I do want to be able to:

require("random") to get the system random module.
fopen("~usr/game1.json") for savefiles
new Image("~??/images/loadingImage.png") for loading images
require("./teleporters"); to load my own teleporters.js file (in same dir as current module)
require("teleporters"); to load the custom teleporter package.

Speaking of packages, are we using CommonJS-ish packages?

I am on IRC today
Title: Re: Sphere 2.0 API: SphereFS
Post by: Fat Cerberus on April 30, 2016, 01:03:46 pm
All of those should work as you describe in the final version of minisphere 4.0/Sphere 2.0.  Note that the prefix is not needed for files which are part of the game--that is, the relative path is canonical so you can specify that image as "images/loadingImage.png" and that would be completely unambiguous.  The ~sgm/ prefix could probably be removed altogether, the main reason I added it was for backward compatibility with the Sphere 1.x APIs which are relative to a subdirectory by default.  Pegasus is not backwards compatible so it's not needed there.
Title: Re: Sphere 2.0 API: SphereFS
Post by: Fat Cerberus on May 04, 2016, 03:41:11 am
In my Pegasus fork I simplified the SphereFS prefixes.  For Pegasus we will have:

@/ - root of sandbox
~/ - for user data, etc. (like UNIX)
#/ - engine assets

I would love to implement this in the main branch also but care must be taken not to break legacy compatibility.  ~/ means "root of sandbox" in Sphere 1.x. :-\  LoadImage() et al. will probably need a custom path handler to resolve the conflict.
Title: Re: Sphere 2.0 API: SphereFS
Post by: Rahkiin on May 04, 2016, 05:40:07 am
Not sure I like this change. The previous setup (which i implemented 15min ago), is much more clear. Also, ~/ is confusing on *nix systems because it actually means the User folder, not the app-data folder.
Now if a path ~/ is passed for being meant as /Users/name/ it cant be picked up as an error.

When using ~usr/ and ~sys/, it is easy to remember which is which. With @ and # it is not
Title: Re: Sphere 2.0 API: SphereFS
Post by: Fat Cerberus on May 04, 2016, 11:03:12 am
~usr/ doesn't actually refer to AppData but rather the user's home directory (or a sub directory thereof, for sandboxing reasons).  I don't particularly like the idea of hiding save data from the user.  And I thought #/ was pretty clear too, since it's often used as shorthand in shells for "you are root" so it makes sense that it refers to the engine itself.

In any case "~usr" is ambiguous, since on a UNIX platform /usr/ isn't actually a user folder but in reality is a system directory.  "UNIX system resources" or something like that, and non-root can't write to it.
Title: Re: Sphere 2.0 API: SphereFS
Post by: Rahkiin on May 04, 2016, 11:58:40 am
You cant do ~usr/ as just the user home dir... I can't assume writability of ANY place for my app other than /User/<user>/Library/Application Support/NameOfApp/. Neither should you. Same for windows with LocalData.

You said it is a Sandbox, then it should be a true sandbox. No easy access to user files. And no assumptions. That is what I implemented and that is what I really liked about it.

Then lets make it ~pref/ or ~user/ instead of ~usr/ if you think it will be confused with /usr.

I would like to hear some opinions of others as well.
Title: Re: Sphere 2.0 API: SphereFS
Post by: Fat Cerberus on May 04, 2016, 12:21:38 pm
How is the user's homedir (specifically, their documents folder) not guaranteed to be writable?  That wouldn't be very useful for the logged-in user.  Anyway I'm not a fan of games that hide their save data from me by putting them in Application Data.

And it IS a sandbox.  In minisphere 3.0 ~usr/ refers to <user-documents>/Sphere 2.0/SaveData and on Linux it's mapped to /home/username/Documents/Sphere 2.0/SaveData.  This is documented.  Since it is impossible to do ~usr/../ there is no risk of the game subverting the sandbox.

Long story short, the semantics haven't changed.  Only the prefixes have.
Title: Re: Sphere 2.0 API: SphereFS
Post by: Rahkiin on May 04, 2016, 12:32:48 pm
I don't like it when apps create folders in my user directories... most of the time i just delete them.
Title: Re: Sphere 2.0 API: SphereFS
Post by: Fat Cerberus on May 04, 2016, 12:34:40 pm
Fair enough.  Either way, I didn't change the meaning, only shortened the prefixes.  That's all.  Sorry if there was any confusion. :)
Title: Re: Sphere 2.0 API: SphereFS
Post by: Flying Jester on May 04, 2016, 05:48:11 pm
on Linux it's mapped to /home/username/Documents/Sphere 2.0/SaveData


More common would be `/home/username/.share/local/Sphere2/SaveData`, that's how most programs store data like that in Linux. And it's different in real Unix descendants...but Allegro doesn't work there anyway.

Linux/Unix folk will also be unhappy if you start putting spaces in their filenames.
Title: Re: Sphere 2.0 API: SphereFS
Post by: Fat Cerberus on May 04, 2016, 06:11:36 pm
Yeah, I'll probably change it from "Sphere 2.0" to just Sphere.  I was leery of putting a space there myself, but having a camelcase directory inside My Documents looks just as odd to a Windows user as "Sphere 2.0" does to a Linux user.  But yes, I agree that we shouldn't be creating paths with spaces under *nix.

In any case the standard doesn't need to specify the exact path, the only requirements for the "user data" directory from an engine implementor's point of view is: 1. It must be writable and 2. It must be user-local whenever possible.  Whether that means the user's documents folder or some other "app data" location doesn't matter as long as the above requirements are met.

That's my take on it anyway.
Title: Re: Sphere 2.0 API: SphereFS
Post by: Fat Cerberus on May 08, 2016, 03:18:42 am
I started work on a specification for SphereFS, GitHub doesn't seem to have a done a good job with the markdown though (or I screwed up the layout, that's possible too...):
https://github.com/sphere-group/pegasus/blob/master/design/SphereFS.md

I've never written a specification before, it's kind of fun being able to throw around words like "shall" and "must" with reckless abandon. :P

:D
Title: Re: Sphere 2.0 API: SphereFS
Post by: Rahkiin on May 08, 2016, 07:25:30 am
Nooo why are you going with ~/ an #/ now :(
Title: Re: Sphere 2.0 API: SphereFS
Post by: Fat Cerberus on May 08, 2016, 12:53:17 pm
What can I say, I prefer the single-character aliases.  I added more to the spec, laying out requirements for writability/non-writability for the different locations.  For example I added a requirement that the engine should not allow write access to the game directory unless it declares support for the Sphere 1.x API.
Title: Re: Sphere 2.0 API: SphereFS
Post by: Flying Jester on May 08, 2016, 02:59:15 pm
That's what I did in TurboSphere. #/ meant the system directory.