It adds features to the editor, not to the engine.
That would change if we figured out how to make a C#
and C-linkaged fat-binary dynamic library.
That would change if we figured out how to make a C# and C-linkaged fat-binary dynamic library.
That would be unbelievably awesome, but I'm somewhat doubtful if that's possible. If that was done, combined with a V8+OpenGL TurboSphere, would push Sphere firmly into the realm of next-generation 2D game engines. Heck, with something like that, it would probably not be to far to call Sphere the most advanced open source game engine.
So basically it would work something like: Someone writes a map engine, like Majestic, in C, and both the editor and the engine could use it? The editor could edit Majestic maps and the engine could run them? Wow, that would be cool. (I'm just using Majestic as a random example, but wouldn't it be cool if engine and editor plugins could be written in JS as well as C?)
With reflection it is not impossible to create bindings between the two. But the bindings must be made, what we want here is a seamless transition, right? C# and C would have to compile to an IL before that could happen. Mono + CIL would then have to be the answer here.
I've been reading about it. The usual thing to do is to use C++/CLI. Which sounds fine on Windows. Not so much anywhere else.
What I'm really imagining is a shared library that has some symbols that are visible and use the C# conventions, and some symbols that have C linkage.
Depending on what a C# DLL needs to be, though, it still might be possible. The linkage and symbols needed for TurboSphere are very simple (theoretically, you only need to return a list of addresses, some ints, and some C strings, and you could get the addresses from, say, a statically linked library that is linked to the plugin), and there should be a way to have some symbols in a DLL or SO be C# and some be C? That has to be possible. I know that theoretically, it is. All the SOs and DLLs for TurboSphere have mixed C and C++ linkage, and in the past I've made DLLs that had mixed Pascal and C linkage, although in both those cases the compiler supports making the binary file that way.
It's a slightly unnecessary idea, though, you could just write the a library and have two separate plugins for the engine and editor that both call the library. But it would just be so incredibly cool and streamlined to have a single DLL that works as both, and is being called from both C/C++ and C#.
Radnen, when you make a DLL with C#, what kind of intermediate object files you get? Is it a similar process to the old Windows compilers before .NET, and the C/C++ compilers?
I've been reading about it. The usual thing to do is to use C++/CLI. Which sounds fine on Windows. Not so much anywhere else.
That why: Mono.
Radnen, when you make a DLL with C#, what kind of intermediate object files you get? Is it a similar process to the old Windows compilers before .NET, and the C/C++ compilers?
I ... don't know! I don't know how to show the individual object files when building a C# program. I'm sure there is a way though.
They don't get left over in an intermediate directory (that way you don't need to recompile those parts if the code hasn't changed for them when you compile again)?
I think (I don't use .NET much) that I'm thinking of the object files the IL would ouput. I'm sort of assuming the whole compilation process is similar to how GCC/MSVC works, just with a C# to IL compiler first, which could be very wrong. Just C# is compiled to IL, which is compiled to object files, which are linked to create a binary. Which could be not how it works at all.
I'm hoping that the pre-linking object files may be the same format across MS development tools. In which case, I'm hoping to do some linker magic and just have it all work out. Otherwise...I'm not sure what, otherwise.
It's a slightly unnecessary idea, though, you could just write the a library and have two separate plugins for the engine and editor that both call the library. But it would just be so incredibly cool and streamlined to have a single DLL that works as both, and is being called from both C/C++ and C#.
There's a big cool factor and a very big convenience factor to having just one DLL. Imagine: you download particleSystem.dll (possibly even through the editor) and you get a particle editor in Sphere Studio and TurboSphere suddenly grows particle capabilities.
That's got me pretty excited. Maybe I'll prototype some possible features for plugins in 1.7. Given the age of Sphere's codebase and the somewhat poor documentation on MDN combined with extensive API changes, SpiderMonkey upgrades will be very hard to pull off, so I think I'll resort to the current Sphere implementation being a testbed for new ideas (which it sort of already is, with the particle system). If you guys don't mind, I think that's the direction I'll take with Sphere 1.7: new features, instead of updated libraries like I originally planned.
That sounds fine to me!
Yeah, the unified DLL/SO thing is really an aesthetic, and partially a convenience. But if it worked, it would also really impress folks who knew what they were looking at--me included! Plus, I'm, not sure anyone else has done just exactly what we're talking about, and this the perfect use for it.
About how I am hoping to do it, I'm hoping that object files generated by MSVC are the same format as the ones made by the MS C# .NET compiler at some point along the way, and one or the other linker (if they are different) could be used to link both the TS plugin's object files and the editor's plugin's object files together and make a single binary. I highly doubt that would work, but it is a good thing to try first I think. The same might work (and I actually suspect will work better) using LLVM or GCC on Linux and Mac.
Yeah, the unified DLL/SO thing is really an aesthetic, and partially a convenience. But if it worked, it would also really impress folks who knew what they were looking at--me included! Plus, I'm, not sure anyone else has done just exactly what we're talking about, and this the perfect use for it.
It just sounds awesome to say, "Yeah, just download particleSystem.dll and put it in Sphere/plugins/" as opposed to downloading several plugin DLLs and putting them in various folders (although if there were a central repository the editor might be able to automate that). But it would still be very nice for distributing plugins on the forums: you'd just have to attach one file for people to download and put in one location.
About how I am hoping to do it, I'm hoping that object files generated by MSVC are the same format as the ones made by the MS C# .NET compiler at some point along the way, and one or the other linker (if they are different) could be used to link both the TS plugin's object files and the editor's plugin's object files together and make a single binary. I highly doubt that would work, but it is a good thing to try first I think. The same might work (and I actually suspect will work better) using LLVM or GCC on Linux and Mac.
Unfortunately, C# doesn't actually generate object files; it just compiles everything at once, every time (this is a side effect of not having header files). The C# compiler is fast enough that this is barely noticeable. :(
I saw this in an article by Eric Lippert recently, who seemed to think it was a good idea (avoids the inconveniences of header files, like writing every function definition twice).
EDIT: Found it: http://blogs.msdn.com/b/ericlippert/archive/2010/02/04/how-many-passes.aspx (http://blogs.msdn.com/b/ericlippert/archive/2010/02/04/how-many-passes.aspx)
I figured this was going off the topic of my Website page and so split it here so it can be found and read a bit more easily. :)
I already knew C# had a multi-pass compiler, so that it didn't require header files (it technically does, it's just that they are the same file as the code file). C++ could easily have a compiler just like how C# works. Anyways, C++ also goes to CIL. I wonder, what happens if TurboSphere was built with Mono's C++ compiler? Does mono even convert C++ into MSIL ("CIL") or the Mono version of it? If so, could the resulting library just be used as if they were any other .NET/Mono DLL?
TurboSphere has accidentally turned out to be written in pretty portable C++ (it has moved from MSVC '08 to 10 to GCC C++ to C++11 with only new warnings each time). It wouldn't surprise me if it also compiled as C++/CLI too.
I've sort of been hoping to add on the C-linkage plugin interface from TurboSphere on the end of a .NET DLL, since C-linkage is very simple, and the plugin interface I specified is very simple since it means that both the plugin side and the engine side talk to V8 directly.
I'll have a closer look at Mono (I don't really know that much about it) and see what it would take, if it is possible, to compile (or..something) TurboSphere to use it (?).
All I know is that C++, VB, and C# compile to CIL. Mono is a reimplementation of C#, so it must also compile to an IL (one that is cross platform, that is). In that case, I wonder if MonoDevelop compiles C++ and VB too to their IL, or if it is *just* C#?
Hmm, well this is a good start: http://www.mono-project.com/CPlusPlus, but it has some limitations.
I know that the way I compile TurboSphere with MSVC, it does not use CIL. I think it doesn't anyway. It outputs object files, which the linker links to make a binary. I could be wrong, if the object files are CIL. In which case, the Visual Studio linker should, somehow, be able to take both the CIL object files from a TurboSphere plugin, and the CIL output for a Sphere Studio plugin, and make a mixed library that contains the code for both. But again, I've never heard (and I've looked) of such a thing being done. In fact, I think (and am more sure that) such a thing is possible for any two languages supported by the GCC, since the object code is output by each language front end to be processed by the single backend, but I can't seem to find anything about that, either.
I think I'll get some MS C#, and see how it works. If this even just worked on Windows, that would be amazing. Radnen, is the code for a Sphere Studio plugin available online? Even an unfinished one. I would like to experiment with this.
Making a single routine handle, say, opening a filetype, when the editor and the engine ask, is another story. I'm not worried about that yet. Even if that isn't possible, the same end result can be achieved.
Another problem is that a TurboSphere plugin needs to also have C++ code internally. The plugin interface, while only given C-linkage, gives the engine addresses to C++ functions. A plugin needs the STL library to talk to V8, and the plugin directly calls V8 functions and V8 quite extensively uses templates. The reason I could implement the plugin system so fast was that I unloaded all the heavy lifting onto the plugins, the engine does very little now. It mainly starts up a V8 context, compiles scripts (which could easily be a plugin's responsibility), and intializes plugins. To change that, I would have pretty much abstract away the interface to JS. Which is an immense undertaking, and begins to tempt that black hole of total extensibility.
EDIT:
Apparently, we aren't the first to seriously wonder. http://blogs.msdn.com/b/texblog/archive/2007/04/05/linking-native-c-into-c-applications.aspx (http://blogs.msdn.com/b/texblog/archive/2007/04/05/linking-native-c-into-c-applications.aspx). And apparently, the MSVC linker is the answer? I'm going to give it a shot.
If anyone else wants to cast their lot in, I created the plugin getkeystring just for testbed scenarios (like this, but I actually expected the first use would be for making plugins entirely in other languages, not for experimenting with kind-of-fat binaries).
Well...
This might not be so hard after all.
Check this DLL out. It works as a TS plugin, and I think it implements IPlugin for your editor as well.
To try this at home, from the Visual Studio 2010 command prompt:
C:\spherestudio\Sphere.Plugins>csc /target:module
/r:"C:\GitHub\spherestudio\Sphere Editor\Libs\dockpanel
suite\WeifenLuo.WinFormsUI.Docking.dll" IPlugin.cs IPluginHost.cs Properties\As
semblyInfo.cs
That makes IPlugin.netmodule.
C:\turbospheresrc\plugins\getkeystring>cl /c /
MD getkeystring.cpp
Which makes getkeystring.obj. Put them in the same directory.
C:\turbospheresrc\plugins\getkeystring>link /L
TCG /DLL /ASSEMBLYMODULE:IPlugin.netmodule "../../lib/v8.lib" /OUT
:fatplugin.dll getkeystring.obj IPlugin.netmodule
Which makes fatplugin.dll. I have no idea how well it works as a Sphere Studio plugin, but it is a completely working TurboSphere plugin that replaces getkeystring, and is 4kb (about the size of the IPlugin.netmodule) larger than the normal getkeystring.dll.
It would be another matter to have it call common routines from both sides (if the C# side is really working correctly at all). But I for one think this is better already than having two separate plugin files for a single purpose.
EDIT:
Well...I see now what that actually did. Clearly not what is needed to make the C# code I wanted to be put into the dll.
I'm going to try to figure out how to get the command line output from Visual C#.
The reason I could implement the plugin system so fast was that I unloaded all the heavy lifting onto the plugins, the engine does very little now. It mainly starts up a V8 context, compiles scripts (which could easily be a plugin's responsibility), and intializes plugins. To change that, I would have pretty much abstract away the interface to JS. Which is an immense undertaking, and begins to tempt that black hole of total extensibility.
With a little C++ template magic, it's not hard at all to bind C++ functions to JS without the C++ knowing much. See the way 0 A.D. (http://play0ad.com) (an open-source RTS, which I'm a fairly active contributor to) does it, particularly ScriptFunctions.cpp (http://trac.wildfiregames.com/browser/ps/trunk/source/gui/scripting/ScriptFunctions.cpp).
Oh, and that link is very interesting!
It would be another matter to have it call common routines from both sides (if the C# side is really working correctly at all). But I for one think this is better already than having two separate plugin files for a single purpose.
That actually wouldn't be too hard. The C# could just make P/invoke calls to the C++ side of it's DLL, so all the functionality would be in C++ and the GUI stuff would be in C#.
Also, I forgot to mention that that dll up there was made with ILmerge. It works more than than the MSVC linker's output that I tried, and Sphere Studio gives a different error trying to open it (instead of badimageformat, it's some error with 'type' in the name, I don't exactly recall what).
But I made that dll using a getkeystring.dll that was compiled for clr, but not clr pure, which made ILmerge give an interesting message semi-complaining about it not being pure. It might actually work if it was clr pure.
I have been making (mostly?) any nontrivial function for TurboSphere have both a clean backend and a V8 frontend function. It just felt right to do it that way, but it certainly should make sharing functions between the editor and the engine easier.
I have to say, I'm not liking this whole fat plugin idea. It's great from the point of the view of someone using the editor, as they can just install one DLL and have it automatically work in the engine, but the issue comes with distribution. When I go to distribute my game, do I really want to add bloat by including design-time code with the engine? Probably not. Let's not forget that more code is usually needed for the IDE side than the engine side...
This is one of the reasons recent versions of Visual Studio actually seperated the design-time components from the runtime stuff, to reduce the size of the redistributables.
If that's a big issue (it might or might not being, I'm thinking no), it would barely be any more effort to also make a pure-TurboSphere plugin without any editor code. However, I'd rather just distribute the fat binaries: one of the nice things about Sphere games is that they are easily moddable, but if you get rid of editor functionality that was used to make the game mods suddenly become a lot harder. In other words: include the fat binaries for the curious people who like to mess with things. ;)
I actually wouldn't mind if there was some way, for large binaries (at least) to have two versions. One version is intended for just the engine and another one for both. The one for both the developers use, and the one just for the engine is for release. I mean it's really the plugin writers choice to create a single fat binary, or two binaries (engine+editor, or just engine).
It's pointless to have a plugin for just the editor, unless all it does is modify something non-game related. I mean, it would be weird to have a battle system editor plugin, but not have the battle system it creates be used.
I have to say, I'm not liking this whole fat plugin idea. It's great from the point of the view of someone using the editor, as they can just install one DLL and have it automatically work in the engine, but the issue comes with distribution. When I go to distribute my game, do I really want to add bloat by including design-time code with the engine? Probably not. Let's not forget that more code is usually needed for the IDE side than the engine side...
This is one of the reasons recent versions of Visual Studio actually separated the design-time components from the runtime stuff, to reduce the size of the redistributables.
I for one would still distribute the engine-only plugins as well. It's quite simple to make them--that's all I have right now. That's also why making a core TS plugin that needs any .NET or Mono or anything other than V8, SDL and maybe OpenGL, and possibly a separate sound library to work rubs me the wrong way. I would be willing to have the fat binaries need .NET, since the editor would also need .NET. But if .NET or Mono was needed to make this idea work, I would make absolute certain that engine-only versions also existed of all the plugins I make.
As far as the editor plugins, it makes as much sense to have both the engine and editor function at once, since what good is editing something if you can't run it? And from my experiments, it seems much simpler to make something that qualifies as a TurboSphere plugin than an editor plugin, too. Every single failed attempt I had to make a fat plugin still worked as a TurboSphere plugin.
At the same time, I know that most engine plugins won't be more than 1 MB, and just about every one should be less than 100 kb. I don't know about the editor, but almost the entire bulk of what TurboSphere requires to work, including itself, is the graphics and audio libraries and V8--and V8 is a vast majority of that size anyway. And the biggest plugins, the graphics plugin (and although I do not know for certain, I believe the sound plugin as well), generally won't need many editor-side functions at all, since editing images should probably be a generic plugin (and be used for image files, fonts, windowstyles, etc.), and for most audio files I wouldn't expect the editor to do much of anything to the files anyway.
In fact, I can fit the entirety of TurboSphere (all the binaries, or alternatively all the source including the V8 source) on a single floppy disk with room to spare. It's not much bloat at all as we have it now. And we can always move away from this system later if we choose. We could always have some plugins that are fat and some that are only distributed as separate files if we wanted, too.
I think, for sounds, a fat plugin would be the best thing ever, it exemplifies everything about why it's needed: 1, we ensure the same filetypes are read by the editor and engine, and 2, that they sound exactly the same. If we can manage that, then it's really the best thing ever. This same logic applies everywhere else, images, maps, spritesets, etc. :)
Pretty much every other PC-based game engine I've seen, from RPGMaker and ika to Unity and XNA, only distributes engine runtimes with playable games. One would need to actively seek out the appropriate editor if they wanted to mod the game in question. Think about offering up the fat binaries only as optional downloads, while engine distribution has the engine-only libraries; that was probably your plan anyways, right?