Hello! What a busy week it has been, and I have some good things to share. In just over a little 3 days of time, I worked for a lot of time to implement the new idea I mentioned I had last week, and it was actually working quite well! I managed to implement the entire basic structure of it, as well as port over 8 of ScummVM’s 79 engines to adapt to it!
So, what’s the deal with this, and what did I do? The “idea” was just me getting a basic understanding of how things “could” work in a way.
Well, we were at the “createInstance” bridge where I left off last week. This is a member function of either a MetaEngine or an AdvanedMetaEngine class. The problem with this was this had engine code in it, so it couldn’t be included in the executable. The basic idea around this was simply to remove it as a class – member function, and instead mark it as an export symbol. What we could then do, is in base class, provide a default functionality. These functions were a Pure Virtual Function, so I simply removed it in order to get what I wanted to achieve.
The “createInstance” method though, would only be exported if we were using a Dynamic-Plugins setup. If the entire engine would be to build into the executable, there wouldn’t be any point in doing this, so for those, we still override the base class’s function.
If a child class overrode this function, the vtable generated would point correctly to the createInstance method, and now the static plugins work as intended. If a child class didn’t override this function, that would mean the class would use it’s default implementation wherever used. What could we possibly do here? That’s right, we can simply hook up plugins to find a symbol in the engine libraries we have loaded.
Before a game is actually started, we unload all engine-type plugins to save memory and then try to run the game. Running the game usually has the first step of instantiating the engine, so normally when we actually reach to the point where the default “createInstance” is called, we only have 1 engine plugin loaded as a library.
Then, the default implementation basically consisted of fetching this plugin and trying to load a symbol. We have 3 main plugin classes in ScummVM, “Plugins”, “StaticPlugins” & “DynamicPlugins”. When I fetch a plugin and try to call a function, it would need to be present in the relevant plugin interface, obviously. So, I made pure virtual functions in the Plugins class and overrode those in the Static & DynamicPlugins classes. This way, when I have a “Plugin *plugin”, and I use a “plugin->tryInstantiateEngineFuncSymbol()”, it looks up in the vtable and proceeds to call the one in DynamicPlugins, as it handles everything dynamic-related things.
Over there, the process is very simple. We find the symbol, and if we can find it, we call it. This way, the engines are instantiated, and then everything proceeds as usual because the bridge has been crossed.
Since we now have a difference between MetaEngines & Engines, they are given different PLUGIN_TYPES, which is basically an identifier. To implement all of this though, I had to make some more changes for the existing functionalities.
For example, we have a unloadPluginsExcept method, which is run before a game starts, and as I mentioned earlier, only keeps a matching plugin in memory. This method takes in a PLUGIN_TYPE, and a plugin itself.
Normally, we just fetch everything in memory of the type mentioned, and comparison between pointers would tell us which one to keep. But, since we now differentiate between MetaEngines & Engines, this method gives in a TYPE_ENGINE and a pointer to a MetaEngine plugin. The way to solve this is pretty simple, we have a method in metaengines called getEngineID. We reconstruct the name of the engine plugin with the help of this, add prefixes and suffixes if necessary (for example on windows, it becomes engine.dll) and then simply fetch each plugin in memory of the type ENGINE, try to match it with our generated filename. This successfully lets us achieve what was intended.
With the basic structure complete, I thought now was the time to go engine by engine and incorporate the new changes. The 8 engines I mentioned were ported earlier were – Agi, Plumbers, Drascula, Dreamweb, Lure, Pink, Sky & Scumm. There were some things that would need shifting around – we just place the statically linked MetaEngine code into one file. After that, do the whole checking thing for createInstance, exporting it or not etc… For the Scumm engine though, there was a bit of an issue which needed a little more working around.
Great! Everything seems fine, so what’s with the “back at square 1” in the title? Well, as it turns out, createInstance is not the only bridge that would need to be crossed over. If you remember when I mentioned what the MetaEngine does, is it usually handles all metadata and things normally the engine wouldn’t do. The engine takes care of running the game, while the MetaEngine for other helper stuff. This helper stuff includes instantiating an engine, but it also handles listing saves, creating thumbnails, and querying for metadata for the savefiles, and many more.
It includes listSaves related things, which can have some engine dependencies, something I found when I was looking at the Sword25 engine. What it does, is basically creates an entirely new class, which itself has many engine dependencies, so obviously it cannot be linked statically. So, this is another bridge.
It’s different for each engine though. In the 5 engines I ported, each had bare minimum for saving related things, so some shifting around of code made it work well. What that meant is even without loading a plugin, or having one entirely, we could still see savefiles, along with the thumbnail if it supported. It’s not the same for all engines, as I pointed out earlier with Sky25 it’s a whole different thing altogether.
But wait, with the basic structure done and solving the createInstance problem couldn’t we do the same thing with these as well? Well, yes. Implement the base variant in the class, which calls DynamicPlugins to load the symbol and call it. We would then need the individual MetaEngines to not be defining it as a class function, so what I did with createInstance could be replicated and would work.
However, the main issue with this is that we have to include it as a class function if using static, otherwise like a normal function, so the code itself looks like
#if PLUGIN_ENABLED_DYNAMIC(ENGINE)
extern "C" PLUGIN_EXPORT function signaturehere {
#else
class function signature here {
#endif
...
normal definition of function
...
}
This doesn’t look nice, does it? If it was once, maybe it would’ve been ok but to do this as many as 3 times would be a bit too much, and code starts looking pretty weird.
So, recent discussions Eugene has pointed out that I could actually try to separate the MetaEngine class itself, so detection related things will be placed in one MetaEngine, while the dynamic things in another. We have so many engines, that I probably need to scope out many more of them and understand how this will work well. For now, I have the basic structure in mind, which involves adding 2 new classes (because we have a MetaEngine & AdvancedMetaEngine), but the work could be done with the addition of just one more class was suggested as well, so I’ll have to keep that in mind.
The existing hierarchy/structure is something like this:
- We have PluginObject.
- MetaEngine derives from a PluginObject.
- AdvancedMetaEngine derives from MetaEngine.
- Plugins have as a member variable, a pointer to a PluginObject.
- A PluginProvider, which basically helps in creating the plugins.
The new additions will be 2 new classes, and they’ll be related to MetaEngines.Those 2 classes will be called (or at least for now, because the name must be nice enough for everyone) MetaEngineConnect (inherits PluginObject) & AdvancedMetaEngineConnect (inherits MetaEngineConnect) so something like the original structure is replicated. These will then have functions like “createInstance, queryMetaDataForSave, listSaves” at least. I’m not sure if that’s exactly going to be it because I have more thinking to do.
So, perhaps not back at square 1 but just the square ahead of it maybe? The engines already ported would need reworking, and many more things, so I just shifted to a new branch today and cherry-picked some of the commits from the old branch, which I think would be helpful. There are 79 engines in ScummVM & 5 engines in ResidualVM, so I best get back to work now! Phew, much more work to do!
That’s the main thing I’m focusing this week, division of MetaEngine classes, and connecting everything once again properly! This possible way could be further improved and because this is something that will touch all engines, I should be more careful than ever in my thought process. That’s it from me for now, see you again next week with some new progress on this hopefully!