Wrapping it up

GSoC is almost over, less than a week now. Time to wrap things up and have a look at what has been done and what is still missing.

The overall goal was to bring the 3d functionality of the original Wintermute Engine to ResidualVM after the 2d parts had already been ported to ScummVM during previous GSoC events (based on Wintermute Lite instead), using OpenGL. Major steps include:

Rendering of 2d sprites using OpenGL. Also adding some of the 2d transformations supported by WME. Some functions are still not implemented although it is not clear if a 3d game is actually using them.

https://github.com/residualvm/residualvm/pull/1643

https://github.com/residualvm/residualvm/pull/1651

https://github.com/residualvm/residualvm/pull/1674

https://github.com/residualvm/residualvm/pull/1678

https://github.com/residualvm/residualvm/pull/1679

Loading of scene geometry and camera settings from a .3ds file and being able to display it for debugging purposes. A lot of the code here was taken from the original engine and adjusted where necessary. The .3ds loader was written from scratch as well as all OpenGL code.

https://github.com/residualvm/residualvm/pull/1647

Loading of models from a .X file, rendering and animating such models. In the first PR, again a lot of code was integrated from WME with minor adjustments (changing the math code or everything related to coordinate systems, for example). Most of my work consisted of the .X loader and everything mesh related like rendering and animation updates of the mesh. The first PR also contains some work related to path-finding. Again, a lot of the code was imported here but parts of it had to be changed.

https://github.com/residualvm/residualvm/pull/1649

https://github.com/residualvm/residualvm/pull/1662

https://github.com/residualvm/residualvm/pull/1663

https://github.com/residualvm/residualvm/pull/1665

https://github.com/residualvm/residualvm/pull/1666

https://github.com/residualvm/residualvm/pull/1668

https://github.com/residualvm/residualvm/pull/1670

https://github.com/residualvm/residualvm/pull/1672

https://github.com/residualvm/residualvm/pull/1676

https://github.com/residualvm/residualvm/pull/1680

https://github.com/residualvm/residualvm/pull/1681

https://github.com/residualvm/residualvm/pull/1682

https://github.com/residualvm/residualvm/pull/1683

https://github.com/residualvm/residualvm/pull/1684

https://github.com/residualvm/residualvm/pull/1686

https://github.com/residualvm/residualvm/pull/1687

https://github.com/residualvm/residualvm/pull/1688

https://github.com/residualvm/residualvm/pull/1689

Object selection and attachment of items to 3d actors.

https://github.com/residualvm/residualvm/pull/1653

Lighting and shadows. Lights are loaded from a .3ds file and set accordingly to script input. Stencil shadow volumes have been implemented and seem to be very stable. Shadow mapping, as supported by the original engine is still missing. Simple shadow textures which are displayed at the position of an actor are implemented but potentially buggy.

https://github.com/residualvm/residualvm/pull/1657

https://github.com/residualvm/residualvm/pull/1669

https://github.com/residualvm/residualvm/pull/1691

https://github.com/residualvm/residualvm/pull/1692

https://github.com/residualvm/residualvm/pull/1671

A shader based renderer has been implemented and the a 3d renderer interface was added to make the rest of the code independent of the specific renderer. Plenty of shader related stuff can be found in the above mentioned PR’s as missing function were implemented or bugs fixed only after the renderer was added.

https://github.com/residualvm/residualvm/pull/1659

Persistence for the ported 3d classes was implemented in a way that there would be compability between savefiles of ScummVM and ResidualVM.

https://github.com/residualvm/residualvm/pull/1655

Some somewhat unrelated PR’s, mainly smaller fixes and cleanup of the code.

https://github.com/residualvm/residualvm/pull/1660

https://github.com/residualvm/residualvm/pull/1661

https://github.com/residualvm/residualvm/pull/1664

https://github.com/residualvm/residualvm/pull/1667

https://github.com/residualvm/residualvm/pull/1677

https://github.com/residualvm/residualvm/pull/1690

https://github.com/residualvm/residualvm/pull/1693

https://github.com/residualvm/residualvm/pull/1697

https://github.com/residualvm/residualvm/pull/1698

https://github.com/residualvm/residualvm/pull/1699

The following PR did not go directly in ResidualVM, but first into ScummVM and was then synchronized as the fixes touched the common code between the two Wintermute ports.

https://github.com/scummvm/scummvm/pull/2345

This PR was just about integrating the existing ScummVM port into ResidualVM, so none of this code was actually written by me. It is included here for completeness sake.

https://github.com/residualvm/residualvm/pull/1644

Displayable results

The Wintermute games “Alpha Polaris” and “Mental Repair” have been successfully completed with the current port. The graphical output matches the original, at least to my current knowledge.

Known issues

Shadow mapping is used by a lot of games instead of shadow volumes and was not implemented, as mentioned above.

Sorting of rendered objects does not work correctly in some games, meaning that objects appear in the wrong order on the screen. At least in one case this seems to be related to viewport settings and 2d <-> 3d coordinate transformation. Also there seem to be issues with general 3d object transformations, leading to models being placed incorrectly on the screen or in the wrong position.

One game uses an unknown blend mode.

Viewport settings are still incomplete and don’t handle mismatches between requested dimensions by games and actual window sizes. The original has some code here which should be ported as well.

Future plans

I plan to tackle the above issues after taking a break from the code base for a few weeks and depending on how much spare time I will have available. Complete support for all Wintermute 3D games will probably be a more long term task, although I kind of like the idea of maintaining such a project.

Fixing .X loader bugs and adding last features

With GSoC coming to an end (less than two weeks now), the time for massive bug fixing has come! Well, it’s not too bad I would say, but certainly this has been the major work of the last few days. It turned out that the .X loader was still buggy and missed certain features. I really wished this file format was a little bit more strict as this would prevent some edge cases and make implementation a lot easier. Anyways, things are much more stable now on this side.

Unfortunately this does not mean that suddenly all WME games will be available in ResidualVM. Some games still behave very buggy when it comes to object positioning, for unknown reasons. On the other hand, Alpha Polaris works, I got Mental Repair to run and there might be a few more, which will work or might require only minor tweaking. Biggest disappointment here for me is that there will be no time for getting Act of Murder 1 to work.

Additionally, some little features were implemented, some missing stuff for 2d rendering and shadow images (pregenerated shadow textures for actors), which are rarely used by games though, it seems. This will probably be it, if I don’t suddenly change my mind about implementing shadow mapping in the shader based renderer. Otherwise, it’s time for some testing and more fixing as well as code clean up.

Testing out more WME games

With the shader renderer being stable and Alpha Polaris looking fine, I was curious about some other WME games. Specifically, I took a more detailed look at Mental Repair (which is free and interestingly, a result of a Bachelor Thesis) and Act of Murder 1: FBI Confidental.

Both games turned out to be buggy, especially Act of Murder 1, mostly due to missing features or details which had been skipped so far, although it also seems, that at least spot lights are not setup correctly, too.

Initial issues in Mental Repair are fixed though and soon I will attempt a complete playthrough, just to see what happens. Act of Murder 1 will require much more work it seems, it has special viewport settings, meaning that 3d content is only rendered into a part of the window/screen and this causes all kinds of problems. Also lights seem to be incorrect here, but lighting has to be worked on anyways. Here is a screenshot:

This is with adjusted spot light settings, so that the models are really light up. I still have to figure out the actual values out. More issues are the shadow being cropped for some reason (the model itself does not suffer from this) and the pathfinding is not really working either, meaning you can just walk trough the chair for example.

Refactoring the renderer interface and getting some work done on binary .X files

So last time we were at a point were the engine port works with OpenGL ES 2 shaders. One reason to implement this was actually to find out what a proper base interface for the rendering part would look like. This interface should abstract away the differences between the fixed and programmable pipeline APIs being used and also at least allow for a potential software renderer based on TinyGL.

The next step was thus to reorganize the current renderer code in a somewhat more proper way and also move some rendering API calls into more suited spots like the renderer classes for example. Also the old OpenGL matrix stack is not used anymore, since it was kind of unnecessary and there is a clear boundary now on which matrices are going to be transposed so that they can be passed to OpenGL (the boundary being the renderer interface actually).

One more thing that has happened is the implementation of a binary .X file lexer. Originally I though that this step would be straightforward, but actually the grammar for binary .X files is slightly different to save some space. Instead of writing a new parser I decided to hack the lexer in such a way (and also introduced some new functions for it) that the parser could stay essentially the same.

Here is a screenshot of “On the track of dinosaurs” demo, which has a binary. X file:

There are some oddities with this file, for example, it has less normal vectors than actual vertices. This might explain the dark spots because no or wrong normal vectors means messed up lighting. Also the animation data is contained in an additional file and support for loading of such a file is still missing.

Finishing basic shader rendering

Now we can render our WME games even with shaders. The most work was actually related to setting up the surrounding code and not so much the actual shaders (minus the one which is responsible for rendering .X models). At the moment, things are somewhat unorganized however and the current code should be refactored, interfaces improved and common code factored out. I would claim that at the moment one can see the fact that WME was designed for the old fixed function pipeline and that the addition of the shader renderer is not completely straight forward. In contrast changing from Direct3D to OpenGL was easier, the only real change that I made was using the OpenGL matrix stack, which now turned out to perhaps been the wrong direction. Well, it can be reimplemented (and is so at the moment), but the old WME code didn’t rely on it either, so I might change this back.

I hoped that lighting done by shaders would produce the same results as the original code but this is not the case.

Let me just assure you that Rune is not supposed to look like he is almost getting sick. Apart from that though, things seem to look just fine.

Starting rendering with shaders

Ok, what’s so special about the following picture:

Well, nothing, except that the scene was rendered with OpenGL shaders instead of the old fixed function pipeline. Which in this case (2d graphics) was not the biggest change, essentially only the 2d projection matrix has to be setup for the shader. 3d graphics will require more work, though. Also some refactoring will be necessary, some of which has already happened. For example, there is a 3d renderer interface class now with two implementations, which can be selected from the ResidualVM menu (a third one, based on TinyGL, is supposed to follow at some point in the future).

For some reason, when walking into the house and also when leaving it, during the animation the cursor becomes completely black, but only with the shader renderer. Now I am not too unhappy about this, at least it shows that the two renderers indeed operate differently.

Speaking of different behavior, the shader renderer might also be helpful when going back to lighting, since currently, the fixed function renderer does not give the same results as the original Wintermute Engine. If the shader renderer can reproduce WME lighting, than something would be wrong with the fixed function settings. If, on the other hand, shader and fixed function renderer produce the same results, something will be wrong with the rest of the code, possible normal vectors. If all three implementations are different, than it’s kitchen sink, though.

Rendering shadows

Another major step is done, rendering shadow volumes of 3d objects:

The shadow looks beautiful, I have to say, stencil shadows are great. Also Rune does not appear to be floating over the ground anymore in this scene.

WME has some more shadow techniques (looks like it got real time shadow mapping, if the graphics card supports it), but they don’t seem to be too relevant, so it might be better to postpone their porting until a later point.

One thing which is bugging me is the fact that lightning does still not give the same results everywhere compared to WME. But I am going to accept this for now. Now is the time to add some more missing stuff and cleanup the existing code.

Turning the lights on .. well, almost

So, saving and loading seems to be fully functional now, including compability with ScummVM save files. The last bug was caused by a mistake during code import from the original Wintermute Engine. I had similar bugs on multiple occasions, which is rather annoying, as these mistakes should be unnecessary, but so is life I guess. Next thing on the menu are lights and shadows, to make the visual output more enjoyable, for example:

The faces in particular look much more round and plastic than before. Now what I don’t like so much is that at this spot Rune should appear a little bit darker but I am not sure what is wrong here.

This is my new favourite:

No textures, but normal vectors are being rendered, just so that I could check if they are correct. Now in this picture they are actually not correct, which might be hard to see, but if you want to, you take a closer look at the chest for example. There a vertices with the same position but different normal vectors, which means that these vertices will receive light with different intensity. This will create more distinct edges which will be desireable in certain situations, but not here. The first picture by the way does not have this issue, hence the smooth skin.

The problem was created in the first place because I decided to recalculate the normals during animation updates (I am not 100% sure anymore if this is really needed, although it probably should be). But different vertices (even if they have the same position) mean different normals, as long as you don’t specify, which vertices should get the same normals. Since this information does not seem to be contained in the .X files, instead now I just update the normals directly via the bone transforms.

Saving the game and memory leaks

So the next thing I wanted to do was getting saving and loading to work. The Wintermute Engine has a, I would say, rather sophisticated save system. Basically it is possible to save an entire object graph into a file and then getting it back later, without the user code having to do anything about it apart from saving it’s own state (actually, the engine handles things in a way such that loading and saving can be handled by the same function for the object being saved/loaded in). Essentially this is achieved by registering newly instantiated objects (via new overload) and keeping them in a special container class.

Now there are some caveats here, for example when one is dealing with a subclass of a class with such a new overload, which actually is not supposed to be registered. In this case one has to disable the whole registering process via a flag, something I had to find out via debugging and comparing with the WME code.

Another problem is compability with ScummVM savegames. The 3d code adds new classing and member variables, which are going to end up in the save file. This would lead to inconsistencies betweem save files from ScummVM and ResidualVM. The proposed solution here is to only save/load the 3d classes and variables in case we are playing a 3d game, which should work but I have not tested this yet.

Also loading a save file in Alpha Polaris seems to introduce at least one bug, certain info texts disappear too quickly and thus are not readable. Also text input (which is required at certain times in the game) is not possible anymore.

One reason I started too work on save files was actually to be able to play the whole game since after a while the performance would drop so much that I had to stop the game. Turned out that the problem here were memory leaks. Fixing them made it possible to play the whole game in one go, without any saves (which basically are broken at the moment). This is very good news as it means there is no big stuff missing which would be required to finish the game. It also showed that the leaks were indeed fixed since memory usage became stable after a while (although 2GB still seems a little bit too much to me).

3D object selection

We are now able to select 3d objects by mouse clicks (2d object selection was already part of the ScummVM Wintermute port). Take a look at the following screenshot:

The cursor indicates that it is pointing onto our main protagonist.

Essentially what happens is that we have a bounding rectangle (constructed from a bounding box) for our model, which is added to a list. Every frame the list is traversed. If the mouse coordinates land in one of the rectangles, a more precise test is performed. In case of our model this means that we send a ray into the scene and check if it hits one of the triangles of the model. If so, the model becomes the active object, which means it can be selected by a mouse click.