Alikhan's GSoC Blog

Project description

The goal for the project was to finish the work on qdEngine and make all games based on it playable and completable. Milestones for the project were:

1. Multiple MinigameManagers could be instantiated. Mini-games for dogncat are running.
2. Games with advanced mini-games are supported.
3. All games on qdEngine are supported.

Because I continued working on the engine since my first contribution at the start of March, I was able to finish the task in the first week on the GSoC period.

Thus, for the GSoC period itself, my task was to finish the work on existing engines.

What was done

The original goal for qdEngine was met and I was able to reach all milestones – all games are completable to the end. The news item with the engine announcement can be found here.

The second engine after qdEngine was SLUDGE. The engine was already very close to be done, so only I added a few minor enhancements:

• Support for extended saves
• Support for Return-to-Launcher and custom quit dialog
• Fixed some crashes during autosaves

Following this, I worked on adding language support for the Teenagent game. At that time it only supported English strings, and my task was add support for Polish, Czech and Russian (fan-translated) strings. The challenge was that all language-related data was stored inside the executables, so I had to first extract them, and then modify create_teenagent tool, so that it generated the .dat file that contained the data for all languages in it. This task took quite some time, but in the end, I was able to extract all strings, pack them into dat file and make changes to the engine to support everything.

Shortly after this, I also added support for Polish voice-overs, the only version with voice acting.

After Teenagent, I worked on WAGE engine to support games made with World Builder game creation system. This included both fixing the bugs and also redumping the games, so that they are in right format (Macbinary). The latter took some time and iterations, however with the help of my mentor, we were able to produce the master collection and announce engine for the public testing.

After this I worked on playtesting and improving the MacVenture engine. I was able to fix a couple of blocking bugs, implement number of missing features and playtest games to the end.

Finishing the work on engine also includes taking screenshots, writing wiki pages and preparing news items. So in addition to writing code, I also helped with screenshots and wrote three news items for WAGE, SLUDGE and qdEngine.

Code merged

There were a number of pull requests created over whole period of GSoC. Below are some of the important ones:

qdEngine

Adding support for advanced minigames

Adding 32bpp support

Adding punycode support

SLUDGE

Add support for extended saves

WAGE

Various bug fixes

Adding fallback detection

More fixes

Teenagent

Adding language (textual) support for Polish, Czech and Russian

Adding Polish voice-overs

MacVenture

Bug fixes and improvements

What’s left to do

Finishing last bits of MacVenture:

• Display Title and StartupScreen images on startup.
• Extend MacDialog to support text input and use it everywhere in the engine instead of existing dialog class. This will make pop-up dialogs look more similar to original.
• Sync Click-To-Continue button’s behavior with original. In other words, enable click-to-continue button when text inside console window needs to scrolled to be fully read to the end.

Conclusion

I really enjoyed my time spend with ScummVM this summer. It was my first time contributing to open source project and doing it with ScummVM was a great experience. I am pleased that I had a chance to work on a real world application used by players all around the world.

Working on this project helped me to improve my skills in navigating enormous codebases, reading and understanding other people’s code and correctly formulating and asking questions when get stuck.

Lastly, I want to thank project organizers at ScummVM team for selecting me and giving me a chance to participate in this program. Of course, I also want to say huge thanks to my mentor – Eugene Sandulenko, aka, sev, who guided and helped through this entire period. He was always there for help when I got stack and provided deep explanations when I did not know certain things. I also would like to thank other developers who reviewed my code, provided suggestions and pointed to mistakes. Finally, I am grateful to GSoC organizers for providing this incredible opportunity.

 

Last week was mainly spent by fixing left-over bugs from previously worked engines: WAGE, MacVenture and SLUDGE.

At the beginning of the week, me and Sev finished work on redumping and preparing ultimate archive of WAGE-based games. Finally, after multiple checks the we were able to prepare the collection and release it on the site, together with the engine announcement. However, a couple of issues was reported by the devs, so I returned back to the engine to fix them. In particular, I had to fix issues with closing the game, where confirmations pop-ups were not really working, i.e. the games were not closing. Also, for some games, closing them in midst of playing sounds resulted in crash, because the thread for audio was referencing the deleted engine object. Lastly, there were crashes due to null pointer accesses in scripts, which are could be either due to buggy scripts or due to engine issues. There is a possibility for the last one, since the game that was crashing (Mormonoids from the Deep) also throws exception in the Java codebase, which was ported to ScummVM.

Another engine to which I returned was SLUDGE. There was a PR by ccawley2011 to add the game (demo) called Otto Experiment. However, the problem was that it was crashing at the startup. The problem as it turned out was that the code was not looking for the file inside the game directory, only in the save game dir. Also, in general, the support for reading .ini files was missing. This was mainly used for language setting, some graphical preferences and so on. This feature appeared to be rare in general, however it was used in Otto Experiment and Cubert Badbone, so it must have been dealt with.

Last but not least, I did some work on on MacVenture. There was a few a differences with the Mac versions that needed be done: main game window names (which were not upper-cased) and console window.

In the previous I continued to work printing the task. I’ve added the initial options to printing such as layout (portrait, landscape), margins as well as the selecting the printer itself. In the process of adding these, I’ve customed myself a bit to Win32 printing API, since this is the only backend with which I was testing the actual printout.

First of all, I continued the work from the last week by introducing PrintingDialog, which will be opened when printout happens. To it, I’ve added layout and printer selection. In Win32 Print Spooler API, printer list is a string array, so for now, in the PrintingManager class, this virtual method is also Common::StringArray. Implement it was quite easy, since the documentation is there and not much code was needed. By adding printer selection, users can select the real physical printer or virtual one to get PDF-output. Since I have the actual printer, I started testing the code and see what was the output:

As you can see, while the logo and were printed, they were too small and not centered. This made me to spend some time actually looking how different platforms handle image printing. After comparing printer dialogs in different browsers, operating systems, I concluded I definitely needed to add scaling (or, rather, fitting to paper) and margins. To not waste more paper, I’ve been testing with PDF output since then, and was able to get the printing centered and fit to paper.

After that I started to think about preview image functionality, which is common these days in printer dialogs. Luckily, there is already an ImageAlbumDialog present, which had more enhanced image output as well as image preview. That meant I only needed to migrate all dialog code there. I’ve managed to make printing work when the right command is send, however, implementing GUI nice looking correctly (so that both preview appears on the left and printer settings on the right) appeared to be a little harder, and is not yet finished. This week I plan to fix this, and as was commented and suggested earlier start adding printing support to actual engines and see if the current API functionality is enough or needs to be adapted. Before that I plan to quickly fix a few things (mostly visual differences) in MacVenture engine, so that it looks and plays like in the original Macintosh.

 

In the previous week I finished with fixing leftover bugs in MacVenture and later started working on bringing printing support.

The last problem that was left from the week before that was polishing console window. In particular, there were problems with how the text was displayed when new text was added or when the window was resized. To make things easier I decided to change that console window from Graphics::MacWindow to Graphics::MacTextWindow. That allowed to take care of text wrapping and scroll management at once. However, to take it fully work, a few adjustments have to be made, as the class probably was initially created for the console with input (editable text). For example, to make the cursor be focused on the right place required to a little workaround, where I made the window editable, append the text, and make it uneditable again. Despite this, after it was implemented, I also added support for saving the text, so the save/loading functionality is also complete now.

After this with Sev’s suggestion, I started working on bringing printing support. It would useful for the MacVenture titles as all four games give a chance to print the diploma after completing the game. In addition it could be added to numerous HE games and others. I did not start from scratch, but rather from the previous attempt from few years back. I took the first few commits, and the later ones with some fixes. Then I added a simple test in testbed, so that I could test how the resulting pdf file. Once that was tested, I started adding code on top of it. I added a dialog for printing and along with it a feature to save the input surface as an image.

This week I plan continue the work on this1 and start actually implementing printing feature for the engines that needs it.

Another week spent on improving and finishing the MacVenture engine. In the last week I focused on improving things that happen after the winning the game, i.e, loading diploma window & dialog, implementing “Clean/Mess up” functions. I also fixed the bitmap problem I had with with a few items in Uninvited game. Lastly, I spent time fixing the resize button and porting the leftover methods from the original codebase.

I started the week with adding support for the diploma window. There was already a pointer variable in the codebase, but until before I did not know what kind of diploma it was referring to. The diploma we are talking about here is the certificate you get after completing the games.

To implement it, a first added another constructor for the ImageAsset class, as the format for the diploma image was a little different from the rest of images. Once that was done, the diploma image already started showing:

After that, I needed to add the dialog that had with “Print” and “Exit” buttons. There was already a Dialog class for that, however only with prebuilt options. Similarly, I added the constructor for this, however, the text was a little off. To mitigate the issue, I added an additional MacText text so that line wrapping can be easily handled. Once that was added the dialog at the bottom started showing correctly:

After that I started fixing the bug where certain items (charm and statue) in Uninvited game were incorrectly displayed:

In fact, you can see the issue in one of the images in the previous post. For debugging, I first identified the object id of the item, then looked how the image was loaded. The problem was in ImageAsset::decodePPIC0() method, in particular, in the section where 32 bits were read and then bitshifted to the right, until most significant byte was taken. At first I thought the problem was that BitStream32BEMSB which was used here was behaving a little different from the GFile class in the original, and because of that these artifacts were happening. And indeed, in the original codebase, the position of the stream did not advance right away after getBits() was called, but was incrementing the internal variable and advanced the position only after that variable was greater than 0xa. I thought maybe I should wrap BitStream template inside the custom class to imitate the behavior of the original code. However, after thinking a little more and checking the walkHuff() method, where similar thing was happening, I realized than none of that actually needed. BitStream32BEMSB was advancing stream position correctly in the first place, so the only thing which was needed was to read the first 16 bits, and skip the other half. And indeed, this two line change fixed the issue:

Another thing I copied from the original javascript codebase was “Clean/Mess up” functions. Clean up is handy when there a lot items in the inventory, and you want the thing sorted. This was a simply copy-n-paste, however, the end result is quite satisfying and as said, useful for players:

At the end of week, I started polishing the console window The window where text (log) is displayed. There were small issues fixed away: numbers not showing and resize button not working. The last one was similar to the close button covered in this post. The last thing which needs to be properly done with console window is getting rid of unnecessary new lines, making the scrolling properly work and showing the text correctly when resizing (it gets clipped, not wrapped).

This week I continued to work on MacVenture engine. I managed to fix a few blocking bugs and complete to the end Déjà Vu, Uninvited and Shadowgate. I also improved text input dialogs, made exit buttons clickable (as a shortcut to “Go” and “Open” commands), fixed the bug where titles are not displayed at saves. There are still a few things that needs to be implemented: diploma window, clean/mess up functionality and to fix how certain item bitmaps are displayed.

The first Déjà Vu went fairly well, and at the start of the week I managed to get through the end of it. After that, I went with playtesting Uninvited, and this is were I encountered the first problem. To progress further in the game, I needed to go through the door, but two deadly dogs were in front of it, blocking the path.

To pass them, you need use the “Speak” action to say the magic words and thus use the spell. The magic words are: “Instantum Illuminaris Abraxas”. I typed the words, however, that did not help and I died:

I tried over and over, with lowercase variants, checked if I needed some other item in inventory, however, everything was right according to walkthrough guides. This is when I started checking the scripts, as clearly, something wasn’t triggering. Luckily, there are only three opcodes involving strings: ScriptEngine::opadEQS, ScriptEngine::opaeCONT and ScriptEngine::opafCONTW. The one responsible for the situation is opafCONTW, which checks for the containing word. So I looked up, I discovered it was checking for the lowercase variant of the magic word: “instantum illuminaris abraxas”. So, fortunately, this turned out to be one line fix. Interestingly, when I debugged and printed out the strings from that method, as I discovered the game (apparently all games made with the engine) was containing a very huge variety of curse words. I won’t write them down here, but it has every word you can think of 🤪.

At the same time, I changed the way input text dialog worked. Before the changes, it was essentially an ordinary dialog, with no cursor of the text, and contained the same hardcoded title for all dialogs – “What would you like to say?”. This was also easy to fix, as the I essentially copied the code for cursor from MacTextWindow, and I added a line to fetch the text from the object, not from the hardcoded global table. The end result looks like this:

Another feature I finished was pressing on the exit buttons to go between the rooms. This is optional feature, as you could use “Open” command to open the doors and “Go” to actually go through the door. However, when playing the games this quality of life feature really comes handy, and was worthwhile implementing. Especially, when the code was all there, and I just had to copy it from Webventure repository.

After finishing Uninvited to the end, I went with playtesting Shadowgate. This game took a little longer, as there is a slightly annoying game mechanic with disposable torches, which are needed to be managed. There was one little blocking element, when reaching the room with the sphynx. Essentially, you have to solve the riddle of the sphynx by giving him the right item. However, when I operated the item on it, the game kept crashing. After looking for the cause a bit, I figured that the reason was dead simple: the engine had two versions of focusObjWin(ObjID objID) function. The safe version focusObjectWindow(ObjID objID) that checked if objID really referred to existing window, and unsafe that did not have this check. And of the course, when the code related to sphynx was executed, the unsafe version was run. The fix was just get rid of the unsafe version.

Lastly, I made sure the main game window and inventory windows kept their titles preserved when loading a save file. This took a bit of rewiring and adding for methods, as the code for managing the windows we have now is a little different from the original sources. In the original, main game window and all inventory windows where closing and opening at the start of the game, winning, losing, and save loading. I added back things for the inventory window, however kept the main game window unchanged for the most part. I make a workaround fix to set it’s objID to zero, so that when script executes, it updates the window info. The reason was that a decent portion of code needed to altered to adhere to the change, but since that one line fixes the issue I decided to leave it as is.

Last I week I started working on the MacVenture engine. The engine is based on the javascript implementation. It is very close to completion, what’s left is to fix a few leftover bugs and playtest the games, so that they are completable. I started with the first Déjà Vu game and close button was not working, drag (lasso) selection is missing, and in certain places the game window was not displaying the correct images. Throughout the week I tackled these problems.

As said above, first thing I noticed is that the close button was not responding, e.g. after clicking on it, nothing was happening. It took quite some time to figure out what was going wrong, but eventually I found the reason for it. The problem was that the code for identifying whether the mouse was inside close button was looking like this:

bool MacWindow::isInCloseButton(int x, int y) const {
    int bLeft = kBorderWidth;
    int bTop = kBorderWidth;
    if (_macBorder.hasOffsets()) {
       bLeft = _macBorder.getOffset().left;
       bTop = _macBorder.getOffset().top;
    }
    return (x >= _innerDims.left - bLeft && x < _innerDims.left && y >= _innerDims.top - bTop && y < _innerDims.top);
}

As can be seen, it returns true if mouse position is at the corners of window borders. And it works correctly if close button is indeed at one of the corners, as in WAGE games:

However, here the close button has a little offset to right and bottom from the left top position of border:

So to make it work, I added offset fields for the closeButton, and made  MacWindow::isInCloseButton() method to look at those offsets, if they are present.

Next problem was absence of drag (or lasso) selection. It is needed to select multiple items in the inventory at once, as can be seen here:

However, making it works wasn’t that easy and took me a couple of tries and rewrites, and to the moment of writing, there is a slight bug that needs to be fixed. But the idea is simple: on the mouse button down event save the pivot point of the selection, on the mouse move event update the the second point, and lastly, on mouse up event create a rectangle from the two points, look if any inventory objects are inside and mark them as selected. The difficulties was there because of the code flow structure which was altered and become noticebly different from the original code.

What really helped me here and in the later problem is that I managed to get original code running and made it debuggable in browser. I updated the jQuery version the fresh one, fixed a few type errors and imported the main game code as module, so that when it is compiled with closure compiler, the separate mac.js is generated.

With this, I stepped through the js code, looked how the lasso was done, and made it essentially working:

 

Afterwards I worked on fixing the issue right at the start of the game. What happened is that, when examining the mirror object, the engine would not pause and quickly skip the image of character. It happened, because the engine was not pausing after MacVentureEngine::clickToContinue() function was called. I added an additional flag that is set when clickToContinue is true, so that the game pauses, and the main game window shows the correct images. This is how it worked before and after the fix:

There are two more bugs left from the last week which I am fixing now. First one related to window resize – when making the window smaller, it stops showing borders:

The second also probably related to it. Clicking at the resize button selects the object if that object happens to be there.

For this week, I plan to continue the work on the MacVenture and fix the above mentioned bugs. Afterwards, I will solely focus on playtesting.

Lastly, I want to mention that I succesfully passed the Midterm evaluation for GSoC. I want to thank my mentor – Eugene. He has been very supportive and always there for help and explanations. Also, the tasks and engines were all diverse and very interesting to work on. I am very enjoying contributing to ScummVM.

Last week I continued to work on TeenAgent, in particular implementing the support for voiceovers. I first wasn’t sure about that since I could not find anything related to voicing in the sources we had. However, because the engine can load the voices and play them, I decided to map the voice indices to each text and play the voices in the same places where Text-To-Speech’s sayText() are called. Apart from this, there was a bug report for the qdEngine game, so I quickly returned to it to add support for punycode.

The idea for implementing the voiceovers is as I said to map the voice indices (1..2043) to each text. I played with the sounds to see how the voice resources are located and figure out the ranges. For the most part, it was sequential. The ranges are as follows:

• 1 – 333 Messages
• 334 – 566 Scene objects descriptions
• 567 – 591 Combination messages
• 592 – 683 Item descriptions
• 902 – 2040 Dialogs

Interestingly the range [684 – 901] between Items and Dialogs is empty. Implementing the first four items were relatively easy. For the display message methods I just added an additional argument for voice ids, for others I extracted that from the newly added method – getVoiceIndex(). However, the dialogs were hardest to get right. At the moment of writing, I am still working on it, most of them are working, but still some parts are wrong. The main problem I discovered is that the voice resources for dialogs are not really ordered, not every dialog line is voiced and some voice resources are empty. For the empty voices, I discovered that checking the size, i.e, if its too small is enough. That is, all of those voice resources are 4 bytes in size and I got the list of them and implemented isVoiceIndexEmpty() helper method. But the fact that some voices are not ordered, really became an issue and I had to manually find, listen and check every problematic dialog/lines. Because of this, the method for computing voice indices became a mess, and I had to rewrite it over and over.

Despite the difficulties presented above, I am close to finishing it, and once the dialogs are done, all text in the game will have a voiceover. Good thing is that I have every line of text extracted from the previous weeks, so even if I don’t know Polish, I still can with relatively no problem find the the exact text from the voices I hear.

For this week, I first plan to finish this task (today) and then move to working on the next engine – MacVenture.

This week I continued to work on adding Russian, Polish and Czech strings. Last week was spent with extracting those strings from the executables, and this time I worked on making the engine load them. The task was a bit challenging and it took a lot of trials and errors the get certain things right. Let’s break down everything I did.

First of all, I had to think how I would store and then load the language specific data. Initially I thought maybe I could fit the data into in their respective locations in the original .dat file, but when looking at Polish and Czech executables, which are much larger, I realized that would not work. So I ended with adding all the strings at the end of original .dat file. In the engine, I added Segment object for each of the items, and made the engine to read from them instead of the data segment.

I started with loading and displaying Credits and Item names, as these two were the easiest. For the loading part, I introduced two Segment objects containing these two resources and made them read the data from .dat file:

I also added Resources::precomputeCreditsOffets() and Resources::precomputeItemOffset() to precompute offsets after the loading is done. This is identical to how dialog offsets are computed and is needed so that the engine can get to correct location where credit/item data start when it asks for a particular credit/item number.

Next, I added message strings. Adding them was also quite easy, and took the same approach as the one above. The only thing that needed to be carefully taken care of was combine error message. In the .exe they don’t come together with the rest of the messages. In fact this message is the last part of the combining table section. However, for the consistency, I placed this message as the last item in messages segment.

Speaking of combination messages, I added them after this. Each combination consists of the following members:

struct {

   byte _obj1Id;

   byte _obj2Id;

   byte _newObjId;

   Common::String _combinationString;

}

I split this structure to two: one containing the first three bytes which are the same for all languages, and the one containing just the combination strings.

After that, I worked on dialogs. Initially I thought it would be the same as with other strings, but at the time I forgot about dialog stacks. Because of this, the dialogs for English and Russian versions were fine, but some dialogs in Polish and Czech versions were completely off the context. Turned out, that some dialogs should be popped from dialog stacks and shown only when certain events are triggered. They were working for English and Russian versions, as the code for reading the dialog stack data was still reading it from data segment, not from the segment that was dedicated to it. After realizing this, I added stack dialog stacks as a resource to .dat, added code for loading it and correct dialogs started popping out.

Lastly, I worked on adding scene object names and descriptions. This part was the trickiest. First of all, I realized during extraction, I made an error, and didn’t not consider cases with items with default description. That is, after the null byte, these items contain 01 byte, which indicated that the objects will be given the default description name – “Cool.” (English), “Miodzio.” (Polish), “Bezva.” (Czech), “Вещь.” (Russian). Once I realized this, I fixed it here.

Another important thing to point out is that this kind structure is a part of savefile, and the objects and their members (including names) are modified in runtime. The last part about the names is crucial, as certain objects, namely four – “girl”, “robot”, “boy”, “bowl” are modified changed to their “real” names – “Anne”, “Mike”,  “Sonny or whatever”, “body”. I implemented the support for this, but forgot crucial thing – I didn’t set enough space for the initial names, so that when they are changed, it does not overwrite description part. Because of this, I was getting constant crashes when trying to load the older saves (before changes in the pr). After finding about this, I was able to fix the issue and thus, preserve compatibility with old saves.

Last week I started working on creating multilanguage dat file for the Teen Agent. Currently, only the English version is supported, and Polish, Czech and Russian versions needs to be added.

The task is to extract language specific data, strings from the executables, put them in the create_teenagent utility and finally generate the .dat file for the use of engine. The extraction tools for the game were already in hand, although I had to write custom python scripts to create cpp code from them. The strings that needed to be extracted are the following:

1. Names and descriptions of inventory items
2. Dialogs
3. Messages
4. Character’s thoughts
5. Credits
6. Names and descriptions of scene objects

Most of them were not difficult extract and put in the code, except for the scene objects. I started with the Russian version, and the problem was that some string could not fit in the original positions, so they had to be copied to the free space of the data segment. That includes the pointers to those objects. So this caused a little issue, and I am in process of writing it properly.

In parallel, I was also looking at the Polish (CD) version. This version brought more issues, which I am yet to solve. The problem is that Polish executable is larger than the English, and has two more scene objects. The sizes of texts are also generally bigger, so I would not be able to fit them into the dat file. I am thinking of putting them after the data segment and dialogs. However, that would require also changing in a couple of places in the engine code, since it relies on the data segment. This means that save system also needs to be adapted for this, as the engine is currently writing the save section part of the data segment. I’ve already tried this idea and separated the credits section the data segment, and put it separately at the end. This allowed to display the Polish text in the credits section:

Similarly, I’ve experimented with the dialogs, and they also started showing up properly:

However, there is also a Dialog stack, which is modified at runtime by the game, so I probably will need to put as a resource in different segment as well.