GSOC: Concluding Remarks

The final evaluation round for GSoC 2019 has ended, and I would like to this moment to summarize the work that I have done for the past 3 months. In case you just want to look at the code, here is the PR link.

Hyperspace Delivery Boy

My project revolved around building a game engine for the Monkeystone game Hyperspace Delivery Boy. The project description for the same can be found here.

My early work on the project feels so long ago that is quite difficult to remember the exact steps I took, and the problems I faced through them. Thankfully, some of the more difficult ones are mentioned in this blog. To summarize, my work included building core systems such as AI, LuaScript, Map, Sound and much more that mimicked the interface and functionality of the original HDB source code.

Once the core functionalities were added, I used them to link the Lua scripts to the engine and define the in-game logic that was present in the engine. This included things such as enemy movement, hit detection, dialog rendering, map-loading, the save system and much more.

I took a STUB-oriented way towards the project, upon Eugene’s suggestion. Whenever I needed some functionality that didn’t exist yet, I would create a warning for it and come back to it later. It made the process of navigating and translating a large codebase much more linear, and simple to understand.

Changes Since Last Time

In my last update, I mentioned that a few bugs remained in the PPC version, which made it unplayable. Those problems have been rooted out, and the game can be completed with the PPC data files as well. In fact, from the playtesting that I have conducted so far, I can say that the game should be completable for all available platforms – Windows, Linux and PocketPC.

What’s next?

For now, I would like to work towards getting the project properly merged into the main ScummVM repo. There is also a scaling bug in the PPC version that still exists. It is far from game-breaking (in fact, some might say it makes the game easier) but I would like to rectify it in order to stay true to the original.

Once again, I would like to thank Eugene(sev) and Arnaud(strangerke) for their help and support throughout this project. I wouldn’t have made it even half the way here without their help.

GSOC Update: Week 12

This is the last week of the GSoC program, and the PPC rendering problem has been fixed. I have created a PR for the HDB engine, and I’ll spend this week preparing Work Product Submission and fixing existing bugs.

The PR moving the Lua code to Common was also merged this week.

Current Problems

There are a few bugs left in the PPC version, and the Linux and PPC demos are crashing for one reason or another. For the Linux demo, I’m aware that the problem lies with the Sound data. When it is processing the Vorbis stream data, it crashes in the refill function. I’ll look there for the reason of the crash

Accomplishments

  1. Fixed rendering issues with the PPC version
  2. Added mouse cursor for PPC version
  3. Fixed double click for PPC
  4. Finished lua-merge PR
  5. Fixed PPC updateFade function
  6. Fixed PPC Deliveries screen

Thanks to Eugene(sev) for help with the rendering code, updating the save system for PPC among other things and thanks to Arnaud(Strangerke) for help cleaning up the codebase.

Objectives

  1. Complete level testing
  2. Ensure PPC and Linux demos are working
  3. Fix remaining bugs in PPC version

GSOC Update: Week 11

I have only a few weeks left until I have to make my Work Product Submission, and most of the larger problems have been dealt with. I have added support for the PPC version of HDB, so that should be playable.

Current Problems

There are a number of rendering bugs in the current PPC code. From pictures that are inverted, to font that is rendered as gibberish. Aside from that, I have found a few bugs in the codebase. They weren’t a problem in the Windows version, but seem to crash the game now. These issues need to be fixed before support for the PPC version can be considered final.

Another issue is that the PPC version is supposed to played with a stylus. As such, it does not have a cursor graphic. This is a problem since the game needs to be played on platforms that do not have a stylus.

Accomplishments

  1. Added support for Linux version
  2. Sectioned off Windows/Linux specific code
  3. Added PPC-specific Delivery/Inventory code
  4. Fixed PPC crashes
  5. Fixed Player Input in PPC version

Thanks to sev(Eugene) for help with the PPC crashes, the Fade-In problem and thanks to Arnaud(Strangerke) for help with simplifying the codebase.

Objectives

  1. Fix rendering issues with the PPC version
  2. Add mouse cursor for PPC version
  3. Ensure that Linux and PPC demos are working properly
  4. Complete level testing

GSOC Update: Week 10

My university has finally openned for a new term, so it has been a hective schedule. Nevertheless, I have managed to get in a decent amount of work this week.

Current Problems

The current build for the PPC version is crashing and I suspect it is crashing because of the Windows-only code that is executed by default. Hence, my immediate objective is to section off the Windows-only code. If that still doesn’t work, I suppose I’ll have to find the cause of the crash through the debugger.

A few levels remain to playtest. The problem isn’t that they’re bugging, but that I haven’t figured out yet how to beat them! A number of the puzzles near the end of the game are greatly challenging.

Accomplishments

  1. Fixed the AudioStream Bug
  2. Added support to FileMan for reading MSD files
  3. Added support for compressed game files
  4. Sectioned-off PPC-exclusive code
  5. Converted Platform-specific Constants from enums into variables

In my previous post, I had described a bug with the AudioStream code. The AudioStream was getting deallocated after playing, regardless of whether I created the stream with DisposeAfterUse::YES or DisposeAfterUse::NO. The problem was deceptively simple. The playStream() function has a DisposeAfterUse condition of its own for some reason, which is independent of the DisposeAfterUse condition of the stream that is being played. Moreover, it is set to DisposeAfterUse::YES as a default parameter, so if you’re not paying attention to those, you could end up using it by accident.

This is what I was doing wrong. The deletion was carried out through a callback (an SDL callback in my case) which made it difficult to find when debugging the Main Thread with the debugger. Setting DisposeAfterUse to NO in the playStream function fixed the problem.

Thanks to Eugene(sev) for help with the Windows Demo, adding the Platform code for PPC and improving the rendering pipeline and fade-in code.

Objectives

  1. Complete adding support for the PPC version
  2. Implement syncSoundSettings
  3. Add support for the Linux versions
  4. Ensure that the PPC and Linux demos are working

GSOC Update: Week 9

This is quite soon after the last update, but I feel this is an appropriate moment for an update. I have added the various Sound functions, including Voice, Music and Sound Effects.

I have also played through a few more levels, and fixed a Lua bugs, so the game won’t crash until MAP20 at the very least.

Current Problems

While Voice and Music are working fine, the Sound Effects code currently has a bug where the data for each sound effect gets corrupted after playing through it once. Running it through the debugger, it seems that the AudioStream holding the Sound Effect data either automatically gets deallocated or corrupted, since the system can read it no longer. I have spent quite a few hours trying to deduce where it is getting deallocated, and I can’t seem to find it.

One of the few remaining portions left is syncSoundSettings(). It is the default method via which ScummVM saves the Sound configuration for the game. I haven’t looked into it yet, which would be my next goal.

I’m not completely sure whether or not there is a problem with MAP19, but I can’t seem to beat it without cheating through the debug interface.

Accomplishments

  1. Added Voice, Music and Sound Effects code
  2. Fixed crash with Deliveries with no GFX
  3. Added additional output to the Debug Interface
  4. Removed unused code

Thanks to Eugene(sev) and Arnaud(Strangerke) for help in cleaning up the codebase, testing support for demo versions of the game and pointing out potential mistranslations in the codebase.

Objectives

  1. Fix the AudioStream bug
  2. Implement syncSoundSettings()
  3. Start adding support for different MPC files

GSOC Update: Week 8

I spent most of this week resolving memory bugs and playtesting the game levels.

The game still isn’t playable from beginning to end right now, but progress has been made. I estimate that you start from the beginning, you could definitely play up till MAP14-19 before you a bug stops you. Also, I’m not completely certain whether or not all the endings in the game are currently achievable.

Current Problems

I have managed to run Sound through the main hdb.cpp file, but the methods defined in the original are not as simple. They involve multiple structs, and functions for each of them. This is my top priority for now.

There are two kinds of sound data in the MPC file: MP3 and WAV. The MP3 files are working fine, but the WAV data in the archive is empty. It seems that all the data values have been set to 0. I’m not sure where to find those sound files, but they need to be found: all the sound effects are in the WAV format!

Finally, I have only tested the engine on the Windows Full-version. I still have to test out PPC and Linux versions, and add support for the compressed and voiceless versions of the game. This shouldn’t be too hard as long as I can find their data files. Moreover, I know that the current engine won’t work with the demo versions of the game. So I have to support those as well.

Accomplishments

  1. Fixed memory leaks
  2. Removed redundant code
  3. Added Patches to MAP10, 11, 15 and more
  4. Fixed problem with Animating Bridges
  5. Fixed moving floats
  6. Fixed aiFatFrogTongueDraw()
  7. Fixed crash caused by LIGHTBARREL in Slime
  8. Fixed indexing problem in the Teleporter code

Thanks to Eugene(sev) for help with the uninitialized reads, simplifying the save system and fixing bugs such as the Loading Screen, the Waypoint system and adding patches for different maps. Thanks to Arnaud(Strangerke) for fixing compilation issues with MSVC9 and pointing out flaws in the Teleporter code.

Objectives

  1. Complete Sound system
  2. Find missing sound data
  3. Start adding support for different MPC files

GSOC Update: Week 7

By the end of this week, almost all of the STUBs – save for the ones related to Sound – in my codebase have been resolved.

The game is getting in pretty good shape now, but it is still far from the point where you could start from the beginning and have a reasonable chance of beating the game. There are three major issues right now.

Current Problems

For starters, HDB is eating up a lot of unnecessary memory, so I’ll be patching up on all the uninitialized reads and memory leaks I have inadvertently caused.

Secondly, the game is riddled with bugs. Not the game crashing kind, but subtle bugs that ruin the experience. Examples include the enemies being unable to hit the player(though I bet certain players would call this a feature), some paths being blocked for no apparent reason, and most importantly, the first level in uncompletable right now.

Finally, I haven’t played through most of the stages and I’m willing to bet that there is still a lot of Lua 4.0 code in the Level code that needs to be patched. I don’t have enough Lua experience to tell this just from reading the files, so I’ll have to test out the game to find out wherever the Lua interpreter is crashing.

Accomplishments

  1. Implemented Menu System
  2. Implemented the in-game Debug Mode
  3. Added support for the Quit Key
  4. Augumented the Command Line Level Select to choose between Action Mode and Puzzle Mode
  5. Implemented the Fade Functions
  6. Implemented the Snow Rendering Functions
  7. Implemented the Progress Bar/Loading Screen
  8. Added the Weapons Code so Guy can now attack and stun the enemies

Objectives

  1. Fix Memory Leaks
  2. Fix gameplay bugs
  3. Add Lua patches

GSOC Update: Week 6

So, I spent most of this implementing Pushing, unstubbing Bots-NPC code and implementing the Save System. Here is a gist of the accomplishments. However, loading the savefiles directly from the command line doesn’t work right now.

Accomplishments

  1. Added all the Bots code, all the Callback Functions and most of the NPC code
  2. Added Extendable Bridges, along with the useTarget function for Use interactions
  3. Implemented Pushing
  4. Added stubbed Sound and Menu system
  5. Added Save/Load system with saveGameState and loadGameState functions

Also, sev added the code for skipping certain quests and loading Game Levels from the command line. Along with the Save system, this should make testing specific parts of the game much easier.

Save System

While the Save system was implemented through the conventional ScummVM method, it still took a fair bit of trial and error to get it working.

The biggest problem I had was how to save the different structs and function pointers. The original saved the different structs directly, without thinking about the endianess of the data member. To remedy this problem, I had to use the ScummVM read and write methods and greatly split the original code.

Since the structs were being saved directly, the pointers and function pointers were being written as well, which would have been useless upon loading the save. Hence, separate code needed to be added to cache the structs and functions those pointers pointed to.

Objectives

  1. Implement Menu System
  2. Implement the Fade Functions
  3. Implement the Snow Functions
  4. Implement the Progress Bar

GSOC Update: Week 5

Continuing the work that I started last week, I have implmeneted most of the functionality in AI and Window, and mostly completed a new sub-system: Input.

The openning cinematic is completely running, all the input controls except the Quit Key and the Pause Key have been implemented, NPCs, Bots and objects like crates have been introduced, and the player can now talk to NPCs. Here it is:

(Source: https://www.youtube.com/watch?v=CbiMT60i3eU)

While most of this week went according to plan, I ran into a few weird bugs that sev had to point out for me. Here are they:

The CineCommand Problem

Shortly after writing the previous blogpost, I ran into a bit of trouble. I noticed that Lua was loading the wrong string for the dialog text, and fixed that which directly caused the game to crash.

After a few hours going over why it was crashing(which among other things involved printing the Lua Stack), sev pointed out that I had been using a preallocated char arrays of 32 length, while trying to load messages as long as 80+ characters. This had led to a memory overwrite, and caused the crash. Changing the CineCommand struct to use a const char * fixed the crash.

Dialog Spacing

In a stupid blunder, I had been miscalulating the whitespace in the dialog messages. Instead of simply adding the designated number of pixels, I was multiplying them leading to a large amount of unnecessary whitespace.

Foreground and Background indices

If one recalls the previous blogpost, one would know I have a tendency to declare values as unsigned variables. This led to another problem recently, when I realized that certain tiles are designed to as to have a -1 index value. This type mismatch had to be corrected.

The Clock That Wouldn’t Disappear

In the openning cinematic, there is a clock that needs to vanish after it is thrown. However, it wasn’t disappearing. sev pointed out that I had been using the _cine[i]->x, _cine[i]->y positions rather than the _cineBlitList[i]->x, _cineBlitList[i]->y positions for the clock, hence its appearance.

The Copy/Paste Error

I had made a nasty copy/paste error. When making a nester for-loop, I had copy-pasted the outer-loop without changing the indices of the new loop. This didn’t show any problems at first, but would definitely cause trouble later.

Changing indice mid-loop

One of the slightly-inconvenient problems I have started to face involves altering an array while iterating over it. Doing so while using an iterator has caused to be trouble some, since it would lead to one or the other pointer error. The best approach seems to be to rewrite those loops without the use of iterators.

Objectives

  1. Add Sound System
  2. Add the remaining Bots and NPC code
  3. Implement Pushing
  4. Find and patch Lua problems

 

GSOC Update: Week 4

It’s been a long week, and I had to implement two whole sub-systems during this time: AI and Window. Both of them are currently a work in progress, but they have produced tangible results on screen. Building out these two systems, I have faced a fair share of problems which will be discussed below. This will be a fairly long read.

Map

In my last update that was two weeks ago, I had partially drawn the first level. I’m glad to say that now I can draw the complete, as one can see below.

(Editor’s notice: Unfortunately, the image/video that was previously integrated here is lost and not archived)

As you can see, I can now:

  1. Completely Background, Foreground and Grating tiles
  2. Scroll the map
  3. Initiate Dialog messages

AI

By far, the AI subsystem has been the biggest subsystem I have encountered so far. With multiple Struct types, constant data and member functions, it defines not only how the characters in the game behave, but also Animation and Drawing code for all of them.

In a nutshell, this is how the AI subsystem works: Each object in the game that is more than just a drawn image is represented by the AIEntity struct. The definition for this struct can be seen below:

As you can see, it contains Enums for defining Entity Type, State, Direction, Entity position, velocity as well as range of function pointers for Initialization, Drawing and possible Actions. A major portion of the AI subsystem is dedicated towards manipulating Entities of the type AIEntity.

The AI subsystem also maintains a series of lists for processing different events such as Actions, AutoActions etc.

Cinematic System

While it is a part of the AI subsystem, I believe it deserves a separate mention. Basically, the Cinematic System is designed to run scripted interactions between Entities – the HDB equivalent cutscenes in a way.

Here’s how it works: The Cinematic System is a giant state machine that manages a queue of CineCommand values. Each value is equivalent to a call made in the current Lua file. It tells us which interaction is to be executed along with details of the interaction if any.

Once the Lua script call is made, it triggers a function that inserts the corresponding CineCommand value into the queue. While the Cinematic System is active, at each frame, it will go through each command in the queue and process them. If a command is completed, it will discarded from queue. If not, it just moves on to the next one.

The WaitUntilDone command is implemented using this mechanism. If there are any commands before it, it fails to register as completed. Once there are no others before, it registers as complete and the System moves onto the next one if one is there.

Window

The Window class draws all windows, dialogs and messages in the game. I had to extend the draw-manager with functions like loadFont and drawTextto render text on screen. Along the way, I also added getter-setter functions for the Cursor, Kerning and Leading of the font.

Currently, this class is being used to draw Dialog messages as you can see above. If you look closely, you may see that there is a artifact problem with Dialogs which will be described in depth below.

In the future, the Deliveries List, the Inventory and Dialog Choice boxes will be added.

Problems

Here is an account of the few problems I’ve had over the past week.

Blitting and Clipping

For the purpose of Blitting Pictures and Tiles, I have introduced a Global Surface. It keeps a track of the of the entire screen and calls the g_system->copyRectToScreen function which keeps a track of the dirty rects and blits only them.

At one point, the Surfaces started to throw out assertions since the Suface being blitting overstepped the bounds of the Global Surface.

Initializing Structs

I ran into a series of Read Access Violations due to reading data into initialized struct pointers. As a result, we have developed a rule to add Constructors to all structs that I define.

Struct Arrays instead of Pointer Arrays

In the beginning, I was keeping a number of struct arrays either in the form of T *arr[size], or using Common::Array<> with the general understanding that it would be easier to reference the structs by pointers rather the value. However, since then I have realized that they merely serve to hold data and not modify them. Moreover, there are a number of them and initializing each and every one of them was getting hectic. So, I have

Cinematic System and Blocking

As you can see above, the Cinematic System is getting blocked while executing the C_MOVEENTITY command for Sarge in the opening cinematic. Instead of stopping when she has reached her goal, she keeps on moving forward and never stops, just blocking the Cinematic system. Not sure why this is happening yet, but it needs to be solved.

The Dissolving Screen Problem

In the previous update, it was mentioned that we had implmeneted a hack to get the scene drawing without calling startMap. While it is still not implemented, the centerXY and drawSky functions are now being called directly through the main game loop for now, so the XY-cructch is no longer needed.

However, while implementing drawSky, a different problem appeared. When it was called, it would creating a strange dissolving effect on screen, wherein the pixels on the scren would drive down with the drawn stars. While bizarre at first, it was being caused because the screen was not being cleared each frame for drawSky. An obvious mistake perhaps, but something that stumped me nonetheless.

Void Pointers

I’ve started to notice a pattern in the original HDB sources. For the purpose of moving data from files to object, they relied heavily on typecasting. They would load and store data in void* pointers, which would be casted at the time of usage to a particular struct. This might not seem like a big deal at first, but two problems arise:

  1. Portablity: Due to different architectures, we cannot blindly casted void pointers and expect no problems. We have to explicity define struct types and load them through the ScummVM API.
  2. Void Pointers can hold anything: Since void pointers can hold references of any type, at many places they have been used to hold values whose type could only be determined at runtime. Worse were situations where functions were returning void pointers, whose type could only be determined at runtime.

    Due to the static nature of C++, converting such code to explicit types required me to split such functions into multiple variants and introduce union types in their structs.

Font Header Problem

At its core, this was another type problem, except perpretrated by myself. Most of the structs in the game hold positive data, so I have developed a tendency to use unsigned types for them.

This caused a problem here since I wasn’t expecting a negative number in the _fontHeader data. Moreover, I had been reading the data 16 bits at a time, whereas I should have employed 32 bits. This led to wildly inaccurate header information due to which I couldn’t render the messages.

Dialog artifact problems

If you look closely at the above image of the Dialog box, you might see it has dark black artifacts near the top line. Here’s how those happened.

The Dialog works is composed of 9 smaller images: TopLeft, TopMiddle, TopRight, Left, Middle, Right, BottomLeft, BottomMiddle and BottomRight.

Depending on the length of the message, the Middle images are repeated as set number of times. When I implemented the drawDialog function, there were transparency artifacts as well. However, replacing some of the draw calls with drawMasked solved that.

Objectives

  1. Debug and fix the moving code for Entities so Sarge stops walking when she reaches her goal.
  2. Figure out the correct file for the Dialog image.
  3. Implement the Bots code