Month: June 2019

It’s been a long week, and I had to implement two whole sub-systems during this time: AI and Window. Both of them are currently a work in progress, but they have produced tangible results on screen. Building out these two systems, I have faced a fair share of problems which will be discussed below. This will be a fairly long read.

Map

In my last update that was two weeks ago, I had partially drawn the first level. I’m glad to say that now I can draw the complete, as one can see below.

(Editor’s notice: Unfortunately, the image/video that was previously integrated here is lost and not archived)

As you can see, I can now:

1. Completely Background, Foreground and Grating tiles
2. Scroll the map
3. Initiate Dialog messages

AI

By far, the AI subsystem has been the biggest subsystem I have encountered so far. With multiple Struct types, constant data and member functions, it defines not only how the characters in the game behave, but also Animation and Drawing code for all of them.

In a nutshell, this is how the AI subsystem works: Each object in the game that is more than just a drawn image is represented by the AIEntity struct. The definition for this struct can be seen below:

As you can see, it contains Enums for defining Entity Type, State, Direction, Entity position, velocity as well as range of function pointers for Initialization, Drawing and possible Actions. A major portion of the AI subsystem is dedicated towards manipulating Entities of the type AIEntity.

The AI subsystem also maintains a series of lists for processing different events such as Actions, AutoActions etc.

Cinematic System

While it is a part of the AI subsystem, I believe it deserves a separate mention. Basically, the Cinematic System is designed to run scripted interactions between Entities – the HDB equivalent cutscenes in a way.

Here’s how it works: The Cinematic System is a giant state machine that manages a queue of CineCommand values. Each value is equivalent to a call made in the current Lua file. It tells us which interaction is to be executed along with details of the interaction if any.

Once the Lua script call is made, it triggers a function that inserts the corresponding CineCommand value into the queue. While the Cinematic System is active, at each frame, it will go through each command in the queue and process them. If a command is completed, it will discarded from queue. If not, it just moves on to the next one.

The WaitUntilDone command is implemented using this mechanism. If there are any commands before it, it fails to register as completed. Once there are no others before, it registers as complete and the System moves onto the next one if one is there.

Window

The Window class draws all windows, dialogs and messages in the game. I had to extend the draw-manager with functions like loadFont and drawTextto render text on screen. Along the way, I also added getter-setter functions for the Cursor, Kerning and Leading of the font.

Currently, this class is being used to draw Dialog messages as you can see above. If you look closely, you may see that there is a artifact problem with Dialogs which will be described in depth below.

In the future, the Deliveries List, the Inventory and Dialog Choice boxes will be added.

Problems

Here is an account of the few problems I’ve had over the past week.

Blitting and Clipping

For the purpose of Blitting Pictures and Tiles, I have introduced a Global Surface. It keeps a track of the of the entire screen and calls the g_system->copyRectToScreen function which keeps a track of the dirty rects and blits only them.

At one point, the Surfaces started to throw out assertions since the Suface being blitting overstepped the bounds of the Global Surface.

Initializing Structs

I ran into a series of Read Access Violations due to reading data into initialized struct pointers. As a result, we have developed a rule to add Constructors to all structs that I define.

Struct Arrays instead of Pointer Arrays

In the beginning, I was keeping a number of struct arrays either in the form of T *arr[size], or using Common::Array<> with the general understanding that it would be easier to reference the structs by pointers rather the value. However, since then I have realized that they merely serve to hold data and not modify them. Moreover, there are a number of them and initializing each and every one of them was getting hectic. So, I have

Cinematic System and Blocking

As you can see above, the Cinematic System is getting blocked while executing the C_MOVEENTITY command for Sarge in the opening cinematic. Instead of stopping when she has reached her goal, she keeps on moving forward and never stops, just blocking the Cinematic system. Not sure why this is happening yet, but it needs to be solved.

The Dissolving Screen Problem

In the previous update, it was mentioned that we had implmeneted a hack to get the scene drawing without calling startMap. While it is still not implemented, the centerXY and drawSky functions are now being called directly through the main game loop for now, so the XY-cructch is no longer needed.

However, while implementing drawSky, a different problem appeared. When it was called, it would creating a strange dissolving effect on screen, wherein the pixels on the scren would drive down with the drawn stars. While bizarre at first, it was being caused because the screen was not being cleared each frame for drawSky. An obvious mistake perhaps, but something that stumped me nonetheless.

Void Pointers

I’ve started to notice a pattern in the original HDB sources. For the purpose of moving data from files to object, they relied heavily on typecasting. They would load and store data in void* pointers, which would be casted at the time of usage to a particular struct. This might not seem like a big deal at first, but two problems arise:

1. Portablity: Due to different architectures, we cannot blindly casted void pointers and expect no problems. We have to explicity define struct types and load them through the ScummVM API.
2. Void Pointers can hold anything: Since void pointers can hold references of any type, at many places they have been used to hold values whose type could only be determined at runtime. Worse were situations where functions were returning void pointers, whose type could only be determined at runtime.

   Due to the static nature of C++, converting such code to explicit types required me to split such functions into multiple variants and introduce union types in their structs.

Font Header Problem

At its core, this was another type problem, except perpretrated by myself. Most of the structs in the game hold positive data, so I have developed a tendency to use unsigned types for them.

This caused a problem here since I wasn’t expecting a negative number in the _fontHeader data. Moreover, I had been reading the data 16 bits at a time, whereas I should have employed 32 bits. This led to wildly inaccurate header information due to which I couldn’t render the messages.

Dialog artifact problems

If you look closely at the above image of the Dialog box, you might see it has dark black artifacts near the top line. Here’s how those happened.

The Dialog works is composed of 9 smaller images: TopLeft, TopMiddle, TopRight, Left, Middle, Right, BottomLeft, BottomMiddle and BottomRight.

Depending on the length of the message, the Middle images are repeated as set number of times. When I implemented the drawDialog function, there were transparency artifacts as well. However, replacing some of the draw calls with drawMasked solved that.

Objectives

1. Debug and fix the moving code for Entities so Sarge stops walking when she reaches her goal.
2. Figure out the correct file for the Dialog image.
3. Implement the Bots code

Since I won’t be available during Week 3 and I have gotten somewhere over the past two days, this seems like an appropriate moment for an update.

I dedicated the past few days with drawing the first level. Naturally, this led to the map-loader class. What I didn’t expect at first was that my draw-manager was woefully incomplete. Hence, it took some back and forth between the two to finally get the first level going.

First Level

The good things first, here’s the partial image of the first level.

It looks blocky and incomplete since more than half of the drawing functionalities aren’t finished yet. In fact, it’s only drawing a portion of foreground tiles at this moment. At this point, I can easily blit Tiles and Pictures to the screen but haven’t got started with Masked Blitting, and Alpha-blended Blitting.

How I Got Here

To put it simply, HDB draws in a number of different layers, ranging from 2 to 4 depending on the level. Each level consists of a Background and a Foreground. Certain levels have an additional layer below the Background, known as the Sky Layer. The Sky tiles are clubbed together with the Background tiles in memory but are drawn separately. Moreover, in certain places, the foreground tiles have gratings placed over them, so those have to be separately rendered as Masked Tiles.

When the draw-manager is initiated, it creates a reference table for all the tiles in the game. Note, no MSM files or tile data is loaded at this point. Alongside the init function, draw-manager provides a set of helper functions for drawing tiles, skies, stars, etc.

Broadly speaking, the map-loader utilizes three different functions to draw the map: loadTiles, load, and draw.

1. load: It takes the MSM file wrapped in a SeekableReadStream as input, and loads the different portions of the file into the Map member variables. It also calls loadTiles.
2. loadTiles: For all the tiles present in the current map, its load their tile data memory, and updates the reference table created in draw-manager. It also tells us which Sky tile, if any.
3. draw: It iterates over all the loaded tiles, and draws all the background tiles. It also marks all the foreground, and grating tiles, indicating which blitting method is to be used with them.

The Skies, Foregrounds, and Gratings are drawn through separate draw functions, which are still stubbed out. They will be discussed in the next update.

The XY-Crutch

This is one of the main problems that I encountered over the past few days. It occurred because a) I didn’t trace the call stack that is used to start a map completely and b) due to the structure of MSM file.

In a nutshell, in each level, the hero has a starting position. The Map::draw function works on the assumption that the map has been centered to the hero’s starting position. The code that fetches the foreground indices does so relative to the position that the map has been centered on, referred to as _mapX and _mapY. If I had fully traced the Game::StartMap function in the original, I would have figured this out sooner. Hence, simply calling the draw function from the main file fed garbage values to the foreground indices, and I kept getting an out-of-bounds error from them.

While tracing the entire Game::StartMap function would have been a good idea, we ended up reading the MSM file in a hex editor. At the foreground offset(0x343C), we found a series of FF values which continued as we reached 0x3A2C. From there on, a clear pattern emerged that you can see below.

So right now, what we have done is set _mapX and _mapY to predefined values in accordance with the above pattern. This has been of some use, and now we can draw the partial map that you saw above. However, even this does not allow all the foreground tiles to be drawn. My next order of business would be to properly implement the Game::StartMap to the extent that I can right now so _mapX and _mapY don’t have to depend on predefined values.

Avoidable Problems and Missing Files

There was another stupid bug that I should’ve caught earlier. Since I managed to spend more than an hour debugging this, I think it deserves a mention. If anything, it demonstrates how easy it is for a game-breaking bug to slip in if you’re not careful.

The draw-manager has a set of three Sky tiles that are always loaded into memory whenever the draw-manager is initiated, regardless of the map-loader. Once I started debugging the Map::draw function, it became apparent to me that these tiles were not getting loaded.

Since I had been dealing with missing for a while now, I decided to check that over. Thankfully, the files were present in the archive. However, it seemed that the draw-manager refused to acknowledge that they were present. From there, I thought that the tile reference table might be messing up the Sky tiles. I went through the init function, checking and rechecking the tile reference table code. Along the way, I improved the code with more standard methods but it still gave the same error.

At this point, I was fairly certain that the Sky tiles existed, and the reference table was aware of them. Then, the logical conclusion becomes that the Sky tile was messing up the indices somehow. The Sky tile code is distributed throughout three functions: drawSky, isSky, setSky. They interact with each other, and that made it slightly hard to think of them intuitively. I soon learnt that the wrong Sky index was being passed into the function calls, and at the root of it was isSky. Simply speaking, isSky checks if a tile is a Sky tile and returns its skyIndex. Since there only 7 Sky tiles, they are indexed in a Sky tile array, indexed from 0 to 6. This array maps the skyIndex to the corresponding index of the tile reference table.

My mistake was that I had set isSky to return index + 1, where index is the tile index of the input tile. So instead of returning an index for the Sky tile array, it was returning an index for the tile reference table — and a completely wrong index at that! Changing index + 1 to i + 1 solved the problem.

Another issue: I’m still getting used to git in a project of this size. One thing that happened this week: I added a few lines that I thought were perfectly correct, and I pushed them to my remote. As one might expect, the changes broke the project to the point it stopped compiling. After I had corrected the problems, sev has given me a simple workflow to standardize commits:

Compile -> Diff -> Commit

A similar thing happened during the debug process. I had to rebase a few commits sev had pushed. There were a few merge conflicts, and I accidentally overwrote the changes that I had rebased for. In a hurry to correct the changes, I reset my local branch by three commits. Thankfully, I had merely added a few declarations and small initializations at that point which was easy to replicate.

Apart from my own foolishness, I found that certain files were missing from the MPC demo. According to the original source code, it should have been packaged with both the demo and the full version. In the beginning, I had started building the HDB engine based on the demo files. After this, I had to switch files and now I’m basing it on the full game.

Objectives

1. Trace and implement the StartMap function.
2. Add the Masked Blitting and Alpha Blitting functions to DrawMan.
3. Complete the stubbed-out drawSky functions.
4. Add code to draw Foregrounds and Gratings.

After a few days of code crunching, I can finally say that my LuaScript subsystem is operational enough that a portion of the game can be run with it.

LuaScript

The LuaScript subsystem allows us to load a Lua script and execute its default code in a new Lua environment. It also register the custom Lua extensions that were created for HDB.

The process consisted of implementing a simple interpreter, hooking it up to the file-manager and executing the bundled Lua scripts through Common::SeekableReadStreams. Alongside with the interpreter, Lua error reporting facilities were added to the engines. In case Lua faces a problem, a stack traceback is printed to the screen.

Along with a standard interpreter, the Lua subsystem in the original had a series of Lua extensions, Global Strings, Global Values, Sound and Entity Spawn names attached with them. As of right now, I have created stubbed-out versions of the extensions and registered them with each Lua environment. I’ll be adding the Globals as and when I need them. The Sound and Entity Spawn names will be added once those subsystems have been implemented.

The coming weeks would involve filling out these stubbed functions by building the subsystems they rely upon.

Changes from Lua 4.0 to 5.1.3

During the proposal period, I had performed a high-level overview of the differences between Lua 4.0 and Lua 5.1.3 in order to gauge how difficult it would be to run Lua 4.0 code through a Lua 5.1.3 interpreter. While the HDB codebase was largely compatible with the changes, it seems that there were a number of smaller changes that were not. Here is the list of the ones I have found so far.

1. Upvalue Syntax: Upvalues are the closet thing to static function variables in Lua. The syntax for accessing them inside a function was changed from %x to x for an upvalue-referenced variable x.
2. setglobal: Lua 4.0 had a predefined function for assignment. In certain cases, such as dynamically determining the name of a variable, it led to cleaner code and has been used in certain places.
3. for Loop Syntax: From Lua 5.0 to 5.1, there was a subtle change in the scope of the implicit variables of the for and for the repeat statement. Now for i, v in table has to be replaced by for i, v in pairs(table).

Another thing I found, which is not a difference in the Lua syntax, is the usage of C++ style comments in the Lua codebase. Apparently, these were put there for a reason since there is code to preprocess it in the original sources. For my implementation, I have replicated their method entirely.

Patches

In the beginning, the plan was to write a preprocessor function to transform the code into valid Lua 5.1.3 code. However, the syntax changes proved to be both unique and countably few that makes more sense to just patch the differences wherever they occur. Now, we have a ScriptPatch system in place that keeps a track of known differences, and updates them before executing a file.

Objectives

1. The current objective to build the map-loader, and draw an entire map to the screen.
2. Add the map animations once a complete single frame of the map has been drawn.
3. Add the main hero to the map, at which time we can work on input and AI.

The first week of GSoC is nearly over, so this seems to be the perfect opportunity for an update.

Accomplishments

Starting on a positive note, let’s go over the things that well. When I originally planned my project, I expected that I will have to build the low-level subsystems before going on to anything higher-level. Now that I have had some time to go through the original engine’s codebase in depth, I have realized that I don’t need that much low-level work to basically get started. So, I am revising my expectations: I would likely spending a lot of time going back and forth between the low-level subsystems and the actual game code.

Moving towards the actual tangible accomplishments, I have gotten pretty comfortable working with Surfaces, and drawing things to the screen. I started by drawing lines, and then moved on to drawing from the data file. The filesystem I had setup before had a few errors and I spent most of yesterday rooting them out. But it definitely paid off since now I can extract the various tiles, pictures and maps from the MPC file.

Today, I finally nailed down the drawing from file portion. Now that I have a good understanding how to extract data through the ScummVM API, I have managed to draw images and tiles to the game screen. Below is a screenshot of the Logo Screen, with a Ground Tile put on top.

As far as Lua is concerned, I’m just getting started with it and its my next target. From the Lua code I have read in other engines, it seems I have write an extended Lua interpreter with code extensions. Seems doable, but I’m still trying to get a feel for the Lua-C API. As far as the compatibility problem is concerned, I’m going to go ahead with 5.1.3 right on, and add the compatibility file if errors start propping up.

Objectives

1. Setup the LuaScript code so I can load and execute Lua chunks.
2. Understand the existing Lua code further, namely how to extend the default interpreter with new functions.
3. Once the Lua system properly works, I can try to load the Lua and execute portions of the game.