Scripting!!!!!!

I just realized that I forgot to do a post last week! I was being so productive, time just flew by.

Last week and the beginning of this week I’ve been working on the script management system for ZEngine. Well, before I get into that, let me go back a little further. According to my original timeline, the next milestone was creating a skeleton engine that could do basic rendering, sounds, and events. So, last Monday, I started by cleaning up the main game loop and splitting everything into separate methods and classes. With that, the run loop looks like this:

Common::Error ZEngine::run() {
    initialize();
    
    // Main loop
    uint32 currentTime = _system->getMillis();
    uint32 lastTime = currentTime;
    const uint32 desiredFrameTime = 33; // ~30 fps
    
    while (!shouldQuit()) {
        processEvents();
        
        currentTime = _system->getMillis();
        uint32 deltaTime = currentTime - lastTime;
        lastTime = currentTime;
        
        updateScripts();
        updateAnimations(deltaTime);
        
        if (_needsScreenUpdate)
        {
            _system->updateScreen();
        }
        
        // Calculate the frame delay based off a desired frame rate
        int delay = desiredFrameTime - (currentTime - _system->getMillis());
        // Ensure non-negative
        delay = delay < 0 ? 0 : delay;
        _system->delayMillis(delay);
    }
    
    return Common::kNoError;
}

No bad, if I do say so myself. 🙂

That done, I started implementing the various method shells, such as processEvents(). It was about that time that I realized the the structure of the scripting system had a huge impact on the structure of the engine as a whole. For example, should the event system call methods directly, or should it just register key presses, etc. and let the script system handle the calls? I had a basic understanding of how it probably worked, knowing the history of adventure games, but it was clear I needed to understand the script system before I could go any further.

The .scr files themselves are rather simple; they’re text-based if-then statements. Here’s an example of a puzzle and a control:

puzzle:5251 {
    criteria { 
        [4188] = 1
        [4209] ! 5
        [7347] = 1
        [67] = 0
    }
    criteria { 
        [4209] > 1
        [7347] = 1
        [67] = 1
        [4188] = [6584]
    }
    results {
        action:assign(5985, 0)
        background:timer:7336(60)
        event:change_location(C,B,C0,1073)
        background:music:5252(1 a000h1tc.raw 1)    
    }
    flags {
        ONCE_PER_INST
    }
}

control:8454 push_toggle {
    flat_hotspot(0,265,511,54)
    cursor(backward)
}

Puzzles:

  • Criteria are a set of comparisons. If ANY of the criteria are satisfied, the results are called.
    • The number in square brackets is the key in a ‘global’ variable hashmap. (The hashmap isn’t actually global in my implementation but rather a member variable in the ScriptManager class)
    • Next is a simplified form of the standard comparison operators ( ==, !=, <, > ).
    • The last number can either be a constant or a key to another global variable.
  • Results are what happens when one of the criteria is met. The first part defines a function, and the remaining parts are the arguments.
  • I haven’t fully figured out flags, but from what I can see it’s a bitwise OR of when results can be called. For example, only once per room.

For those of you that understand code better than words:

if (criteriaOne || criteriaTwo) {
    assign(5985, 0);
    timer(7336, 60);
    change_location('C', 'B', "C0", 1073);
    music(5252, 1, "a000h1tc.raw", 1);
}

Controls:

  • I haven’t done much work on controls yet, but from what I have done, they look to be similar to results and are just called whenever interacted with. For example, a lever being toggled.

The majority of the week was spent working on the best way to store this information so all the conditions could be readily tested and actions fired. The best way I’ve come up with so far, is to have a Criteria struct and a Results struct as follows:

/** Criteria for a Puzzle result to be fired */
struct Criteria {
    /** The id of a global state */
    uint32 id;
    /**  
     * What we're comparing the value of the global state against
     * This can either be a pure value or it can be the id of another global state
     */
    uint32 argument;
    /** How to do the comparison */
    CriteriaOperator criteriaOperator;
    /** Is 'argument' the id of a global state or a pure value */
    bool argumentIsAnId;
};

 

/** What happens when Puzzle criteria are met */
struct Result {
    ResultAction action;
    Common::List<Object> arguments;
};

CriteriaOperator is an enum of the operators and ResultAction is an enum of all the possible actions. The other variables are pretty self explanatory.

Using the Criteria and Result structs, the Puzzle struct is:

struct Puzzle {
    uint32 id;
    Common::List<criteria> criteriaList;
    Common::List<result> resultList;
    byte flags;
};

Thus, the process is: read a script file, parse the puzzles into structs and load the structs into a linked list representing all the currently active puzzles. Elegant and exceedingly fast to iterate for criteria comparison checking. Now, some of you may have noticed the ‘Object’ class and are probably thinking to yourselves, “I thought this was c++, not c# or <insert terrible coffee-named language here>.” It is, but that is a whole post to itself, which I will be writing after this one.

So, a couple hundred words in, what have I said? Well, over this past week I discovered how the script system determines what events to fire. This has helped me not only to design the script system code, but also has given me insight into how to design the other systems in the engine. For example, I now know that mouse and keyboard events will just translate to setting global state variables.

What I have left to do in the ScriptManager:

  • Figure out what CriteriaFlags are used for
  • Create shell methods for all the Result ‘actions’
  • Write the parser and storage for control and figure out how they are called

Well that’s about it for this post, so until next time,
-RichieSams

ZFS File format

Over the years I’ve reverse engineered quite a few file formats, but I’ve never really sat down and picked apart why a format was designed the way it was. With that said, I wanted to show the ZFS archive file format and highlight some of the peculiarities I saw and perhaps you guys can answer some of my questions.

For some context, Z-engine was created around 1995 and was used on Macintosh, MS-DOS, and Windows 95.

Format
The main file header is defined as:
struct ZfsHeader {
    uint32 magic;
    uint32 unknown1;
    uint32 maxNameLength;
    uint32 filesPerBlock;
    uint32 fileCount;
    byte xorKey[4];
    uint32 fileSectionOffset;
};
  • magic and unknown1 are self explanatory
  • maxNameLength refers to the length of the block that stores a file’s name. Any extra spaces are null.
  • The archive is split into ‘pages’ or ‘blocks’. Each ‘page’ contains, at max, filesPerBlock files
  • fileCount is total number of files the archive contains
  • xorKey is the XOR cipher used for encryption of the files
  • fileSectionOffset is the offset of the main data section, aka fileLength – mainHeaderLength
The file entry header is defined as:
struct ZfsEntryHeader {
    char name[16];
    uint32 offset;
    uint32 id;
    uint32 size;
    uint32 time;
    uint32 unknown;
};
  • name is the file name right-padded with null characters
  • offset is the offset to the actual file data
  • id is a the numeric id of the file. The id’s increment from 0 to fileCount
  • size is the length of the file
  • unknown is self explanatory
Therefore, the entire file structure is as follows:
[Main Header]
 
[uint32 offsetToPage2]
[Page 1 File Entry Headers]
[Page 1 File Data]
 
[uint32 offsetToPage3]
[Page 2 File Entry Headers]
[Page 2 File Data]
 
etc.
Questions and Observations

maxNameLength
Why have a fixed size name block vs. null terminated or [size][string]? Was that just the popular thing to do back then so the entire header to could be cast directly to a struct?

filesPerBlock
What is the benefit to pagination? The only explanation I can see atm is that it was some artifact of their asset compiler max memory. Maybe I’m missing something since I’ve never programmed for that type of hardware.

fileSectionOffset
I’ve seen things like this a lot in my reverse engineering; they give the offset to a section that’s literally just after the header. Even if they were doing straight casting instead of incremental reading, a simple sizeof(mainHeader) would give them the offset to the next section. Again, if I’m missing something, please let me know.

Well that’s it for now,
-RichieSams

Git is hard, but ScummVM Common is awesome

This week I started working on Z-engine proper… And immediately ran face-first into the complexity of git. Well, let me restate that. Git isn’t hard, per-se, but has so many features and facets that it can very easily go over your head. Anybody with a brain can mindlessly commit and push things to a git repo. However, if you really want structured and concise commit flow, it takes not only knowing the tools, but actually sitting back and thinking about what changes should be put in what commits and which branches.

So that said, I’ll go over the things I really like about git or just distributed source control in general.

Branchy development is absolutely a must. It’s really really helpful to separate different parts of a project or even different parts of the same section of a project. It makes identifying and diff-ing changes really easy. Also, I found it’s really helpful to have a local “work-in-progess” version of the branch I’m working on. That allows me to commit really often and not really have to worry about commit message formatting or general structure. Then when I’m ready to do a push to the repo, I rebase my commits in my WIP branch to fit all my needs, then rebase them to the main branch before pushing.

On that note, rebase is AMAZING!!! It’s like the “Jesus” answer in Sunday school, or “Hydrogen bonding” in chemistry class. However, “With great power comes great responsibility”. So I try my hardest to only use rebase on my local repo.

On to details about Z-engine work!!

My first milestone for Z-engine was to get a file manager fully working, seeing how pretty much every other part of the engine relies on files. When I was writing my proposal for GSoC, I thought I was going to have to write my own file manager, but Common::SearchManager to the rescue!

By default, the SearchManager will register every file within the game’s directory. So any calls to

Common::File.open(Common::String filePath);

will search the game’s directory for the filePath and open that file if found.
Well that was easy. Done before lunch…. Well, not quite. Z-engine games store their script files in archive files. The format is really really simple, but I’ll save that for a post of itself. Ideally, I wanted to be able to do:

Common::File.open("fileInsideArchive.scr");

After some searching and asking about irc, I found that I can do exactly that by implementing Common::Archive:

class ZfsArchive : public Common::Archive {
public:
    ZfsArchive(const Common::String &fileName);
    ZfsArchive(const Common::String &fileName, Common::SeekableReadStream *stream);
    ~ZfsArchive();
    
    /**
     * Check if a member with the given name is present in the Archive.
     * Patterns are not allowed, as this is meant to be a quick File::exists()
     * replacement.
     */
    bool hasFile(const Common::String &fileName) const;
    
    /**
     * Add all members of the Archive to list.
     * Must only append to list, and not remove elements from it.
     *
     * @return the number of names added to list
     */
    int listMembers(Common::ArchiveMemberList &list) const;
    
    /**
     * Returns a ArchiveMember representation of the given file.
     */
    const Common::ArchiveMemberPtr getMember(const Common::String &name) const;
    
    /**
     * Create a stream bound to a member with the specified name in the
     * archive. If no member with this name exists, 0 is returned.
     * @return the newly created input stream
     */
    Common::SeekableReadStream *createReadStreamForMember(const Common::String &name) const;
}

and then registering each archive with the SearchManager like so:

// Search for .zfs archive files
Common::ArchiveMemberList list;
SearchMan.listMatchingMembers(list, "*.zfs");
 
// Register the files within the zfs archive files with the SearchMan
for (Common::ArchiveMemberList::iterator iter = list.begin(); iter != list.end(); ++iter) {
    Common::String name = (*iter)->getName();
    ZfsArchive *archive = new ZfsArchive(name, (*iter)->createReadStream());
    
    SearchMan.add(name, archive);
}

In summary, git can be complicated, but it has a wealth of potential and is extremely powereful. Also, the ScummVM Common classes are absolutely fantastic and make the lives of engine developers sooooo much easier. A toast to the wonderful people who developed them. Well, that’s all for now.

So until next time, happy coding. 🙂
-RichieSams

Obligatory “Hello world!”

Hello world!

Welcome to my new blog: ‘RichieSam’s Adventures in Code-ville’. This will be the place I share the coding experiences have and learn while working on my various projects. With that said, who am I and what am I working on?

I’m a 21 year old, fourth year student studying at The University of Texas at Austin. I’m majoring in Mechanical Engineering with a minor in Computer Science. I thoroughly enjoy programming, both the thrill of gettting something to work and the science/math of algorithms and data structures. The majority of my programming projects have revolved around games. The first major project I did was creating an application that tracked guild currency for my guild. My latest project is a suite of tools to let users install, modify, and create game asset modifications of the game League of Legends. It required reverse engineering quite a few file formats and learning how to hook the game process in order to allow run-time asset swapping.

The two big projects I’m working on right now are The Dargon Project and Z-engine for ScummVM. The Dargon Project is the aforementioned suite of applications. Z-engine is my project for Google Summer of Code.

Z-engine:

The Z-Engine is used in the games Zork Nemesis and Zork Grand Inquisitor. Marisa Chan created a C implementation of the engine, but it is only for desktop and requires configuration files. The project aims to create a ScummVM engine using Marisa Chan’s implementation code as a guide into the Zork file structure and engine design. That is, it will not simply adapt the current implementation to the ScummVM engine structure. Rather, it will create a new engine, using the file structures and event implementations in Marisa Chan’s code as a reference. ScummVM will allow these games to be played on a variety of platforms and a redesign will remove the need for configuration files. Lastly, it will mean that ScummVM will support all of the Zork point’n’click adventure games.
I’m absolutely thrilled be one of the lucky people to be a part of Google Summer of Code. ScummVM is an amazing group of developers and I’m really looking forward to being a part of that.
Well, I guess that’s it for now. My next post will most likely be about that start of GSoC.
Until then,
-RichieSams