Tuesday
Not a lot happened this week because I was travelling.
the hfs file system
The file system used by old mac computers was the Hierarchical File System (HFS). And for director engine that meant a bunch of issues, like allowing the weird file names (solved by punycode) and resource forks.
Mac files have two parts: a data fork and a resource fork. For Director games, the projector executable lives in the data fork, but the startup movie (the first thing the game loads) is in the resource fork.
When ScummVM detects a Mac Director game, it hashes the resource fork specifically (using the r: prefix in detection entries) rather than the data fork, because the data fork is just the generic Director player and would be identical across many games.
This week I added detection entries for Gus Goes to Cyberopolis, both Windows and Mac versions. The process was interesting: you add a placeholder entry with a garbage hash, compile, point ScummVM at the game folder, and it reports back the real hash and file size for you to fill in.
I also learned why two-file detection entries (MACGAME2, WINGAME2) matter: with a single file, ScummVM just picks whichever game has the most files matching, which can cause false matches when multiple games share a disc. Two files makes the match unambiguous.
For Windows, it’s a bit different. The .exe file is actually three things concatenated together: the generic Director projector code, the startup movie, and a small header at the end that stores the offset to the movie. The executable would read its own tail to find the header, then seek back to load the startup movie. This is why ScummVM can’t hash the first 5000 bytes of a Windows Director executable, they’re always identical across every game built with the same Director version. Instead it hashes the last 5000 bytes (t: prefix), which come from the embedded startup movie and are unique to each game.
ImageDiff Integration (final)
I also worked on integrating ImageDiff into the ScummVM buildbot.
I have mentioned about the imagediff tool in my past blogs in detail. It was currently running on a detached tmux session, which is kind of a hack, so sev gave me some resources to read and integrate the tool into actual buildbot.
This process unfortuntely took a lot of time for me, because this was my first time in a long time dealing with a different kind of code.
The integration used a buildbot plugin called buildbot-wsgi-dashboards, which lets you embed a Flask web app directly into the buildbot UI. Getting it to work involved fixing a few issues. Most of the issues were trivial but there was this one issue which was causing a lot of problem and I wasnt able to figure out the root cause for, for the longest time.
The issue was, whenever I loaded the imagediff page on the buildbot, it would load unreliably i.e every time I would refresh the page, I couldn’t predict whether I would get a “Resource not found” error or the page would actually load. On top of that the table wasn’t loading properly.
The fix was actually two separate issues. First, the environment variables weren’t being loaded early enough, the SCREENSHOTS_DIR variable was being read at import time before the .env file had been loaded, so it defaulted to ./screenshots/ which didn’t exist on the server. Adding load_dotenv() at the top of config.py fixed that.
The second issue was the JavaScript on the frontend constructing API URLs without the correct path prefix. Since ImageDiff is served under /plugins/wsgi_dashboards/imagediff/, a hardcoded /api/target_data/... URL would 404 every time. The fix was deriving the base path dynamically from window.location.pathname at runtime. That’s why the page was loading unreliably, depending on timing and caching, sometimes the old URL worked by accident and sometimes it didn’t.
After sorting all of that out, ImageDiff is now properly integrated into the ScummVM buildbot at john.scummvm.org. But it’s still rough around the edges, and the remaining issues will be handled by sev.
PRs: