Month: June 2026

Categories
Week 2

Following the Warning Lines

Tuesday

This week I finally got SSH access to john.scummvm.net, the machine that runs the Director buildbot. I’d seen its output for weeks (those BUILDBOT: warning lines I kept triggering) but had no idea how it actually worked. Getting access and reading through the code was very satisfying because I finally met the machine that had been yelling at me for the past couple months.

The Buildbot Architecture

buildbot_diagram

How a build starts

The first thing that clicked was how the build is scoped. The buildbot watches the ScummVM GitHub repo via a webhook, but it doesn’t rebuild on every commit. master.py checks whether any changed files are in engines/director or graphics/macgui. If yes, it kicks off a build. If not, it ignores the commit entirely. Simple filter, makes sense.

Triggering the tests
Once the ScummVM binary is compiled and uploaded to the master, build_factory.py calls steps.Trigger(schedulerNames=["Director Tests"]). This tells the buildbot master to fire the Director Tests scheduler, which is a Triggerable scheduler defined in master.py. That scheduler then queues up all the test builders at once; one for each entry in targets.json, so they all start running in parallel on the available workers. The build step waits (waitForFinish=True) for all of them to complete before marking the overall build as done.

Running the tests and error checking

The test runners themselves are defined in targets.py. Each one downloads the binary, rsyncs game files from /storage, and runs ScummVM against a list of movies. Screenshots are saved to /home/director-buildbot/screenshots/ on every run. Error checking is done in steps.py, which watches the output for lines matching “BUILDBOT: incorrect check for line:” – that’s the log-replay mechanism from the director-tests repo, where expected output is recorded directly into the test movie file, and on each buildbot run the live output is compared against it line by line.

Reading through this, the separation of concerns became clear: master.py handles scheduling, build_factory.py handles the build pipeline, targets.py defines what gets tested, and steps.py defines how a single test run behaves. Once I had that mental map, the rest followed quickly.

The ImageDiff tool

pic of the tool from dev chat

One thing the buildbot produces but doesn’t analyze automatically is screenshots. Sev pointed me to ImageDiff, a tool built to catch visual regressions, cases where the engine produces slightly different output without triggering any log-based errors.

It’s a Flask web app that reads from the screenshots directory and shows a frame-by-frame diff between any two builds for a given target and movie. The core logic uses PIL’s ImageChops.difference to compute a pixel-level diff. If there’s any difference, the diff image is rendered alongside the source and comparison frames so you can see exactly what changed. Results are cached to disk so repeated comparisons don’t recompute.

I temporarily deployed it on john.scummvm.net on port 5002. The two targets currently generating screenshots are director (from the director-tests-* folders) and theapartment-mac, both of which showed up with their full build history. It’s a simple tool but it fills a gap that log-replay can’t, visual changes.

Bugs

This week I did not work on any game bug fixes. I only made changes to the visual debugger and its bugs.

The gus games bugs are mostly fixed already and the remaining one bug has not been very consistently reproducible. So, this week I’ll try to find out how to replicate it.

Here is a brief on the DT changes:

Windows Panel

  • Added a new Windows panel showing all currently loaded windows and all .DIR movie files in the game directory
  • Clicking a movie navigates to it via func_goto

Search

  • Added variable search mode to the search bar
  • Improved search with new modes (Handlers, Variables, Body) and a cleaner 3-column results table with keyboard focus on open

PRs:

https://github.com/scummvm/scummvm/pull/7564

https://github.com/scummvm/scummvm/pull/7553

last week’s changes: https://github.com/scummvm/scummvm/pull/7563

What I am currently working on

  • Currently I am working on some more DT changes.
  • Working on the checksum function.
    • Some Director movies have a VWCF resource with a checksum that our implementation fails to verify correctly, causing a crash when navigating to those movies in the debugger.
    • Sev suggested – before diving deep – running the mismatched movies through ProjectorRays first to see if it also miscalculates, that way we know whether the bug is in our implementation or the movies themselves. The code once completely written, will be identical to the Projector Rays checksum code.

P.S. The buildbot integration will be re-worked soon by one of our devs rvanlaar, so the current architecture might not be relevant in the future