Welcome to this week’s blog.
This week primarily involved manually reviewing around 100+ set.dat files—excluding a few, such as Scumm and all GLK engines, since their detection entries are not yet available for seeding.
During the process, I fixed several issues wherever possible, improved the matching, manually removed unwanted entries, and documented everything in a spreadsheet. Some key fixes included adding additional filtering based on platform to reduce the number of candidate matches. In some cases, the platform could be extracted from the gameid (e.g., goldenwake-win). This filtering was needed because many detection entries from the seeding process were missing file size information (i.e., size = -1). While I was already filtering candidates by file size, I also had to include those with size = -1 to avoid missing the correct match due to incomplete data. However, this approach in some cases, significantly increased the number of candidates requiring manual merging. Introducing platform-based filtering helped reduce this count, though the improvement wasn’t as substantial as expected.
Another issue stemmed from duplicate files added during seeding for detection purposes. While the detection code in ScummVM intentionally includes these duplicates, I should have removed them during seeding. Cleaning them up did reduce the manual merging effort in some cases.
There were also complications caused by file paths. Initially, the filtering considered full file paths, but I later changed it to use only the filename as mentioned in the last blog. This led to situations where the same detection file appeared in multiple directories. I’ve now resolved this by designating only one file as the detection file and treating the others as non-detection files.
A significant portion of time also went into manually removing extra entries from set.dat, e.g, different language variants. These often caused dropouts in the matching process, but removing them allowed the main entry to be automatically merged.
Some smaller fixes included:
-
Ensuring all checksums are added on a match when the file size is less than the checksum size (since all checksum would be identical in that case). This logic was already implemented, but previously only applied when creating a new entry.
-
Increasing the log text size limit to prevent log creation from failing due to overly large text in the database.
Next, I’ll begin working on the scan utility while waiting for the Scumm and GLK detection entries to become available.