Most of this week was spent rewriting the logic for processing set.dat<\/code> files, as the previous implementation had several inconsistencies. As shown in Figure 1<\/em>, the earlier logic directly compared checksums to determine matches. However, this approach only worked when the file size was smaller than the checksum size (typically md5-5000<\/code>), since set.dat<\/code> files only include full file checksums. This caused us to miss many opportunities to merge filesets that could otherwise be uniquely matched by filename and size alone.<\/p>\n
\nFig.1 : Previous query used in matching.<\/figcaption><\/figure>\n
Since set.dat<\/code> files only contain entries that are already present in the detection results (with the exception of some rare variants I discovered later in the week), we should typically expect a one-to-one mapping in most cases. However, some filesets in set.dat<\/code> can correspond to multiple candidate entries. This happens when the name and size match, but the checksum differs\u2014often due to file variants. This case needs to be handled with manual merge.<\/p>\n
Previously, the logic for handling different .dat<\/code> file types was tightly coupled, making it hard to understand and maintain. I started by decoupling the logic for set.dat<\/code> entirely. Now, the candidates for the match for\u00a0 set.dat<\/code> filesets are filtered by engine name, filename, and file size (if it’s not -1<\/code>), excluding out the checksum. It is made sure that all the detection files(files with detection flag set to 1) follow the condition.<\/p>\n
<\/p>\n
Initially, I was filtering out the fileset only with the highest number of matches, assuming it was correct. However, that approach isn’t reliable\u2014sometimes the correct match might not be the largest group. So all these candidates need to go for the manual merge. Only when all checksums match across candidates can we be confident in an automatic match.<\/p>\n
I also added logic to handle partial or full matches of candidate filesets. This can happen when a set.dat<\/code> is reuploaded with changes. In such cases, all files are compared: if there’s no difference, the fileset is dropped. If differences exist, the fileset is flagged for manual merge.<\/p>\n
Finally, I handled an issue with Mac files in set.dat<\/code>. These files aren’t correctly represented there: they lack prefixes and have checksums computed for the full file rather than individual forks. So, these filesets are dropped early by checking if no candidates are found for that fileset after SQL filtering.<\/p>\n
Other Updates<\/h5>\n
During seeding, I found some entries with the same megakey, differing only by game name or title. Sev advised treating them as a single fileset. So now, only the first such entry is added, and the rest are logged as warnings with metadata, including links to the conflicting fileset.<\/p>\n
Other fixes this week included:<\/p>\n
\n
\n
Removing support for m<\/code>-type checksums entirely (Sev also removed them from the detections).<\/p>\n<\/li>\n
\n
Dropping sha1<\/code> and crc<\/code> checksums, which mainly came from set.dat<\/code>.<\/p>\n<\/li>\n<\/ul>\n
Next Steps<\/h5>\n
With the seeding logic refined, the next step is to begin populating the database with individual set.dat<\/code> entries and confirm everything works as expected.<\/p>\n
After that, I\u2019ll start working on fixing the scan.dat<\/code> functionality. This feature will allow developers to manually scan their game data files and upload the relevant data to the database.<\/p>\n","protected":false},"excerpt":{"rendered":"
Welcome to this week’s blog update. Most of this week was spent rewriting the logic for processing set.dat files, as the previous implementation had several inconsistencies. As shown in Figure 1, the earlier logic directly compared checksums to determine matches. However, this approach only worked when the file size was smaller than the checksum size […]<\/p>\n","protected":false},"author":29,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-28","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/posts\/28","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/users\/29"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/comments?post=28"}],"version-history":[{"count":1,"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/posts\/28\/revisions"}],"predecessor-version":[{"id":30,"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/posts\/28\/revisions\/30"}],"wp:attachment":[{"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/media?parent=28"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/categories?post=28"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/tags?post=28"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
![\"\"](\"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-content\/uploads\/sites\/81\/2025\/06\/Screenshot-2025-06-23-at-3.23.50\u202fAM.png\")