{"id":31,"date":"2025-06-30T09:31:54","date_gmt":"2025-06-30T09:31:54","guid":{"rendered":"https:\/\/blogs.scummvm.org\/shivangnagta\/?p=31"},"modified":"2025-06-30T09:31:54","modified_gmt":"2025-06-30T09:31:54","slug":"week-4","status":"publish","type":"post","link":"https:\/\/blogs.scummvm.org\/shivangnagta\/2025\/06\/30\/week-4\/","title":{"rendered":"Week 4"},"content":{"rendered":"

Welcome to this week\u2019s blog update. This week was focused on fixing bugs related to seeding and set.dat<\/code> uploading, improving filtering mechanisms, and rewriting parts of the processing logic for scan.dat<\/code>.<\/p>\n

After some manual checks with the recent set.dat<\/code> updates, a few issues surfaced:<\/p>\n

1. Identical Detection Fix<\/strong><\/h5>\n

Some detection entries had identical file sets, sizes, and checksums, which led to issues during the automatic merging of set.dat<\/code>. Previously, we used a megakey<\/code> to prevent such duplicates, but since it included not only file data but also language and platform information, some identical versions still slipped through.
To solve this, I replaced the megakey<\/code> check with a more focused comparison: filename, size, and checksum only, and logging the details of the fileset that got clashed.<\/p>\n

2. Punycode Encoding Misplacement<\/strong><\/h5>\n

The filenames were being encoded using Punycode every time they were processed for database insertion. However, there was no requirement for this encoding as it should have occurred earlier \u2014 ideally at the parsing stage, either by the scanning utility that generates .dat<\/code> files or on the application\u2019s upload interface. I have removed the encoding during database updates. Though I still have to add it at the scan utility side, which I’ll do this week.<\/p>\n

3. Path Format Normalization<\/strong><\/h5>\n

Another issue was related to inconsistent file paths. Some set.dat<\/code> entries used Windows-style paths (xyz\\abc<\/code>), while their corresponding detection entries used Unix-style (xyz\/abc<\/code>). Since filtering was done using simple string matching, these mismatches caused failures. I resolved this by normalizing all paths to use forward slashes (\/<\/code>) before storing them in the database.<\/p>\n

4. Improving Extra File Detection with clonof<\/code> and romof<\/code><\/strong><\/h5>\n

While analyzing filesets, I encountered a previously unnoticed clonof<\/code> field (similar to romof<\/code>). These fields indicate that extra files might be listed elsewhere in the .dat<\/code> file. The previous logic only looked in the resource<\/code> section, but I found that:<\/p>\n

    \n
  • \n

    Extra files could also exist in the game<\/code> section.<\/p>\n<\/li>\n

  • \n

    The file references could chain across multiple sections (e.g., A \u2192 B \u2192 C).<\/p>\n<\/li>\n<\/ul>\n

    To address this, I implemented an iterative lookup for extra files, ensuring all relevant files across multiple levels are properly detected.<\/p>\n

    scan.dat Processing Improvements<\/h4>\n

    For scan.dat<\/code>, I introduced a file update strategy that runs before full fileset matching. All files that match based on file size and checksum are updated first. This allows us to update matching files early, without relying solely on complete fileset comparisons.<\/p>\n

    Minor Fixes & UI Enhancements<\/h4>\n
      \n
    • \n

      Prevented reprocessing of filesets in set.dat<\/code> if a key already exists in subsequent runs.<\/p>\n<\/li>\n

    • \n

      Passing the --skiplog<\/code> CLI argument to set.dat processing to\u00a0 suppress verbose logs during fileset creation and automatic merging.<\/p>\n<\/li>\n

    • \n

      Improved filtering in the dashboard adding more fields like engineid, transcation number and fileset id, and fixing some older issues.<\/p>\n<\/li>\n

    • \n

      Introduced a new “Possible Merges” button in the filesets dashboard to manually inspect and confirm suggested merges.This feature is backed by a new database table that stores fileset matches for later manual review.<\/p>\n<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"

      Welcome to this week\u2019s blog update. This week was focused on fixing bugs related to seeding and set.dat uploading, improving filtering mechanisms, and rewriting parts of the processing logic for scan.dat. After some manual checks with the recent set.dat updates, a few issues surfaced: 1. Identical Detection Fix Some detection entries had identical file sets, […]<\/p>\n","protected":false},"author":29,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-31","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/posts\/31","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/users\/29"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/comments?post=31"}],"version-history":[{"count":1,"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/posts\/31\/revisions"}],"predecessor-version":[{"id":32,"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/posts\/31\/revisions\/32"}],"wp:attachment":[{"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/media?parent=31"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/categories?post=31"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/tags?post=31"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}