{"id":26,"date":"2025-06-16T23:43:18","date_gmt":"2025-06-16T23:43:18","guid":{"rendered":"https:\/\/blogs.scummvm.org\/shivangnagta\/?p=26"},"modified":"2025-06-16T23:43:18","modified_gmt":"2025-06-16T23:43:18","slug":"week-2","status":"publish","type":"post","link":"https:\/\/blogs.scummvm.org\/shivangnagta\/2025\/06\/16\/week-2\/","title":{"rendered":"Week 2"},"content":{"rendered":"

Welcome to the weekly blog.
\nAfter wrapping up work on macfiles last week, I finally moved on to testing the first two phases of the data upload workflow \u2014 starting with scummvm.dat<\/code> (which contains detection entries from ScummVM used for populating the database) and set.dat<\/code> (data from older collections, provides more information).<\/p>\n

Database Changes: File Size Enhancements<\/h4>\n

Before diving into the testing, there was a change Sev asked for \u2014 to extend the database schema to store three types of file sizes instead of just one. This was necessary due to the nature of macfiles, which have:<\/p>\n

    \n
  • \n

    A data fork<\/strong><\/p>\n<\/li>\n

  • \n

    A resource fork<\/strong><\/p>\n<\/li>\n

  • \n

    A third size: the data section of the resource fork<\/strong> itself<\/p>\n<\/li>\n<\/ul>\n

    This change introduced significant modifications to db_functions.py<\/code>, which contains the core logic for working with .dat<\/code> files. I had to be careful to ensure nothing broke during this transition.<\/p>\n

    Punycode Encoding Logic Fixes<\/h4>\n

    At the same time, I fixed the punycode logic in db_functions.py<\/code>. Punycode encoding (or rather, an extended version of the standard used in URL encoding) is employed by ScummVM to convert filenames into filesystem-independent ASCII-only representations.<\/p>\n

    There were inconsistencies between punycode logic in db_functions.py<\/code> and the original implementation in the Dumper Companion.\u00a0I made sure both implementations now align, and I ran unit tests from the Dumper Companion to verify correctness.<\/p>\n

    Feeding the Database – scummvm.dat<\/h4>\n

    With those fixes in place, I moved on to populating the database with data from scummvm.dat<\/code>. While Sev was working on the C++ side to add the correct filesize tags for detections, I ran manual tests using existing data. The parsing logic worked well, though I had to add support for the new “extra size” fields.<\/p>\n

    Additionally, I fixed the megakey calculation<\/strong>, which is used later when uploading the scummvm.dat again with updates. This involved sorting files alphabetically before computing the key to ensure consistent results.<\/p>\n

    I also introduced a small optimization: if a file is less than 5000 bytes, we can safely assume that all checksum types (e.g., md5-full_file<\/code>, md5-5000B<\/code>, md5-tail-5000B<\/code>, md5-oneMB<\/code>, or the macfile variants like -d<\/code>\/-r<\/code> ) will be the same. In such cases, we now automatically fill all checksum fields with the same value used in detection.<\/p>\n

    Uploading and Matching – set.dat<\/h4>\n

    Finally, I worked on uploading set.dat<\/code> to the database, which usually contains the follwoing – metadata (mostly irrelevant), full size checksums only and filesizes.<\/p>\n

      \n
    • \n
        \n
      • scummvm.dat<\/code> doesn’t contain full file checksums like set.dat, so a match between files from set.dat<\/code> and scummvm.dat<\/code> is only possible when a file\u2019s size is less than the detection checksum size, generally 5000 Bytes.<\/li>\n
      • \n

        This transitions the status from \u201cdetection\u201d to \u201cpartial\u201d \u2014 we now know all files in the game, but not all checksum types.<\/p>\n<\/li>\n

      • \n

        If there is no match, we create a new entry<\/strong> in the database with the status dat.<\/code><\/p>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n

        Fixes :<\/h6>\n

        There was an issue with the session variable @fileset_last<\/code>, which was mistakenly referencing the filechecksum<\/code><\/strong>table instead of the latest entry in the fileset<\/code><\/strong>table. This broke the logic for matching entries.<\/p>\n

        When a detection matched a file, only one checksum was previously being transferred. I fixed this to include all relevant checksums from the detection file.<\/p>\n

        \u00a0Bug Fixes and Improvements<\/h4>\n

        Fixed redirection logic in the logs: previously, when a matched detection entry was removed, the log URL still pointed to the deleted fileset ID. I updated this to redirect correctly to the matched fileset.<\/p>\n

        Updated the dashboard to show unmatched dat<\/code><\/strong>entries. These were missing earlier because the SQL query used an inner JOIN with the game<\/code> table, and since set.dat<\/code> files don\u2019t have game table references, they were filtered out. I replaced it with a LEFT JOIN on fileset<\/code> to include them.<\/p>\n

        \n

        That\u2019s everything I worked on this past week. I\u2019m still a bit unsure about the set.dat<\/code> matching logic, so I\u2019ll be discussing it further with Sev to make sure everything is aligned.<\/p>\n

        Thanks for reading!<\/p>\n","protected":false},"excerpt":{"rendered":"

        Welcome to the weekly blog. After wrapping up work on macfiles last week, I finally moved on to testing the first two phases of the data upload workflow \u2014 starting with scummvm.dat (which contains detection entries from ScummVM used for populating the database) and set.dat (data from older collections, provides more information). Database Changes: File […]<\/p>\n","protected":false},"author":29,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-26","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/posts\/26","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/users\/29"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/comments?post=26"}],"version-history":[{"count":1,"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/posts\/26\/revisions"}],"predecessor-version":[{"id":27,"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/posts\/26\/revisions\/27"}],"wp:attachment":[{"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/media?parent=26"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/categories?post=26"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.scummvm.org\/shivangnagta\/wp-json\/wp\/v2\/tags?post=26"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}