Month: July 2023

This week’s blog post will contain work from both week 8 and week 9, since I had college during week 8 and I wasn’t able to get much work done.

These two weeks were spent starting the final piece of the puzzle – handling user checksums, and adding some extra features to the website and CLI tool.

User Checksums

The entire point of this project is to allow users to validate their game files against those present in the db that we spent the first 2/3rds of this project creating.

The user checksums will be validated against whatever we have in the database, and there are two outcomes – the game variant in question has fully populated checksums, or it doesn’t. If it does, the process is pretty simple – just verify the integrity of the game files one-by-one, store the results in a JSON object, and send it back as a response.

If the game variant doesn’t exist in the database, we can insert the submitted checksums in the database. Since the user can submit even incorrect files for verification, we will be adding these filesets to a queue where either a moderator can approve them manually.

Framework Improvements

This week introduced a whole bunch of improvements to the website, parser and CLI scanner.

I’ve introduced a megakey column in filesets to uniquely identify non-detection filesets. This means we can skip insertion of all filesets that already exist.

Game entries in games_list.php now have clickable rows that point to their respective fileset pages. You can also sort by any column by clicking on the heading.

In the fileset page, you can now view checksums of all types and sizes, thanks to a new ‘Expand table’ button. I’ve also added the ability to calculate checksums of AppleDouble and .rsrc files with the scan utility.

Log entries have also gotten an overhaul, now every upload gets its own ‘Transaction’ id and every fileset inserted (or not inserted) is logged.

That’s all for now, hope you check in next week for more progress!

Thanks for reading!

Revisiting the DATs

This week was spent adding additional features to the main framework of the project – the parser and the scan utility. I also changed up the UI of the website.

MacBinaries

In Classic MacOS, one file could have both a resource fork and a data fork. These were physically separate files, but were treated as one object by the OS. While MacOS handled this fine, transmission protocols couldn’t do the same.

Enter MacBinary – a file format that combines both the forks into one file, suitable for transmission over the interwebs, and also, capable of being used on other file systems. ScummVM relies on MacBinaries to handle files from classic MacOS games, which means my scanner should handle them too.

MacBinary files need to be treated differently because the detection happens based on the checksums of the forks, not the whole file – which means I needed to create functionality to detect MacBinaries, and then calculate checksums of their forks for matching.

The difficulty of this part came from identification of the MacBinaries. I had to translate C++ code into some complicated looking Python, and implement a custom CRC function. Took a couple tries, but it works now!

Incremental DATs

The parser is built to read and interpret DATs, and then insert them into the db as is. This feature would add an additional check to prevent the duplicate entries from being inserted when a DAT with the same entries is uploaded.

Implementing this feature efficiently took some thinking (and refactoring!), but I managed to get it to work!

I first had to add an extra column – timestamp – to the fileset table to determine whether a detection entry was outdated (meaning it wasn’t present in the latest detection entries DAT). These entries were marked as obsolete. During insertion, I checked if the detection fileset was already present in the table, with the help of the value in the key column (a unique identifier of sorts). These entries were skipped during insertion.

For DATs coming from the scanner or clrmamepro, it’s a little more tricky. If a game is detected, we can compare the filesets associated with that game, and mark the new fileset for removal if they are the same. We decided to leave removal of unmatched duplicate filesets for a later date, as handling this scenario needs a little more set up.

Some smaller changes

I had also updated the scanner to handle filenames with punycode, and added support for custom checksum sizes (the printing didn’t work before).

I also altered the UI for the filters in paginated tables – there’s one filter for each column now, with support for multiple filters.

That’s all for now, hope you check in next week for more progress!

Thanks for reading!

 

Revamping the Website

Last week when I showed the detection entries webpage, it was far from impressive to say the least. This week was spent upgrading the website in both looks and functionality.

The Looks

Not too much say here, except I spent a little time to glam up the website and give it some functional UI improvements as well. The data is displayed labelled table, with a visual distinction on alternate rows. I also added a color change on hovering over the row so it’s easier to tell which row you’re looking at.

The Functionality

Last week, I had made some very rudimentary pages for displaying the detection entries and logs. This week, I gave them essentially a full overhaul.

First, I extracted the common code for pagination into it’s own file. Then I added some more sophisticated navigation buttons, allowing users to skip to the first and last pages, along with the two pages immediately before and after the current page (as you can see in the screenshot). There’s also a text box to go to a specific page number if you don’t want to manually navigate with the buttons.

As you can see at the top of the page, I’ve implemented filters to narrow down the entries shown. In the screenshot, I’m filtering only the entries that have platform: windows.

In the logs page, for matching logs, the log text now includes a hyperlink to the fileset page of the fileset that was matched.

The Fileset Page

The fileset page exists so developers can manage individual filesets, and it works even for the ones that don’t have a game matched to them.

Fileset details shows the source of the fileset, along with game data and status if it is matched to a game. The next table shows the files in the fileset (also with pagination, some games have a lot of files).

The next two parts are more interesting – Developer Actions and Fileset history. Developer Actions has a list of actions that the developer can do on the fileset. Eventually, this will include approving/rejecting the fileset if it was submitted by a user, but for now it just include a button to mark the fileset for deletion.

The Fileset history table displays the previous IDs that the current fileset had before it was merged. This is useful for redirecting older IDs to their new homes. As shown in the screenshot, there’s also a link to the logs page that shows the log entry for the matching of the filesets.

That’s all for now, hope you check in next week for more progress!

Thanks for reading!

This week was spent completing the main foundation of the project by matching entries, and starting work on the website. I’ve also implemented a logging system for when DATs are uploaded and matched, and set up a test instance for my code.

Matching Entries

When we upload a DAT from, let’s say, our CLI scanner utility, we calculate checksums of the files in the specified folders. But how do we know what games those checksums correspond to? The directory names could be gibberish, and unless a developer manually adds metadata to every single fileset, there’s no way of knowing which game the fileset corresponds to by parsing the DAT alone. So after we parse, we have to match the entries to a specific game.

Matching entries involves identifying which game a fileset is for, by a method similar to what ScummVM itself uses. ScummVM stores the checksums of a couple files from the game, and then uses this to identify what game files a directory contains. In the same fashion, if we happen to find a detection entry where all detection files are found in the new entry, a match is made. After matching, we must merge the filesets, so we can actually use the files in the new fileset.

Log your changes!

This part is pretty simple – know when the database is changed. Whenever we write to the database (i.e when we upload or match entries), we want to make sure that we have a ledger to keep track of our actions.

I added a logging table to the schema and store in it some useful info, like a timestamp, what kind of action was done, who did it, what changed etc. This will show up on the website we’ll be talking about in just a bit.

The Website

One of the biggest parts of the project is having a website that developers can visit and use as a dashboard of sorts to manage the stuff in the database.

This week I implemented two webpages – one to display info about the detection entries, and one to show you the logs. Obviously showing the several thousand games on the same page is a bad idea, so I also implemented pagination to show a more manageable 25 entries per page.

I also spent a little bit of time setting up an apache2 vhost so me and my mentor could test the code on the same environment.

That’s all for now, hope you check in next week for more progress!

Thanks for reading!