Welcome to this week’s blog. This week, I primarily worked on the moderation system for user.dat, improved the merge dashboard page for manual merges, and made several other changes related to the website’s user interface.
Moderation System for user.dat
For every incoming user fileset submission, we check whether we already have a matching full fileset. If a full fileset match is found, we return the status of the files. If not, the submission is added to the database as a possible new fileset or a variant, which then goes through a moderation queue.
I worked on implementing this moderation queue. A submitted fileset is added to the review queue only if it is submitted by a minimum number of different users (currently set to three). To prevent redundant count increases from the same user, there is an IP-based check in place. We already had masking logic that anonymizes the last octet of an IPv4 address (e.g., 127.0.0.X) to avoid storing Personally Identifiable Information (PII).
Merge Dashboard
I also worked on enhancing the merge dashboard. The goal was to implement a page that displays both filesets side-by-side for comparison, including checksums, file sizes, and metadata. Moderators can choose the checksums and file sizes from either fileset to perform manual merge. A check was also added to prevent selecting both options for the same comparison field.
However, the page still needs some optimization, as it currently freezes when dealing with a large number of files.
Other Improvements
Some of the other fixes and improvements include:
-
Correctly redirecting filesets after automatic or manual matches
-
Enhancing the homepage
-
Adding error handling while parsing
.datfiles -
Implementing proper transaction rollbacks in case of failure
-
Fixing filtering issues that previously caused some data with null fields to not filter.