Month: July 2023

The backend part of handling the downloading of DLCs (games) is mostly done for ScummVM Cloud. In the context of DLC downloading, ScummVM Cloud refers to the hosted DLCs on ScummVM’s servers.

Let me first quickly iterate the overall lifecycle of downloading a DLC from ScummVM Cloud:

1. Users request and retrieve the DLC list, which contains information about downloadable DLCs. This list will be cached until ScummVM is restarted.
2. Users can select any DLC and press the Download button to add a download task to the queue. The download tasks in queue will be processed one by one.
3. The DLCs are downloaded as .zip file in dlcpath.
4. The .zip file is then extracted in a new directory inside dlcpath, and after the extraction process, the .zip file is automatically deleted if possible.
5. An additional game entry is then automatically added to the ScummVM configuration file (scummvm.ini). These game entries will also have a special download key.
6. The newly downloaded DLC is now available in the game list of the launcher, and users can now enjoy playing this DLC.

Initially, I implemented the greying out of the text for downloaded DLCs using an index (integer) based download key. Since I had already changed the DLC download key to a string, I have now adapted the code to work with a string. For each game entry in ScummVM’s configuration file with a download key, we find the corresponding game and mark them for greying out.

Also, there’s a simple way for greying out the text in a List widget. We just prepend \001C{alternate} in the string which we want to color grey. And that’s it.

From the first point of the lifecycle of a DLC above, you can see that we needed to send the HTTP request to the server only when the users requested it. This request is basically clicking on the button for browsing DLCs. Before, I was fetching the DLC list on the ScummVM launch. I have now shifted the call of the fetching process from scummvm_main() (on start) to the constructor of the DLC browsing dialog. We show the text: “Fetching DLCs…” while waiting for the list, and once we finish the fetching process, we send a command to our DLC browsing dialog (GUI) to redraw itself with the fetched DLC list. This DLC list is cached until ScummVM is closed and gets reused the next time we browse the DLCs.

The other thing I implemented was adding the game entry in ScummVM’s configuration file after the extraction process was completed. And after that, refresh the launcher’s game list right away. Initially, I was under the impression that I had to first detect the game and then add the entry. There’s an existing method already implemented that does both. This method executes after we select a game directory using “Add Game” button. After discussing with my mentors, we decided to go with adding the details that the ScummVM config file requires in the JSON response itself. Since we already know the details of what exact game we are providing, we don’t have to run the detection code and can just put the information in the JSON file. I have now temporarily shifted the hosting of the JSON server to my Firebase instance. I can add/edit information directly from the browser, and its web address will also remain the same. Once the entry is added to ScummVM’s configuration file, we refresh (rebuild) the launcher GUI.

Finally, I added a new path for storing DLCs called dlcspath. Before, I was using the iconspath. I had to set the path in each of the platform’s backends. I have currently set it to platforms based on SDL, like Linux, macOS, Windows, etc.

Here’s how the overall progress looks:

I spent my time implementing the extracting the DLCs methods this week. Since the DLCs were downloaded as .zip files. We needed a way to extract the files from the zip file.

The process of extracting the contents was simple: use the existing method called dumpArchive() after creating the new Archive object. You can read more about the ScummVM’s Archive class on the doxygen documentation. We already have a method called makeZipArchive() that does the job of creating an archive object using the zip in the provided path.

However, there were some problems with the current dumpArchive() method. All files extracted were getting renamed to something like xn--lureprotect.pdf-ua69e. The original corresponding file is protect.pdf of a game called Lure of the Temptress. This was an issue because ScummVM’s game detector could not automatically recognize these files with the new name. When adding a new game from the ScummVM’s GUI, we were presented with the could not find any game error.

The new file names are punycode encoded file names. The purpose of punycode encoding is to ensure compatibility with all operating systems by processing different symbols (like /, :, etc.) in filenames. If we were to remove the punycode encoding, certain special symbols in filenames might not be allowed/supported on certain operating systems. For example, we could not name Echo I/O:Room in MacOS since it didn’t allow colon : for naming a file/folder as it used : as a path separator.

The problem in the current punycode encoding was that the file name we were initially getting from the zip file was a relative file path instead of just the file name (like lure/protect.pdf and not protect.pdf). Indeed it should be a path; how else are we going to preserve the original location of the zip’s contents with multiple folder hierarchies? Punycode encoding this path name with the special symbol slash (/) prevented it from getting detected. We already have a class called Path, which can contain string components of any file path. For example, the components of a path abc/def/g.h are abc, def, g.h. By using Path instead of String, we can differentiate between the /, the path separator, and / that appears in the file name. To get the Path for each file in the zip, I used the method that was recently created in this PR. Also, the punycode encoding method for the path object was already implemented, which executed the original punycode encoding of string for all the path’s components.

Once the zip file is downloaded and extracted, it should be deleted to free up the user’s storage. For deleting the zip file, a simple call to unlink() with the file’s path is sufficient, and the file will be deleted. This unlink() is included in <unistd.h>, which is included in POSIX-supported operating systems. On Windows, we needed to call _unlink() instead.

Apart from the extracting, there was also a bug in my session request code for starting a download session of DLC. Once the DLC was downloaded, this session was not automatically killed and remained in paused state. And I was getting the error callback called whenever I quit the ScummVM program. This was fixed by calling the session request’s close() method in the download file callback at the end of the stream (i.e., DLC is downloaded) to delete the request.

This week I started with implementing the helper functions for Google Play Store in Java. However, we later decided to initially use ScummVM’s exisiting download manager since it will let us quickly implement the downloading functions and can give us the opportunity to design a POC (proof of concept) for the ScummVM Downloader. It will also be better to initially have basic functions and not be biased towards implementing functions for a particular distribution platform as we need to extend it to other distribution platforms.

The existing download manager utilizes cURL and has appropriate methods to download/fetch something from a URL. In my code, I have DLCManager class which provides the interface for calling DLC-related methods. There’s also the Store abstract class, deriving ScummVMCloud class, where I have used the method provided by the existing download manager. Right now, there were only two required functions that I had to implement in ScummVMCloud. Both are asynchronous methods as they both are network requests.

The first is getAllDLCs(), which sends a request to a URL and expects a JSON response. Initially, I was thinking of the simple comma-separated approach for the schema (similar to https://downloads.scummvm.org/frs/icons/LIST), but that would not work since there were many special characters (including the comma) in the names and URLs of DLCs. Fortunately, the ScummVM codebase has already implemented CurlJsonRequest, which can handle the JSON response. The JSON response will contain an array where each item will contain the description for DLC – name, id, URL, size, etc. This method will run once on start (in scummvm_main()) and set our DLC Array. The DLC Array maintains the information about DLC as well as their state (available, downloading, downloaded, cancelled, error downloading, etc.).

The second method is startDownloadAsync(), which starts the downloading process as the name suggests. Under the hood, it creates a SessionRequest (already implemented), which maintains the session and downloads the file on the URL. The downloadFileCallback() repeatedly runs after downloading a fixed chunk and lets us run any function like update state, update the downloaded size, calculate speed, etc. There is a method for identifying if the file is completely downloaded so we can start extracting the file (if compressed). Once downloaded and extracted, I can add the game entry in the scummvm.ini file (ScummVM’s configuration file).

Finally, the downloading works and the list shown in the dialog for browsing DLC is no longer hardcoded.

This week I came up with a DLC downloader design and implemented some of the functions in DLC Manager.

DLC Manager before was just a shell with empty definitions, and I had not given much thought to what the functions or how the DLC Manager should behave.

Now that I have given some thought, here is a rough flow of the DLC Downloader: For downloading DLCs, we are providing a GUI with an available downloadable games list. Users can select a DLC and click the Download button to start the download. However, the download button under the hood adds this DLC as a download task in the DLCManager’s queue. The queue maintains the queued download tasks. We will be processing each download task one at a time. Once a download task is finished/canceled/interrupted, then the next one in the queue starts downloading.

But how are we achieving this? We have a processDownload() function which looks at this queue. If it’s not empty (i.e., there is/are pending download task(s) that we need to handle), we will use an async function called startDownloadAsync() to handle the downloading from the distribution store. The startDownloadAsync() will, of course, be different for different distribution stores and will be executed in another thread to not block the GUI.

The callback function provided in the startDownloadAsync() will run the processDownload() after startDownloadAsync() finishes executing. The callback function will do two things – first, notify download has finished/canceled/interrupted and then, invoke the processDownload() function again to process the next download task in the queue.

There is now a further GUI requirement; since users will need a way to see what is currently downloading and its progress, we would need another modal that can be opened from a button. I have the rough UI design in mind for this new “downloading” modal.

Finally, here’s the overall flow of the DLC downloader:

This week I started extending DLC Manager to support Play Store.

For handling the downloading of on-demand games, Google has provided two SDKs – one for native (C/C++) and another one for Java/Kotlin.

At first, I was going with native play asset SDK. This SDK consists of some header files (like asset_pack.h), a .cmake file, and shared (.so) and static (.a) libraries. The .cmake file has helper functions that add shared/static libraries to the project. Using native play asset SDK caused me some troubles – it required us to include these SDK-specific includes in our build system, which might not be an easy task. However, the major reason I had to switch to using Java SDK was – developers will need to download manually and set up the native SDK. I could probably add the complete SDK itself in our ScummVM repository to save some setup time, but that will be around 400 MB.

So, we decided to take the “easy and maintainable” approach – Just use the Play Asset SDK for Java. There’s no need to manually download the SDK, and we only require to add a new dependency (com.google.android.play:asset-delivery:2.1.0) in our Android’s build.gradle. Then we can simply use all its functions (like fetch() that requests download) in java. However, it is not that easy since we need to call these functions from C++.

That’s where JNI (Java Native Interface) comes into play. It acts as a bridge between Java code and Native (C/C++) code. Using JNI, we can call any method of an object implemented in Java from C++ or vice versa.

Currently, in Java, I am creating a new AssetPackManager’s object using AssetPackManagerFactory’s getInstance() static method, and from C++, I am trying to call its function using JNI and getting its response. The response would be a jobject type. Initially, I thought I could handle this returned jobject in my C++ code (i.e. manipulate it to return the C++ function’s expected return type) but it seems to be difficult. Lephilousophe suggested me to handle those DLC-related functions on Java side first. So, I would have a simple API that I can call from C++ that can return something simple (like String) instead of  something complex (like AssetPackLocation).

As a side note, there was also some problem with my code that made builds on Mac and Windows fail. I was also able to fix it. But I still need to fix my commit history for failing builds.