ankushdutt's Blog

Google Summer of Code is coming to an end. Thanks to my mentors, it has been a pleasant experience.

Here are the things I have worked on during this Summer:

Universal Packaging Script

Code: https://github.com/scummvm/scummvm/pull/5300

Tasks Done

1. Identify metadata requirement for DLC download support for various distribution stores. It has been compiled here.
2. Create schema for metadata and JSON files like dlc-games.json and export-platforms.json to maintain all required metadatas for games and how to process those metadatas for different distribution stores.
3. Write a python script that takes game location, platform type and the ScummVM binary location as parameter. It will determine the output based on the export-platforms.json and dlc-games.json. E.g. For Android Play Store, we will get a new AAB file bundled with games.
4. Add entry for Android Play Store in export-platforms.json and all required dependencies (bundletool.jar, python scripts, etc.).

Possible Improvements

1. It takes a lot of time to package games one by one. Create an option to mass package games from a folder.
2. There are some dependencies that are required to be installed on the host computer. E.g. specific version of Android SDK, Java, Python, etc. Utilize container based approach for quick setup.

In-app DLC Downloader

Code: https://github.com/scummvm/scummvm/pull/5134

Tasks Done

1. Create GUI dialogs and widgets to interact with the DLC Downloader and to show download progress, in-progress DLCs, etc.
2. Create DLC Manager to manage downloading DLCs.
3. Create an abstract class called Store. Different distribution store will need to implement this class using their store-specific API/SDK.
4. Create a new path called dlcspath for storing the downloaded DLCs.
5. Create a new compile time flag, USE_DLC for including DLC downloading support. This is disabled by default. To enable it, use --enable-dlc with ./configure. It will only be included on backends where it is supported (kFeatureDLC is true).

Possible Improvements

1. GUI dialog for DLC Downloader could see some visual overhaul like list with game icons.
2. Generalize DLC to allow other types of data like themes, icons, shaders, etc. Currently, only games is supported.

ScummVM Cloud Integration for DLC Downloader

Code: https://github.com/scummvm/scummvm/pull/5134

Tasks Done

1. Implement ScummVM Cloud to download games from a URL (ScummVM’s server) by utilizing existing networking methods.
2. Handle zip extraction and deletion of zip after extraction.
3. Improve existing method Archive::dumpArchive().

Android Play Store Integration for DLC Downloader

Code: https://github.com/ankushdutt/scummvm/commit/c313a43cf1415598144c816474eb38ddbe56a8b1

Tasks Done

1. Implement Play Store DLC support by using Play Core SDK for Java.
2. Create JNI methods in C++ to interact with Java code for Play Store or vice-versa. Java code utilizes the SDK to implement functions necessary for handling download.

Possible Improvements

1. Create script to generate the list of assets with required details like – id, name and detection keys for ScummVM configuration like engineid, gameid, extra, guioptions, language, platform and description directly by utilizing the dlc-games.json.

Documentation

Link: https://wiki.scummvm.org/index.php?title=Game_Packaging

The implementation of Play Store is mostly done. The downloading works along with the download progress and cancel functions. These functions need to be implemented on the java side since we use the Play Asset SDK for Java. I also needed to call these functions from C++ and vice versa. So, for this, JNI comes into play. The current code for Android also uses JNI extensively for some of its functions. So, the base work was already done. I had to check the current implementation of one of the JNI functions and follow the steps used. The other thing I needed to do was create simple functions that handle Playstore SDK on the Java side.

In Playstore SDK, there is an interface called AssetPackManager that contains the functions used in managing the downloads for the DLCs from Playstore. For starting a download, we can use the fetch() method. It only needs a packname, which is a unique string to identify a game package. I also had to create a listener that keeps track of the download progress, completion, and cancel events to call required functions.

To support the download of the DLCs, Play Store requires us to bundle the base APK (ScummVM) together with all the on-demand assets in an AAB format. I had previously created a Python script to bundle the AAB (containing only base APK) with games. The workflow for creating the final ready-to-upload AAB should be: extract games somewhere, fill some file (JSON or something else), and run gradle to generate the full AAB. The current workflow is more involved. I will also simplify this process next week.

Now only one week is remaining. I will focus on completing the Play Store support and simplifying the process of the AAB build. And, of course, writing proper documentation.

This week I had my work reviewed by my mentors, and I mostly worked on resolving these.

This involved forward-declaration of several classes in the header files, which only refer to those classes as pointers. This will help reduce the compilation time since the compiler will not have to process additional headers.

Also, I have now introduced a new configure feature called USE_DLC along with USE_SCUMMVMDLC. This USE_DLC can be activated by --enable-dlc configure flag. However, if developers want to specifically use the ScummVM cloud (hosted on ScummVM’s server), they need to provide --enable-scummvmdlc with configure. The USE_DLC is a general flag, which can be used with any dlc-supported backend if needed. For example, for android play store release, developers will use --enable-dlc, and for android apk distribution on ScummVM website, they can omit it. Both --enable-dlc and --enable-scummvmdlc is disabled by default.

To determine if the DLC feature is supported in the current architecture (backend), we check if kFeatureDLC is set. We set it in the backend of supported platforms.

Another thing I worked on was error handling during the extraction process. I had to handle errors like archive is broken/incomplete and storage is full. For this, I have to make changes to the existing dumpArchive() function, which extracts the archive in the provided path. I had to make the return type of this function Common::Error and return the error. Once dumpArchive()‘s change is merged, I added the checks in ScummVM Cloud’s code so if there is any error, we show the error to the user, and also we set the downloading DLC’s state to kDownloadingError and not add the DLC to the scummvm configuration file.

I have also removed the incomplete Play Store code from my PR. So, it will add support for the ScummVM cloud only. I have started working on adding the Play Store support on a separate branch. The Android building (using Docker) on Mac with silicon chip was very slow. So, I had to shift to my old laptop running Ubuntu. There has been a bit of progress in adding Play Store, but I think it’s doable in a few days.

Now we are very close to completion of Google Summer of Code program. I plan to make a PR for the Playstore support and add documentation and HOW-TOs to the wiki next week.

This week I have finished working on the GUI for showing the progress of the download of a DLC. Other than the progress, all the queued downloads are also listed. There’s also a text widget to show an error if there is an error in extracting the zip (i.e., the archive is broken). Users can select a DLC and cancel the download by clicking on the Cancel button. Here’s a video for the GUI:

The cancel button works as follows – If the DLC is already in download progress, it will interrupt the download by closing the request (and remove any incomplete .zip). If the DLC is scheduled in the download queue, it will change the state of the DLC to kCancelled to get it skipped. I am checking if the DLC state is kInProgress before starting an async download request.

I have also worked on several bug fixes and improvements like disabling the download/cancel button when no DLC (list item) is selected, fixing the launcher not refreshing when the DLC browsing modal is closed, restricting adding downloads to the queue if it is already in the queue, etc.

Finally, as discussed with my mentors, I have wrapped this ScummVM cloud in configure enabled feature, which will be disabled by default. It can be enabled using the --enable-scummvmdlc flag.

What’s next? After cleaning up the code and preparing the PR for review, I will work on adding support for Play Store DLCs on a separate branch in the coming weeks.

The backend part of handling the downloading of DLCs (games) is mostly done for ScummVM Cloud. In the context of DLC downloading, ScummVM Cloud refers to the hosted DLCs on ScummVM’s servers.

Let me first quickly iterate the overall lifecycle of downloading a DLC from ScummVM Cloud:

1. Users request and retrieve the DLC list, which contains information about downloadable DLCs. This list will be cached until ScummVM is restarted.
2. Users can select any DLC and press the Download button to add a download task to the queue. The download tasks in queue will be processed one by one.
3. The DLCs are downloaded as .zip file in dlcpath.
4. The .zip file is then extracted in a new directory inside dlcpath, and after the extraction process, the .zip file is automatically deleted if possible.
5. An additional game entry is then automatically added to the ScummVM configuration file (scummvm.ini). These game entries will also have a special download key.
6. The newly downloaded DLC is now available in the game list of the launcher, and users can now enjoy playing this DLC.

Initially, I implemented the greying out of the text for downloaded DLCs using an index (integer) based download key. Since I had already changed the DLC download key to a string, I have now adapted the code to work with a string. For each game entry in ScummVM’s configuration file with a download key, we find the corresponding game and mark them for greying out.

Also, there’s a simple way for greying out the text in a List widget. We just prepend \001C{alternate} in the string which we want to color grey. And that’s it.

From the first point of the lifecycle of a DLC above, you can see that we needed to send the HTTP request to the server only when the users requested it. This request is basically clicking on the button for browsing DLCs. Before, I was fetching the DLC list on the ScummVM launch. I have now shifted the call of the fetching process from scummvm_main() (on start) to the constructor of the DLC browsing dialog. We show the text: “Fetching DLCs…” while waiting for the list, and once we finish the fetching process, we send a command to our DLC browsing dialog (GUI) to redraw itself with the fetched DLC list. This DLC list is cached until ScummVM is closed and gets reused the next time we browse the DLCs.

The other thing I implemented was adding the game entry in ScummVM’s configuration file after the extraction process was completed. And after that, refresh the launcher’s game list right away. Initially, I was under the impression that I had to first detect the game and then add the entry. There’s an existing method already implemented that does both. This method executes after we select a game directory using “Add Game” button. After discussing with my mentors, we decided to go with adding the details that the ScummVM config file requires in the JSON response itself. Since we already know the details of what exact game we are providing, we don’t have to run the detection code and can just put the information in the JSON file. I have now temporarily shifted the hosting of the JSON server to my Firebase instance. I can add/edit information directly from the browser, and its web address will also remain the same. Once the entry is added to ScummVM’s configuration file, we refresh (rebuild) the launcher GUI.

Finally, I added a new path for storing DLCs called dlcspath. Before, I was using the iconspath. I had to set the path in each of the platform’s backends. I have currently set it to platforms based on SDL, like Linux, macOS, Windows, etc.

Here’s how the overall progress looks:

I spent my time implementing the extracting the DLCs methods this week. Since the DLCs were downloaded as .zip files. We needed a way to extract the files from the zip file.

The process of extracting the contents was simple: use the existing method called dumpArchive() after creating the new Archive object. You can read more about the ScummVM’s Archive class on the doxygen documentation. We already have a method called makeZipArchive() that does the job of creating an archive object using the zip in the provided path.

However, there were some problems with the current dumpArchive() method. All files extracted were getting renamed to something like xn--lureprotect.pdf-ua69e. The original corresponding file is protect.pdf of a game called Lure of the Temptress. This was an issue because ScummVM’s game detector could not automatically recognize these files with the new name. When adding a new game from the ScummVM’s GUI, we were presented with the could not find any game error.

The new file names are punycode encoded file names. The purpose of punycode encoding is to ensure compatibility with all operating systems by processing different symbols (like /, :, etc.) in filenames. If we were to remove the punycode encoding, certain special symbols in filenames might not be allowed/supported on certain operating systems. For example, we could not name Echo I/O:Room in MacOS since it didn’t allow colon : for naming a file/folder as it used : as a path separator.

The problem in the current punycode encoding was that the file name we were initially getting from the zip file was a relative file path instead of just the file name (like lure/protect.pdf and not protect.pdf). Indeed it should be a path; how else are we going to preserve the original location of the zip’s contents with multiple folder hierarchies? Punycode encoding this path name with the special symbol slash (/) prevented it from getting detected. We already have a class called Path, which can contain string components of any file path. For example, the components of a path abc/def/g.h are abc, def, g.h. By using Path instead of String, we can differentiate between the /, the path separator, and / that appears in the file name. To get the Path for each file in the zip, I used the method that was recently created in this PR. Also, the punycode encoding method for the path object was already implemented, which executed the original punycode encoding of string for all the path’s components.

Once the zip file is downloaded and extracted, it should be deleted to free up the user’s storage. For deleting the zip file, a simple call to unlink() with the file’s path is sufficient, and the file will be deleted. This unlink() is included in <unistd.h>, which is included in POSIX-supported operating systems. On Windows, we needed to call _unlink() instead.

Apart from the extracting, there was also a bug in my session request code for starting a download session of DLC. Once the DLC was downloaded, this session was not automatically killed and remained in paused state. And I was getting the error callback called whenever I quit the ScummVM program. This was fixed by calling the session request’s close() method in the download file callback at the end of the stream (i.e., DLC is downloaded) to delete the request.

This week I started with implementing the helper functions for Google Play Store in Java. However, we later decided to initially use ScummVM’s exisiting download manager since it will let us quickly implement the downloading functions and can give us the opportunity to design a POC (proof of concept) for the ScummVM Downloader. It will also be better to initially have basic functions and not be biased towards implementing functions for a particular distribution platform as we need to extend it to other distribution platforms.

The existing download manager utilizes cURL and has appropriate methods to download/fetch something from a URL. In my code, I have DLCManager class which provides the interface for calling DLC-related methods. There’s also the Store abstract class, deriving ScummVMCloud class, where I have used the method provided by the existing download manager. Right now, there were only two required functions that I had to implement in ScummVMCloud. Both are asynchronous methods as they both are network requests.

The first is getAllDLCs(), which sends a request to a URL and expects a JSON response. Initially, I was thinking of the simple comma-separated approach for the schema (similar to https://downloads.scummvm.org/frs/icons/LIST), but that would not work since there were many special characters (including the comma) in the names and URLs of DLCs. Fortunately, the ScummVM codebase has already implemented CurlJsonRequest, which can handle the JSON response. The JSON response will contain an array where each item will contain the description for DLC – name, id, URL, size, etc. This method will run once on start (in scummvm_main()) and set our DLC Array. The DLC Array maintains the information about DLC as well as their state (available, downloading, downloaded, cancelled, error downloading, etc.).

The second method is startDownloadAsync(), which starts the downloading process as the name suggests. Under the hood, it creates a SessionRequest (already implemented), which maintains the session and downloads the file on the URL. The downloadFileCallback() repeatedly runs after downloading a fixed chunk and lets us run any function like update state, update the downloaded size, calculate speed, etc. There is a method for identifying if the file is completely downloaded so we can start extracting the file (if compressed). Once downloaded and extracted, I can add the game entry in the scummvm.ini file (ScummVM’s configuration file).

Finally, the downloading works and the list shown in the dialog for browsing DLC is no longer hardcoded.

This week I came up with a DLC downloader design and implemented some of the functions in DLC Manager.

DLC Manager before was just a shell with empty definitions, and I had not given much thought to what the functions or how the DLC Manager should behave.

Now that I have given some thought, here is a rough flow of the DLC Downloader: For downloading DLCs, we are providing a GUI with an available downloadable games list. Users can select a DLC and click the Download button to start the download. However, the download button under the hood adds this DLC as a download task in the DLCManager’s queue. The queue maintains the queued download tasks. We will be processing each download task one at a time. Once a download task is finished/canceled/interrupted, then the next one in the queue starts downloading.

But how are we achieving this? We have a processDownload() function which looks at this queue. If it’s not empty (i.e., there is/are pending download task(s) that we need to handle), we will use an async function called startDownloadAsync() to handle the downloading from the distribution store. The startDownloadAsync() will, of course, be different for different distribution stores and will be executed in another thread to not block the GUI.

The callback function provided in the startDownloadAsync() will run the processDownload() after startDownloadAsync() finishes executing. The callback function will do two things – first, notify download has finished/canceled/interrupted and then, invoke the processDownload() function again to process the next download task in the queue.

There is now a further GUI requirement; since users will need a way to see what is currently downloading and its progress, we would need another modal that can be opened from a button. I have the rough UI design in mind for this new “downloading” modal.

Finally, here’s the overall flow of the DLC downloader:

This week I started extending DLC Manager to support Play Store.

For handling the downloading of on-demand games, Google has provided two SDKs – one for native (C/C++) and another one for Java/Kotlin.

At first, I was going with native play asset SDK. This SDK consists of some header files (like asset_pack.h), a .cmake file, and shared (.so) and static (.a) libraries. The .cmake file has helper functions that add shared/static libraries to the project. Using native play asset SDK caused me some troubles – it required us to include these SDK-specific includes in our build system, which might not be an easy task. However, the major reason I had to switch to using Java SDK was – developers will need to download manually and set up the native SDK. I could probably add the complete SDK itself in our ScummVM repository to save some setup time, but that will be around 400 MB.

So, we decided to take the “easy and maintainable” approach – Just use the Play Asset SDK for Java. There’s no need to manually download the SDK, and we only require to add a new dependency (com.google.android.play:asset-delivery:2.1.0) in our Android’s build.gradle. Then we can simply use all its functions (like fetch() that requests download) in java. However, it is not that easy since we need to call these functions from C++.

That’s where JNI (Java Native Interface) comes into play. It acts as a bridge between Java code and Native (C/C++) code. Using JNI, we can call any method of an object implemented in Java from C++ or vice versa.

Currently, in Java, I am creating a new AssetPackManager’s object using AssetPackManagerFactory’s getInstance() static method, and from C++, I am trying to call its function using JNI and getting its response. The response would be a jobject type. Initially, I thought I could handle this returned jobject in my C++ code (i.e. manipulate it to return the C++ function’s expected return type) but it seems to be difficult. Lephilousophe suggested me to handle those DLC-related functions on Java side first. So, I would have a simple API that I can call from C++ that can return something simple (like String) instead of  something complex (like AssetPackLocation).

As a side note, there was also some problem with my code that made builds on Mac and Windows fail. I was also able to fix it. But I still need to fix my commit history for failing builds.

This week I worked on the backend part for integrating DLC downloader in ScummVM. In the backend, I need to implement download-related functions like requesting a download, getting downloaded progress, getting downloadable games list (with information like name, size, id, etc.), getting downloaded location, handling errors, etc.

Since every distribution stores have its own API/SDKs for implementing these functionalities, I required a way to make the ScummVM know which implementation to run. We can do this via dynamic polymorphism. In my case, I had to create a new abstract class called Store containing prototype for store-specific functions. Using Store, we can derive different classes like PlayStore with implementations specific to the distribution stores.

I need a way to use these download-related functions in my downloader GUI or even in the background. For this, I have created a new class called DLCManager, which gets instantiated as DLCMan on startup (in scummvm_main()). DLCMan will provide the interface for all the functions I will require for the DLC downloader, handling errors appropriately and also showing what went wrong in a message modal.

Here’s the complete overview:

Additionally, in the non-android ScummVM executable, I was able to hide the “Download Games” button, which opens the DLC browser. I have introduced a new OSystem feature, kFeatureDLC, that would determine whether the ScummVM executable supports downloading DLC. If it doesn’t, then I have to hide DLC-related widgets.

You can check all the code changes here.

The backend code is pretty complex. Initially, it was very intimidating to get something done. Thanks to my mentors, I’ve made some decent progress in sorting out the classes and functions. I learned some advanced C++ concepts as well. Hopefully, by next week I will be able to implement the download-related functions for Android Play Store.