tkachov's GSoC blog

First of all, I finally passed my exams! I’ve got a few things to do, but in July I’d be completely free to work on my project. That’s why I wasn’t too active this week and wouldn’t be on the next one.

So, this week I was working on non-cloud-related thingie: a new GUI widget, which would contain other widgets of unlimited total height — the Container. Well, it’s actually a ScrollContainerWidget, and the main thing it has is a scrollbar. As I said in the previous post, one of my dialogs — the storage wizard — is too lengthy to be displayed in that area where ScummVM shows its dialogs. We decided that I should implement the Container, so we can put dialog contents in it, and in these cases when there is no room to display all of them, the scrollbar would automatically appear and no overlap will happen.

The first thing I changed was label — it already has a clipping rectangle, so I only had to pass the right values. Then the fun (to be read as difficult) part started. Widgets doesn’t have any clipping rectangle, which means you can’t just make the button draw only part of it. And, well, each widget has its own method, so it means I’ve got to change all of these in order to make them support clipping.

I started with buttons and PopUpWidget, because those are used in the Cloud tab of the Options dialog. Their drawWidget() method calls the corresponding drawButton()/drawPopUpWidget() method of ThemeEngine. ThemeEngine uses queueDD() and queueDDText(). Some DrawData objects are created in there, then their draw steps are called. If I recall correctly, draw steps are methods of VectorRenderer, which has VectorRendererSpec backends with implementations of these. So, I’ve got to pass that clipping area information from Widget though ThemeEngine and DrawData objects right to VectorRendererSpec and use it there.

Each of those draw step implementations has some additional functions in VectorRendererSpec. And these functions have some complicated logic and lots of macros usage. And none of these happen to think that clipping would ever be needed.

So, buttons are using drawRoundedSquare() draw step method. To minimize the possibitility of breaking something, I decided to leave these methods and add new ones — drawRoundedSquareClip() in this case. It checks that clipping is actually needed (i.e. button area is not completely in the clipping area), and if not, uses the original one. If clipping is needed, then other Clip-versions are used. So, drawRoundedSquareAlgClip(), drawBorderRoundedSquareAlgClip(), drawInteriorRoundedSquareAlgClip(), gradientFillClip(), blendFillClip(), colorFillClip() and a lot of macros with names like BE_DRAWCIRCLE_XCOLOR_BOTTOM_CLIP() arrived.

All this — for rounded squares. And there are also triangles, circles and other draw steps, which I didn’t touch yet. But still, Container already works with buttons, PopUpWidget and labels, and I guess in some cases it would be enough to add new draw<WidgetName>() and use all these methods, which already do clipping, so it should be getting easier as I go further. The plan is to make all other widgets support clipping, so Container could be used with any of them. Apart from that — it seems to work fine.

This week I have upgraded the StorageWizardDialog. Instead of one big text field for code, we decided to put eight small text fields. The scummvm.org page shows the code you should enter, and each code group correctness is checked separately.

ScummVM also automatically checks that the whole code is correct, too. When you enter a valid code, «Connect» button becomes active and you can get your cloud storage working.

However, we don’t think that 40 symbols long code is an easy thing to enter. We’re working on different ways to simplify that task, and one of them — probably the ultimate one — is not to enter that code at all! To achieve that, we’re adding a local webserver into ScummVM. For now, it looks like this:

When you open the StorageWizardDialog in the ScummVM built with SDL_Net (i.e. with local webserver support), there are no fields. Only a short URL for you to visit.

When you do, you’re redirected to service page, where you can allow ScummVM to use that service’s API. When you press the «Allow» button, you’re automatically redirected to ScummVM’s local webserver’s page — and ScummVM automatically gets the code!

I’ve also done a few updates of cloud sync procedure, so now it automatically started when you connect to a storage, or when you save the game or when the autosave happen. The last time I said that if you decide to cancel the sync, the currently downloading file would be left damaged. Well, not anymore — such files are automatically removed.

Finally, we came up with the plan! 😀

First, some of my GUI dialogs require more space than ScummVM can provide. So, we decided that I should implement a «container box» widget — an area with scrollbars, which would be providing more virtual space than we really have.

After I do that, I’d start working on cloud files management: users would be able to upload and download their game data right through ScummVM interface.

Then we’re going to add a «Wi-Fi sharing» feature. It basically means that local webserver would be also used to access files on the device through a browser. No more USB cables! You can copy files from your phone by clicking a link in your browser or upload files onto your phone with that.

Then comes the next cloud storage support — Box. I know nothing about it yet, but hopefully it should be similar to those three I already added.

And in the end, I’d work on those things that would make it easier to enter the code if your ScummVM doesn’t have the local webserver feature. Those are browser opening and clipboard support. So, ScummVM would be able to automatically open a browser for you to allow it to use storage’s API and you’d be able to copy the code and paste it in the ScummVM’s text fields. These are interesting features, but as far as I know, those are highly not portable. For example, there is ShellExecute on Windows and special Intent on Android, but no such thing in Unix. And not all platforms even have the clipboard. From the other hand, there is SDL_GetClipboardText()… in SDL2.

Anyway, I’ve got one (and probably the most difficult) exam to pass the next week. Yeah, an exam and the midterm at the same week. I hope I’ll be able to do at least that «container box» 😀

So, this week I worked on the GUI mostly. That Cloud Options tab is almost fully functional — we want to modify Connection Wizard a little bit, so it helps users enter the code (which is quite long, about 40 characters). Apart from that everything in the tab works as it should, so you can easily switch between your connected storages, see how much place your saves occupied on your cloud drive and when the last successful saves sync was. The Connection Wizard works too, yet, as I mentioned, it won’t notify you of typos. We’re working on corresponding scummvm.org URLs, and these work quite fine.

I’ve also added Google Drive support this week. As Google Drive uses file ids instead of paths, that wasn’t a very pleasant adventure. I mean, imagine you want to download a file knowing a path to it. However, you can’t just ask to find that file because there are no paths in Google Drive. So, you have to divide the path by separators, start from the Google Drive root folder and implement path traversal by listing the directory, finding a corresponding subdirectory and repeating these steps. And, as there are no paths in Google Drive, there could be files with the same name! So, there could be two ScummVM folders or three Saves folders or five soltys.000 files. Well, I just use the first one’s id, so if you’d try to mess up with the Google Drive’s ScummVM folder — you’d succeed. Also, as we wanted to give more freedom to the users, we had to ask for the whole Google Drive access, because application data folders are completely hidden from users. ScummVM only creates its own folder in the Google Drive root directory, so fear not!

Finally, I worked on the progress dialog for the save/load manager. During the saves sync some slots could be unavailable because they are being downloaded. You’d see the following dialog in this case:

Most engines use the common pattern for their save files, so I called these engines «simple». If it’s «simple», you can press that «Run in background» button and you’d see all the slots. Some of them would be «locked» and would lack the thumbnail, but all the others would be easily available to save or load. Unfortunately, there are also «complex» engines. I was unable to implement that «locked» slots feature for these because of their complex nature, so «Run in background» would be unavailable. You would have to wait until all saves are downloaded, and only then save/load feature would be available. You can always hit the «Cancel» button, but that could leave the currently downloading file damaged, so use it on your own risk =)

I’ll have to pass my exams the next week, so I already started working less. The next week plan is to upgrade that Connection Wizard dialog and start working on the local webserver feature. (Yeah, I don’t want you to enter that code when ScummVM could do it for you instead.)

Not sure what was the original week plan, but I managed to finish Dropbox and OneDrive storage implementations and add the saves sync feature. And that was a midterm milestone in my proposal plan (mostly because I tried to schedule all the difficult work after midterm as I’m going to have my exams very soon — right before the midterm).

So, now ScummVM knows how to upload and download saves from Dropbox and OneDrive and automatically does that on launch. When I did that, I decided to work on some GUI-related tasks: add an indication icon that is shown when active storage is syncing something, for starters. Well, that took the whole week 😀 But now it’s an extra nice icon in the corner which automatically appears and disappears when needed. And it’s pulsating when active!

Now I’d work on the GUI further: add a sync progress dialog in save/load manager and forbid using save slots which are being synced. In case I wouldn’t know what to do with the GUI, I can always work on Google Drive storage implementation.

UPD: by the way, I’ve updated the icon, so now it’s completely created by me.

After discussing API design, I’ve done some research on Dropbox, Google Drive and OneDrive API and discussed further steps with Peter and Eugene. I’ve identified some preparation steps I should do before GSoC starts. Doing this preparation work would also get me familiar with ScummVM code, coding & formatting conventions and commit guidelines.

I’m working on it in my public fork of scummvm repo on Github right in the master branch. Eugene helped me with configure (which I’m not using yet, as I’ve started with MSVC here) and watches how I’m doing.

My preparation work plan includes the following tasks:

• integrating JSON parser;
• writing API skeleton;
• adding libcurl and writing some simple wrapper for it;
• adding some cloud-related configs keys, so ScummVM would remember my access token and I won’t have to authenticate every time I launch it.

 

We’ve decided that SimpleJSON is a good library to use as JSON parser in ScummVM. It’s really quite simple library, which consists of two classes and uses C++ standard library. It is now available as Common::JSON in my fork and uses ScummVM classes.

A few weeks back, I still was thinking that all cloud-related work would be done within separate thread spawned in main(). But when I’ve decided to actually add such thread (so my JSON examples could work and not delay ScummVM’s launcher), I understood that the original idea to use libcurl’s blocking functions is a bad idea. libcurl’s functions block thread execution until the request is complete. Because of that, our cloud thread won’t react to the user’s commands when it’s «busy». And that means if you started downloading 2 GB on 56 kbps, you can’t cancel it without killing ScummVM process!

That’s totally not what we want. There is no direct thread spawning in ScummVM and even though TimerManager actually runs callbacks in separate threads, such threads might work even when ScummVM main thread is finished (and TimerManager could be implemented so it won’t be running callbacks in separate threads anymore). So, I had to think of something. At first my idea was to add a callback through TimerManager, which would be executed every second. It would’ve checked whether the user wanted ScummVM to make some API requests and whether these requests were complete. But most of the time we don’t work with cloud, and thus such callback would be called even though it was unnecessary.

Now the idea is a bit more complex. We would have to use async libcurl functions, because we don’t want our callback to block. So, we can’t make our API methods to return the requested information, but instead those should receive some function pointer and call the passed function when this information is ready. I also believe that there could be more than one request pending, and some methods would require more than one REST API request.

Thus, I think of making special Request class, which would act as a base class for all methods implementations. For example, ListDirectoryRequest would be not only requesting the first «page» of directory contents, but make such requests until whole directory is listed. When all the information is gathered, it would call the callback.

Storage class, which would actually represent cloud storage API in ScummVM, has all the API methods we need. But these methods would be creating new Request objects only. The implementation would be hidden within these Request classes. So, Storage class would also contain some Request list. When it’s empty, there is nothing to do for Storage, and it does nothing. But when it is not, Storage would start a timer and poll these requests, whether they have complete what they were doing. When the request is complete, it is removed from this list. When list gets empty, Storage stops the timer.

Finally, there is also a CloudManager. This is a class, which would be loading all connected storages from ScummVM configs and providing fast access to the currently active Storage. So, the idea is to have all the storages loaded and ready, so user can easily switch between them in the Options menu. But only one Storage would be «active». This storage would be used to sync saves, upload and download files until user changes that. CloudManager would not only have a getter for it, but also a few «shortcut» methods, which would be using current storage within. For example, you won’t have to do cloudManager->getCurrentStorage()->syncSaves(). Instead, you can just do cloudManager->syncSaves().

I have not implemented Requests yet, but there are already some simple CloudManager and Storage stubs.

UPD: I’ve implemented first simple Request today. It’s not doing anything useful, but it shows that requests can «work» for a few handler() calls and then stop. When no requests are working, timer is automatically stopped.

Why do we need timer in the first place? Well, that’s because we want to know whether request is complete and we have to poll curl_multi_info_read to do that. Plus, this allows us to react to user’s commands. For example, user might want to cancel the download, and then Request would be stopped on the next timer tick.

One of the most important tasks for my project is to design the API right. That’s something I would start with, so if there are any mistakes or wrong decisions, I’d be suffering from those later. Of course, I don’t want to suffer or rewrite the API over and over again, so I should think it through.

I already mentioned I wrote a small prototype, which uses Dropbox API and has simple sync feature. I actually thought I should write prototypes for all cloud providers I’m going to add support of, so I get experience of using those. That would help me find common parts and differences, and with all this experience I would be ready to design a good API — considering all pitfalls I met writing prototypes.

But a few days back I thought that I can just imagine an «ideal» API: something simple, designed exactly for my goal. What do I need? I need listing files, uploading and downloading. OK, I also would need deleting, updating modification date and creating directories. And… that’s it. And, as that’s an API, I don’t care how it’s implemented. I care that backend does these operations, no matter what it does in order to do these. (Even though it’s me who will be writing these backends.)

I still would study Google Drive and OneDrive APIs, and may be I’ll get some free time in order to write prototypes for those too. But for now I’ve designed two simple sketches I’ve shown to my mentor, Peter Bozsó (uruk-hai), and Eugene Sandulenko (sev).

First one is quite similar to what I’m using in my prototype:

file {
    path
    name
    size
    timestamp
    is_directory
}

service {
    file[] list_directory(path)
    bool upload(path, file_contents)    //may be "save"
    file_contents download(path)        //may be "read"
    bool delete(path)
    bool sync(service)
    bool create_directory(path) 
    bool touch(path)                    //update modification date
    service_info info()                 //username, available disk space, etc
    bool is_syncing()
}

In this one file is a struct with a few important fields. What I need is path, size, timestamp and a flag to know whether this file is a directory. No service-specific file id and stuff. All the operations are declared in service and different backends implement those, using this common file struct.

The other is a little bit more object-oriented:

file {
    path
    name
    size
    timestamp
    is_directory
    service

    file[] list()
    file_contents get_contents()
    bool set_contents(file_contents)
    bool delete()
    bool touch()            //update modification date
    bool sync(file)
}

service {
    file get_root_directory()
    file get_saves_directory()
    bool create_directory(path) 
    service_info info()     //username, available disk space, etc
    bool is_syncing()
}

There file has a reference to the service it belongs to and includes all file-related operations. That means I would have to implement file backends as well.

The thing is I’m fine with both of these approaches, and mentors opinions were divided. So, they advised me to write this post and get some feedback about it.

By the way, I also have an idea of making a backend for local filesystem as well. There is an AbstractFS in ScummVM, but I think it would be better to use one small Wrapper over it in order to work with local files in terms of the same API I’m working with remote ones.

OK, so that’s the first post. Looks like this separate GSoC blog page works like I want it to.

I’m participating in Google Summer of Code for the second time, and today I’ve learned that my proposal to ScummVM was accepted! I’m going to add cloud storage integration into ScummVM, so users would be able to sync their saves and game data between their devices.

I wasn’t posting anything about my work last year (even though I mentioned I probably would in my proposal), so I don’t have a lot of experience writing blog posts in English (actually, I don’t write blog posts in Russian too often as well). Keeping this blog is one of the requirements of ScummVM, so I’d be posting something there at least every week during GSoC.

I’m a little bit busy these days with all my studies, but I had some free time a few weeks back, so I made the first prototype. It has no GUI and it can use Dropbox to upload or download files. I also wrote some basic syncing, which works with files and directories (meaning if you have old file on local machine and a directory with newer files on remote server and names of this file and that directory are the same, it would correctly remove the file, create a directory and then download its contents).

And I guess that’s it for today. I’m not yet sure when I’ll write again.