tkachov's GSoC blog

(* my site puts a post into RSS only if it has «GSoC» in it, so I kinda had to)

One of my favourite adventure games from my childhood are Red Comrades — these, actually, might even be the reason why I liked the genre. So, I’ve been keeping my eye on it after it was announced that the engine will be added by one of the GSoC students. Finally, there was a support announcement, and I’ve decided to play the games during the holidays.

This post is some kind of a less-formalized-than-bugtracker feedback. I’m going to actually submit some reports later, providing comparison with how the actual game worked, with screenshots, saves and probably even videos. For now it is only my experience compared with the impression of the game I remember.

Overall, the support is good. Both games are fully playable and there are no gamebreaking bugs — you can play it from the start to the end. Now onto my grumbling.

The Fonts

First thing that caught my eye: the fonts. I’m not sure why I’ve always played the game with subtitles, but now for me the game is incomplete if subtitles are off. I didn’t compare screenshots of the font yet, and IIRC it was just a usual Arial — but it looked a little bit off to me. To be more precise, there are multiple microissues with fonts/subtitles that I have.

First: I can see subtitles «jump» and I don’t remember that in the original game. Meaning that line height is different for different glyphs: for example, if line has «p», it will draw higher than a line without it. It can be easily seen in the dialogs how such lines go after each other and show up on different position. If the subtitles text is long, it will appear in two or three lines on the screen, and you could also see sometimes that these lines have different line height, as if one line is glued too close to another.

Second: there was a different font for inventory descriptions. ScummVM uses the same as in subtitles.

Third: when player is prompted to select an option in a dialog, these options have no outline or something. I don’t remember how exactly it looked in the original game, but for some reason I believe it was easier to read than in ScummVM.

The Walking

A really long time ago I and some other folks were making a fan continuation of the Red Comrades on Wintermute Engine. We were tinkering with original game resources, and ripped off all the backgrounds, music, sounds, animations, text lines and even walking regions. Now I don’t remember exactly if the original game also had some anchor points or did we add these in WME — but I remember that in original you could see heroes walking along some curved roads in the backgrounds pretty naturally. In ScummVM, however, this behaviour seems to be broken sometimes. I will prepare some save files to illustrate that in issues, but for now I will try to describe it. Again, there are two small issues that might (or might not) be independent.

First: a hero might ignore the walking region. Sometimes you could see how they are walking on the air or on the wall — obviously, outside the walking region. Sometimes I could see «Walk bug: Point doesn’t belong to any convex» message in the ScummVM terminal window, but I guess that only happened if heroes were spawned outside the walking region after fast travelling via the map. But when that happened, they just jump to the closest point which is in the region, so I’m not sure that’s the reason.

Second: pathfinding is a little bit strange. For example, we have three buildings on the screen: on the left, on the top and on the right. Say, our hero is at the top building entrance. We click at the right building. The hero starts walking towards the left building, and when they reach the left building entrance, they turn around and walk to the right building where we asked them to go. Not really critical, as they go where we want to in the end, but seems pretty strange, and, again, I don’t remember that in the original game.

The Rendering

I only remember that in the first game, and it’s not like that was happening all the time, but there were a few places where animations could «glitch» a little. What I mean is that a part of the sprite is not erased, so, for example, we could see two heads of the same character.

In some cutscenes there could be some «rainbow rubbish» pixels at the bottom. Not frequently, and I’m not sure that is easily reproduceable, but I’ve seen it happen.

The Cutscenes

Sometimes (mostly in Red Comrades 2?) there was just a lag instead of an animation/cutscene. For example, a hero walks towards an object and must play a unique animation of picking up or something, but instead your game freezes for a moment and then continues after (as if the animation was played and nothing happened).

Another variation: no animation is playing, but you can hear the sound effects, and the game continues after cutscene «has played». Most striking example is in the beginning of «Part 3» in Red Comrades 1, and I even have a save file for it. It shows a pretty long cutscene, but all you see is a static background.

In Red Comrades 2 there was another issue once: the game shows you that something has changed on the other screen, but in ScummVM this animation is very short, so you could even miss it.

Finally, there was an audio desync in the first cutscene in Red Comrades 2: you already seen the action on the screen, and only then you hear the corresponding sound effect.

The only almost-gamebreaking bug

In Red Comrades 1 I once had a strange dialog situation (maybe it activated twice or something): the dialog has stopped, as if it was waiting for me to choose an option, but there were no options on the screen, no cursor and no context menu available. So I couldn’t walk away or select an option to stop the dialog or pretty much do anything at all. But I think I just pressed Escape and dialog finished, so I didn’t lose any progress because of reloading.

The Speed

OK, another thing that was one of the first I’ve encountered, is walking speed. It is configured on the same screen you can turn subtitles on, so I’ve made it a little bit faster, and got a very fast walking heroes. I’ll have to compare that with the original, but I think I’ve got the max speed while it wasn’t on max in the options. I’ve just lowered it and played normally, so that’s not much of an issue.

The loading speed, however, is. It wasn’t painfully slow while I was playing, but when I’ve launched the original game, I was surprised how fluent it was. When switching screens, or loading animations, or loading some assets like inventory suitcase and options screen, you can easily see that ScummVM loads it, because it takes some time. In the original game it is barely noticeable. This might be the reason of some issues I’ve mentioned up there — such as audio desync or lags instead of the animations. TODO: I think the game was able to work with unpacked archives, and maybe ScummVM would work faster if I’d unpack its archives manually.

The Saves

Almost forgot about these! It was a little bit strange to see that you can’t use ScummVM own saving dialog. The cloud sync is not triggered when game is saved (or loaded via in-game menu), which is also a little bit unusual. And there are no autosaves, which could’ve been useful for me, because I didn’t save really frequently and thus I was lazy to play through the entire sequence again just to reproduce a bug I have just seen. Otherwise I would’ve have more saves to demonstrate these issues with.

Some technical information

I’ve used a daily build, ScummVM 2.3.0git11675-ga455b4c84a (Dec 31 2020 04:59:17).

I’m aware of two versions of the Red Comrades 1 game (which have different .exe files), but the only important difference between these is probably that one version has a compressed audio. I believe I’ve been playing the lossless version, and the game .exe version is 1.0.0.1.

I’ve been mostly playing on a laptop with Win 10 1909 (x64), and I also did a few quick runs on a desktop with Win 7 (x64).

I’ve posted this picture on IRC yesterday:

It inherits that green/white scheme of mine. The navigation menu from the side moved in dropdown lists on the top panel. ScummVM description is a bit shorter, shows an actual application screenshot and calls to download the newest version. Then goes the two-column layout, and the right column can contain not only screenshots section, but also Contribute and Donate sections.

Making of logotype & color scheme

As my original idea was to make a clean minimalistic look, not to change familiar colors, let me show you how exactly I came to what I have now:

So, let’s try designing a logotype!

Step 0. Open an editor.

Step 1. OK, what’s the most important part of ScummVM logo? Green “Scumm” and gray “VM” parts, for sure! As we’re trying to achieve the minimalistic look, borders and stuff would be left out:

Looks good for me!

Step 2. So, the other part of logo is background color, which is familiar ScummVM orange:

Oh. Probably gradients did all the job. Or the borders. May be try different tones?

Not really better.

Step 3. OK, so they say that you could make a solid color scheme not only of complementary colors, but also of analogous. Let’s try replacing orange with green:

Hey, it’s not that bad. And it’s not exactly green, it has some yellow tones in it. The closest to orange I could achieve =)

Recently I’ve thought to try designing a new fresh look for ScummVM. This is what I came up with:

As you might’ve noticed, it’s a little bit inspired by Steam: games listed on the left and current game displayed on the right. Saves are available right from the very first screen.

Options dialog stayed pretty much the same, but tabs are replaced with a list. That allows to add more tabs without any scrolling and makes it look similar to main screen.

I was unable to add the familiar orange color into this scheme without ruining it, so I replaced it with pale green. The whole look is kind of minimalistic: no gradients, borders, rounded angles and such. The similar approach could be used to redesign ScummVM site.

These sketches show a 1024×768 desktop look, while not doing anything about platforms with smaller resolutions. I think it’s OK to go with a simple skin for those, meaning the layout stays the same and only the colors/fonts/images change.

I worked with ScummVM GUI system, so AFAIK it wouldn’t be too hard to change those layouts to correspond these sketches. That couldn’t be achieved with a simple skin, because some minor changes needed to place saves dialog in main dialog or to replace tabs with a list.

P.S. It’s not about GSoC, but my site is configured to show only those posts which has “GSoC” prefix in the RSS feed Planet is aggregating.

What I was working on during GSoC is Cloud storages support in ScummVM. Describing this feature in my proposal (mirror), I mentioned that it would include an API to interact with supported storage providers (which are Dropbox, OneDrive, Google Drive and Box), saves syncing mechanism, functionality to upload and download games data, and, of course, GUI for all of these. Proposal also has some extra tasks mostly making user experience better.

Some things were rediscussed during my work, but the main idea remained intact. The work is done and pull request already awaits final review before getting merged. API for all four announced storage providers works fine, saves are syncing and games data could be easily downloaded. Not only described extra tasks were complete, but also some functionality not mentioned in the proposal was added.

Saves sync is probably the main reason why Cloud storages support is needed at all. It allows users to easily continue playing the game on another device by simply connecting both to the same storage and doing the sync. It’s automatically started on ScummVM launch, on games saving (including autosaves) and when user opens Save/Load dialog. This dialog was updated to show a progress bar while syncing and also to «lock» slots which are being synced. To indicate that there is a sync in progress, small Cloud icon is shown in the corner.

To use the feature, users must connect a storage first. To do so, they should navigate into Cloud tab of Options dialog, select a storage provider and press «Connect» button. It opens a special Storage Connection Wizard, which provides the instructions on connecting. It has different variations depending on set of libraries ScummVM was compiled with. In the most simple case it says users should navigate to a special short link (to scummvm.org), which redirects them to provider’s page. When they allow ScummVM to use their storage on that page, they are redirected back to scummvm.org, where the code is shown. This code should be typed in the wizard dialog. It’s used by ScummVM to connect to the storage and use provider’s REST API then.

ScummVM page makes the code that way so wizard could check that code has no mistakes in it. If there is a mistake, it notifies user where it probably is. If ScummVM was built with SDL2, pasting from clipboard is supported. Wizard also has «Open URL» button, which makes it easier to navigate to provider’s page on platforms where URL opening was implemtented (these are Windows, Linux, Mac OS X, Android — iOS and Symbian are coming).

But it’s much easier to connect a storage when ScummVM’s built with SDL_net support, because then ScummVM runs a local webserver. In this case users are not redirected to ScummVM site from provider’s page — instead, they navigate directly to webserver’s page. No code typing is needed then, because ScummVM automatically gets it from user browser’s HTTP request. This webserver makes connecting a storage really fast and simple.

Another thing Cloud storage might be used for is games data download. Users can put their games into storage and then easily download on all their devices. A special «Download» button in Cloud tab opens Download Dialog, where users can select a remote directory to download and a local directory to download into. It shows a progress bar there and automatically tries to detect a game when download is complete. Users are also free to run download in background: no detection will happen, but a message will appear on the screen to notify them of finished download.

 

Both storage connecting and game downloading are shown in a video I’ve recorded. I’ve also posted information about my progress in the blog every week. Feature is documented on the wiki pages, with some diagrams included.

And, finally, we’ve decided that I should do a big extra task. Local webserver, which I originally proposed to simplify storage connecting process only, has been extended to be used for «Wi-Fi Sharing» feature. It means that while ScummVM’s local webserver is running, one can use browser on another device to navigate through directories, download files, create new directories or upload files!

Users can specify server’s port and their ’/root/’ directory within the same Cloud tab. Only files under specified directory and ScummVM’s saves directory are available, so users secure data is safe.

This week I’ve worked on some security-related updates for LocalWebserver. It now has a «minimal mode» — no Files Manager features are available, only redirect_uri. It’s used automatically in Storage Connection Wizard, and server goes offline the moment it gets the code.

Paths are now dealt with more strictly: «../» is forbidden to use, ScummVM has some «blacklisted» directories and user can define where the «/root/» is. Files outside that folder are not available through Files Manager. Plus, if no rootpath specified, «/root/» is not even listed — only «/saves/» is available then. GUI for changing rootpath is added to the same Cloud tab.

One other thing I did was rebasing. Travis checks PRs by merging with master and then building. Titanic, which wasn’t there when I started, was using OutSaveFile as if it’s typedefed WriteStream. But in my Cloud PR OutSaveFile is a real class (which starts saves sync when finalized). So, I had to rebase and add a simple fix in Titanic, and now Travis checks are passed again.

In order to update WAGE saves, I had to add pos() into WriteStream class. Managed to break a few builds that way.

Also, fixed a few crashes in WAGE games. One was because code tried to copy pixels outside of the Surface, and I added checks, so copied rectangle is clipped to always be within surface area. The other was because operator[] was used on an empty String, and now that code uses it only when String it not empty.

Well, this week I was mostly working on some fixes for Cloud PR, some minor warnings, TODOs. Updated GUI a little bit: added a «Paste» button in Storage Connection Dialog, so user’s clipboard could be automatically put in the 8 code fields. Dialog also contains a Container now, because buttons couldn’t fit in 320×200. Storage providers logos are now shown, which is just a nice thing to have.

Apart from that, I’m waiting for PR feedback, and thus have to do something while waiting. I’ve joined the ScummVM team and now working on WAGE engine. Well, the only thing I did for now was «porting» save/load code from existing Java implementation of WAGE engine into our ScummVM one.

In my previous post I’ve mentioned my Container PR — and voila — it was merged! So the first thing I’ve done this week was merging it from upstream’s master into my cloud fork’s master and using the Container in the Cloud tab. I’ve also fixed a few TabWidget’s height issues in that a bit later.

Box support was added this week, so the original plan to add 4 cloud services is accomplished. Box is a bit similar to Google Drive, as it uses files ids instead of paths. The other interesting thing about it is that it has yet another approach to file uploading, so I had to tune some things one more time to support this one too.

I’m not planning to add more cloud services yet. I was unable to even find out how to work with iCloud. OwnCloud uses WebDAV instead of OAuth2 + REST API and they are not going to implement one. I’m not sure how long will it take to add WebDAV support, so I’d be working on other things for now.

A few upgrades were made in the DownloadDialog and local webserver’s Files Manager. The first one now shows progress a bit more precisely because it’s calculated based on downloaded size, not the number of files downloaded. It also shows the downloaded size, total size and current speed there.

Files Manager now supports directories uploading (works in Chrome only, as that’s the only browser where this feature is implemented). I’ve also added code which should correctly determine server’s IP on Linux. Some minor improvements like file type icons were made:

Finally, I started working on openUrl and clipboard support. Windows, Linux and Android already support opening URL in the browser, OS X and iOS backends are on the way. I also have some Symbian code, but it could take some time to implement it properly.

Clipboard is already supported in SDL2, and I just started working on adding it. Right now it’s only Ctrl+V support for EditableWidget. We’re also thinking on adding selecting feature there, so users could also use Ctrl+C. I’m not sure how that should work on platforms like Android, where we usually don’t have keyboards attached. I mean, EditableWidgets are not Android EditText fields, so they don’t show a menu on long tap, which means we either implement it or make users suffer.

These two features are what I’m going to work on. Then I also should add some tests into testbed engine, and do some documenting/refactoring where needed.

This week started with the DownloadDialog, which allows users to download directories from their cloud storage onto their device. Navigating through the remote directories is a bit slower than browsing local ones, but it works fine. Download is done in the background. On the completion, OSD message is shown. If DownloadDialog was not closed, ScummVM automatically detects the game in the directory (the way it works in «Add Game») and suggests options. «Add Game» also checks whether you’re trying to specify the currently downloading directory — and notifies that this is forbidden. I’ve also recorded a video showing how storage connecting and game downloading features work:

(Source: https://www.youtube.com/watch?v=CFVPsv_deAo)

Then I started working on the next feature on the plan — Wi-Fi Sharing. Now there is a special «Run Server» button in the Cloud tab, which one can use to start local webserver. It also shows its IP there (works on Windows only yet):

I’ve added «/files» path to the webserver, so that’s where one can browse directories on the device. Clicking on a file causes its download, obviously. There are also two buttons: «Create directory» and «Upload files». One can’t upload directories yet, but multiple files uploading works fine.

I’ve tried downloading a 349 MB file from that webserver and it took me 16 seconds. That’s about 22 megabytes per second! Unfortunately, I can’t say the same about uploading. It took me 20 seconds to upload a 9,5 MB file and 11:15 to upload that 349 MB file. That’s about a 0.5 megabyte per second. Handling multipart/form-data POST is not an easy thing, and it doesn’t work fast in my webserver implementation yet.

What’s next? My Container PR is not merged yet and this week features might need some polishing after review. The next thing in the plan is to implement the fourth cloud support.

As I mentioned in the previous post, I had a few things to do this week: our university team was playing AltayCTF (we won the first place) and I had to move out of the dormitory. To compensate these days off, I’m working this weekend.

As planned, I’ve implemented clipping in all the other drawsteps and used these in all widgets. Now we can put any widget in the container and it would be clipped as it should be. More screenshots could be found on the special page.

Then I got all container-related commits in a new branch (because my separate container branch was based on cloud branch, as I had to test my container on Cloud tab) and issued a pull request.

I guess I’d spend some time polishing that when I’d get some feedback, but until then I’ve decided to test my cloud feature on Unix. All this time I worked on Windows and was using MinGW to check it could be built in Unix-like environment, but never launched that on actual Unix. I installed Kubuntu in VirtualBox and cloned my repo.

The first thing I found out is that our configure stops if there is no curl_config. I guess that’s not OK, because if there is no curl_config, we probably should build ScummVM without curl support (i.e. without clouds). Yet it works fine without SDL_Net installed, so my first built ScummVM was supporting clouds, but not local webserver — that’s how I checked that storage connection wizard works fine.

Then I installed SDL_Net and rebuilt ScummVM to check that the local webserver works fine too. It is. The saves sync wasn’t, though. I had to fix a few minor bugs, but that was easy. Now saves sync is OK too. I was lazy to edit .gitconfig or to setup a shared folder, so I just used ScummVM’s sync to upload necessary source files into my Dropbox 😀

When I’d be done with the container, I’d start working on cloud files management — some GUI dialogs which allow users to upload and download game data.