Category: Uncategorized

Project Goals

The main goal of this project was to add text-to-speech, or TTS, to a variety of ScummVM engines. TTS enhances the accessibility of games supported by these engines and helps language learners. Furthermore, adding TTS to more engines standardizes the implementation of this feature, meaning the inclusion of TTS will be more consistent across engines. Adding TTS to an engine entails adding responsive voicing of text elements such as dialogue, menus, user input, objects, and credits, as well as adapting TTS to user actions and the functionalities of the games themselves.

What was Done

The original goal of the project was to add TTS to 12 engines, but I had enough time to add TTS to an additional 3 engines. Therefore, I added full TTS support to a total of 15 engines over the summer. In addition, I added TTS to 2 engines prior to the official start of GSoC, and I helped fix a few bugs for one engine.

Current State

All of my pull requests have been merged, meaning 17 engines now have full TTS implementations. Games that use these engines should have comprehensive and usable TTS.

What’s Left

All of my project’s goals, as well as a few stretch goals, have been completed. Nonetheless, some engines that support a wide variety of games, such as SCUMM, may need additional testing by end users. Any required changes found from this testing should only be tweaks to behavior: the core of the TTS implementations themselves should be complete. In addition, there are still a few engines left that need TTS support.

Code

Most PRs are for adding TTS to certain engines, but some are fixes for previously merged code. The following is a list of each PR I opened:

1. TTS for Drascula: https://github.com/scummvm/scummvm/pull/6526
2. TTS for TeenAgent: https://github.com/scummvm/scummvm/pull/6566
3. TTS for WAGE: https://github.com/scummvm/scummvm/pull/6669
4. Fix for Russian TTS for TeenAgent: https://github.com/scummvm/scummvm/pull/6722
5. TTS for Cine: https://github.com/scummvm/scummvm/pull/6700
6. TTS for CruisE: https://github.com/scummvm/scummvm/pull/6710
7. TTS for Draci: https://github.com/scummvm/scummvm/pull/6742
8. Fixes for TTS for CruisE: https://github.com/scummvm/scummvm/pull/6752
9. TTS for MADE: https://github.com/scummvm/scummvm/pull/6779
10. TTS for ADL: https://github.com/scummvm/scummvm/pull/6785
11. TTS for Parallaction: https://github.com/scummvm/scummvm/pull/6795
12. TTS for Prince: https://github.com/scummvm/scummvm/pull/6807
13. TTS for EFH: https://github.com/scummvm/scummvm/pull/6820
14. Bug fixes for EFH: https://github.com/scummvm/scummvm/pull/6821
15. TTS for MM: https://github.com/scummvm/scummvm/pull/6835
16. Czech and Polish TTS for TeenAgent: https://github.com/scummvm/scummvm/pull/6867
17. TTS for SCUMM: https://github.com/scummvm/scummvm/pull/6856
18. TTS for AGI: https://github.com/scummvm/scummvm/pull/6862
19. TTS for Gob: https://github.com/scummvm/scummvm/pull/6880
20. TTS for Got: https://github.com/scummvm/scummvm/pull/6888
21. Fix for TTS for Got: https://github.com/scummvm/scummvm/pull/6900
22. TTS for Hugo: https://github.com/scummvm/scummvm/pull/6897

Challenges and What was Learned

Ensuring TTS compatibility across numerous games, languages, and versions that a single engine may support was sometimes challenging to comprehensively address. Furthermore, some engines handle their text almost entirely with game scripts, which makes it more difficult to narrow down where text is being displayed and how to feed it to TTS. To address these problems, I had to test thoroughly, think about limitations and features that differ between games or platforms, and employ strategies such as recreating click boxes to properly voice text as the user interacts with the game. Thus, most challenges with adding TTS revolved around compatibility and logical voicing of text elements.

I learned a great amount this summer about working with a team, understanding and modifying code written by other developers, and writing code that fits into an existing codebase. I am now more comfortable with navigating and working on larger projects.

Conclusion

I enjoyed working on my Google Summer of Code project, and I found it to be a very entertaining and rewarding experience. I am happy that I was able to contribute something to ScummVM, and I hope that it will be helpful to users.

I would like to thank my mentor, criezy, for kindly guiding me throughout the summer, and sev, for guiding me as well. I would also like to thank all members of the ScummVM team that I interacted with for their patience and assistance, and the entire ScummVM team for their work on this application.

Introduction

This week, I opened PRs for adding text-to-speech to Got and Hugo. They were relatively simple engines, and neither had any major challenges.

Got

I started this week by finishing TTS for Got. As previously mentioned, Got was quite simple for TTS: most of its text was easy to locate in the code, and text was rarely primarily handled by the game scripts. Thus, most of the game’s text could be voiced rather easily. For cleaner voicing, I included checks for when the score, jewels, or keys change, allowing their values to only be voiced when they change; delays as the credits are voiced to prevent them from moving too quickly for TTS to keep up; and voicing of dialogue as soon as it is started, so that it syncs well with the text while it appears one character at a time.

Nevertheless, one consideration for Got was the best means of cleaning text. In most cases, newlines break up sentences in dialogue, which results in choppy voicing if they aren’t replaced. However, some dialogue, such as signs, use newlines instead of punctuation to break up distinct sentences. If these newlines are replaced, TTS awkwardly pronounces everything as one quick sentence. My solution to this issue was to keep the newlines only if there are two or more in a row, as this is usually the case for sentences separated exclusively by newlines. Singular newlines tend to break up whole sentences, and thus should be replaced.

Ultimately, Got was one of the simpler engines that I’ve worked on. It required some care for properly cleaning up text and guarding against awkward voicing, but it otherwise offered few unique challenges.

Hugo

After Got, I worked on adding TTS to Hugo. Like Got, Hugo was a fairly simple engine for TTS. Its games are straightforward and similar to each other, which made it easier to implement functional TTS for all scenarios. Furthermore, it has few unique procedures of displaying text, so voicing most or all of its text only required adding TTS calls to a few methods. Nevertheless, Hugo required some consideration for its dialog boxes. Unlike most other engines that I’ve worked with, Hugo displays a considerable amount of its text in ScummVM dialog boxes, which have their own TTS. This means that opening or working with some dialog boxes, like the top menu, stops all TTS. Stopping TTS in this manner, however, can interrupt voicing of the sound setting, which is shown in Hugo’s score line. Voicing it as soon as it changes doesn’t work properly, as in many cases, the top menu is immediately opened again, thus interrupting the voicing. Therefore, my solution was to keep trying to voice the new sound setting for as long as the cursor is in a position to open the top menu. In this way, once the top menu finally fully closes and doesn’t open again, the sound setting will be voiced.

Other aspects of Hugo were rather simple. It mostly entailed implementing more TTS for the aforementioned dialog boxes, as their built-in TTS only voices text as it’s hovered over and only one line at a time, and it seemed to me that these boxes needed to be voiced as one clean paragraph or sentence and as soon as they appear; voicing scoring changes; and correctly queuing voicing when dialog boxes, user input, and scores are voiced, to prevent them from interrupting each other as dialog boxes appear.

In conclusion, Hugo was fairly simple. Because its games involved few different means of displaying text and simple controls, it didn’t offer that many unique challenges.

Conclusion

This week, I finished adding TTS to Got and Hugo. Both of these engines were simple to work with, and they didn’t have any major caveats or challenges.

Introduction

This week, I opened a PR for adding text-to-speech to Gob, my first stretch goal. It was an engine that posed a few more challenges than usual, but that was entertaining to work on. I also started early work on adding TTS to Got as my next stretch goal.

Gob

Gob was a rather interesting engine for adding TTS to, primarily because of how it handles text and buttons. Unlike many of the engines that I’ve worked on, hotspots and text are inherently separated in Gob. This meant that there was no easy way to retrieve the text of a button or menu as the user hovers over it, which makes it more difficult to voice. Furthermore, without knowing which text is part of a clickable hotspot and which text is not, text that shouldn’t be voiced the instant it appears will be voiced. My solution to this issue was to maintain an array, _hotspotText, which holds printed text and their positions. Then, as hotspots are added, their collision boxes are checked against those of the text array. If they intersect, the hotspot text’s collision rect is expanded to that of the hotspot. Through this method, hotspot text can be voiced as the user hovers over it, while most other non-hotspot text is voiced by a separate method to make sure that all text on screen is voiced. Nonetheless, refining this strategy took some time, as I encountered issues with properly removing elements from _hotspotText when text is removed from the screen, which I resolved by deleting unassigned hotspot text in certain methods and removing pieces of assigned hotspot text when their corresponding hotspots are removed; the cursor being instantly placed over a hotspot in some cases, interrupting the speaking of non-hotspot text, which was resolved by queuing hotspot text when a non-hotspot piece of text is voiced; and having to move hotspot text in o1_copySprite, as the text itself is sometimes moved there, and not moving the hotspot text with it results in a disjointed collision.

Another problem with Gob was timing of dialogue. From what I could tell, there are no unique indicators for when dialogue starts, progresses, or ends in Gob, which causes issues during dialogue interactions, as they may move too fast and interrupt voicing. In addition, dialogue appears to be displayed as TOT text, but so are object names, which means that TOT text can’t be set to always interrupt or always queue without causing awkward voicing. My solution was to only queue text if the user isn’t hovering over a hotspot, as in most cases, this means that it’s text that should be queued. After resolving this issue, most of Gob’s other problems – including needing to improve voicing of Ween’s notepad, which awkwardly tries to voice one character or section at a time as the user types, by only voicing it when it first opens – were rather straightforward.

Ultimately, Gob was an interesting engine because of how it handles hotspots and dialogue. It required different solutions from previous engines, and I’m quite happy with the result, though there may be more problems to resolve in the future.

Got

After Gob, I started work on adding TTS to Got. So far, Got seems very simple for TTS. Most timing and menus do not seem to be handled by game scripts, which makes it easier to locate where to voice text and to implement clean voicing. At the moment, nothing difficult has emerged, but this may change as I add more TTS to the engine.

Conclusion

This week, I added TTS to Gob, and started work on adding TTS to Got. Gob had several considerable intricacies, while Got appears to be rather simple so far. Next week, I’ll be continuing work on Got, and perhaps starting work on adding TTS to another engine if time permits. However, my semester starts next week, which will limit the time I have to work on my project.

Introduction

This week, I opened a PR for adding text-to-speech to AGI. It was the last engine marked on my proposal, which means that I have now made PRs for every engine that was planned for my project. This is a major milestone, and gives me time to work on Gob as a stretch goal.

AGI

I spent much of the week working on AGI, which was fortunately a rather simple engine. While it supports many different games and fangames, most of these games seem to function very similarly: there may be introductions or other screens that display text using TextMgr::display, with most other text being displayed through popup windows, typed commands, or menus. Thus, AGI was fairly straightforward, requiring only a handful of TTS calls in key methods, as well as some consideration for when to voice the status menu and clock, which are always visible – I settled for voicing one or the other primarily when the game is loaded, the status menu changes, and the clock is enabled, to avoid voicing them too frequently – and when to stop TTS.

The major concerns with AGI were how it handles timing and TextMgr::display across games. Some games appear to use timer variables to signal when it’s time to change the text on screen, while others use when a sound ends. Therefore, from what I could tell, there was no consistent way to predict when text will change, which made it difficult to delay text changes until TTS is finished. I eventually settled for queuing text and delaying room and window changes until TTS finishes, which seems to be enough to allow TTS to voice everything displayed on screen in a reasonable manner. In addition, TextMgr::display differs between games: some call it only once, while others call it every frame, which necessitates keeping track of the previously said text to avoid speech loops. However, much of the text passed through this method is displayed in sentences or chunks, rather than all at once in a single call. This makes simply tracking the previously said text fail and results in awkward, choppy voicing when a sentence is broken across lines. My solution to this issue was to combine the text passed to TextMgr::display in a _combinedText variable, and then voice this when the game script returns or halts. Using this method, paragraphs and blocks of text are spoken cleanly as one sentence, and the previously said text can simply be set to this combined text to prevent speech loops. I also decided to add in newlines between these pieces of text if they aren’t displayed on subsequent rows, since in most cases, pieces of text that aren’t in subsequent rows shouldn’t be voiced as one quick sentence.

In conclusion, AGI was a fairly simple engine, since its games didn’t have much complexity. Most of the work went into handling timing and subsequently displayed text, as these factors can differ significantly between its many games. Otherwise, AGI fangames seem to have little variation in how they display text, which simplified work on it, though there’s a chance that games that aren’t fangames may function differently, as I mainly worked with fangames.

Gob

After opening a PR for AGI, I started work on Gob. So far, Gob doesn’t seem too complex: its text seems limited and appears to be displayed through only a few select methods. However, certain games, like Adibou 2, separate text from many of their buttons, which requires a technique to sync them. So far, I’ve decided on a method of checking whether the displayed text intersects with a hotspot, then expanding the collision rectangle of the text accordingly, which seems to work fairly well. Nonetheless, I would like to see if I can make this solution more robust for better compatibility, as it seems that Gob supports a variety of games with many differences between them.

Conclusion

This week, I opened a PR for my last planned engine, AGI, and started work on Gob as a stretch goal. I plan to have a PR up for Gob by the end of this week, which may possibly give me enough time to start another stretch goal.

Introduction

This week, I opened a PR for adding text-to-speech to SCUMM, which was an interesting engine to work on. I’m quite happy with the result, though with the wide variety of games supported by SCUMM, it may need more work in the future.

SCUMM

Most of my week was spent on adding TTS to SCUMM, an engine that was neither especially difficult nor especially easy to work on. On one hand, SCUMM games are very similar to most of the games I’ve worked on before. Most of them have few menus and rather simple means of displaying text, unlike engines like Efh or MM. Thus, creating a user-friendly TTS system was fairly easy for SCUMM. In addition, finding where text is displayed wasn’t difficult: most of it goes through printString and drawString, with actor speech being displayed with displayDialog. Furthermore, most of the GUI controls and buttons come with labels built into them, which makes voicing buttons when they’re hovered over a very simple process. As such, much of the work for SCUMM involved simply adding TTS voicing calls to these functions, while accounting for situations such as subtitles for voiced dialog and the need to delay the disappearance of text until TTS is finished.

On the other hand, however, SCUMM is larger than some of the engines I’ve worked with previously, and it supports many different games. This means that compatibility issues are a recurring problem, as what works for TTS in one version of the SCUMM engine may not work in others. A good example of this is verbMouseOver. When I first came across this method, I thought that it would be a great place to voice verbs when they’re hovered over. However, SCUMM version 5 – or at least, Indiana Jones and the Fate of Atlantis – doesn’t seem to reliably use this function for detection of hovering over verbs. In addition, while some games only call drawVerb once as a verb is hovered over, games like Fate of Atlantis call it every frame. Thus, to try to voice verbs as robustly as possible, I decided to add code to drawVerb, which most SCUMM versions seem to go through for verb drawing, to check whether the current verb is highlighted before voicing it, a strategy that seemed to work for many games. There were other compatibility issues that I had to resolve as well, such as SCUMM versions 0, 1, and 2 using custom text encodings that needed to be replicated; drawVerb sometimes being used to print strings that aren’t verbs, requiring them to be voiced even if they aren’t highlighted; and Passport to Adventure having a special help menu with buttons that aren’t considered GUI controls, which required a means of storing the text for each button and detecting when they’re hovered over.

Another concern was that SCUMM versions 7 and 8 use their own methods for displaying text, though they were fortunately similar enough to those used by earlier versions that it wasn’t too difficult to voice them. Humongous Entertainment games also seem to have different means of handling text, but because they don’t have much text in the first place, I didn’t have to worry as much about them. Thus, most of the compatibility issues were in earlier SCUMM versions, with later ones having fewer problems.

Ultimately, the most difficult component of SCUMM was the wide variety of games supported. Each version has its own ways of handling text that have to be considered, requiring careful thought about the best places to voice or stop text. I’m fairly happy with my implementation of TTS for this engine, but because of its many games, there may be some oddities that will need to be resolved.

Conclusion

I opened a PR for SCUMM this week, which was an interesting engine to explore, due to its greater size and variety of versions. I also revisited my MADE PR, an engine that had its own compatibility issues, with text indices varying across game versions, that should be solved now. Next week, I’ll be working on AGI, the last engine listed on my project.

Introduction

This week, I opened a PR for adding text-to-speech to MM, which proved to be one of the more challenging engines of my project. The process of adding TTS to it took up most of the week, but I’m quite satisfied with the result. Nonetheless, I do have concerns about its compatibility across versions and about the possibility of missed menus, so there may be more work for it in the future.

MM

I began work on MM last week, when I finished most of the difficult work for it, mainly in the form of coming up with a strategy to clean and parse text and voice buttons. However, after that was finished, there was a great amount of time-consuming work to be done. Might and Magic games – or at least, Xeen games – have a relatively high number of menus and user input. In general, each menu has its own dialogs file, which handles text, graphics, and menu input. On the positive side, this meant that it was quite easy to find text for menus. On the negative side, there is practically no pattern among this text, and it’s very often in an order that doesn’t match the order that it’s displayed on screen. Button text may be listed first, or it may be listed somewhere closer to the end; labels and their values may be listed next to each other, or they may be listed in completely different blocks – for instance, “Might, Intellect, Hit Points, 9, 5, 15” instead of “Might, 9, Intellect, 5, Hit Points, 15”, which would be much easier to voice; and button text may be listed in the order that the buttons themselves are displayed, or in a completely different order. This lack of pattern applies even across similar situations: for example, locations order button text at the end, except for banks, which order them at the beginning. Such a problem occurs because text is displayed using a wide variety of formatting characters for factors such as position and justification, but it seems that they can’t easily be used for voicing, since these traits alone are usually not predictive of the best voicing order.

This lack of standardization prompted several concerns, because it meant that voicing the text directly would result in awkward ordering that may not make sense. It was possible to simply let FontSurface::writeString – which I discovered last week – handle the voicing, as it’s seemingly used for all displayed text. Such a strategy would be easier to implement, but wouldn’t be nearly as responsive or logical, since no buttons would be voiced with input and text would often be voiced out of order. However, some text is in the correct order, and so it makes sense to use this method without changing anything in a few cases. Thus, I decided to only override voicing in FontSurface::writeString individually for each dialog where it’s necessary, allowing for much smoother and cleaner voicing and providing a means of always voicing any text that I may have missed.

Voicing this text with different code for each dialog was rather time-consuming. It required looking through each menu, seeing how it orders text, and splitting it up accordingly, with each dialog requiring a different process. I decided to split the cleaned text along newlines using a method called getNextTextSection, as it was the most reliable means of getting pieces of text, and made voicing each dialog only a matter of switching around getNextTextSection calls. Unfortunately, this still required a fair amount of work, especially because of the occasional occurrence of oddities, such as the party screen having twice the text after leaving the character creation screen, which causes the first repetition of text – an outdated version – to be voiced. Fortunately, this process of tailoring each dialog wasn’t particularly difficult, but it did take a while.

Thus, after deciding on a procedure for voicing buttons – which involves a separate array of strings, with each applicable button having an index that corresponds to a string in this array – and for cleaning and splitting text, most of the work for MM was simply time-consuming. Nonetheless, as I added support to more menus, I did have to consider whether my methods were optimal, and I went over several possibilities. I considered storing text directly in each button without using any index variable, but this would be unreliable: the strings for each button are rarely in the same order as the buttons themselves, which would require hardcoding indices (for example, _buttons[5] = getNextTextSection(...)). This strategy would break completely if, for some reason, the order that buttons are added to the _buttons array is changed. I also considered reordering the buttons themselves, but after this caused the wrong images to be rendered on the character info sheet, I decided it was too dangerous. In contrast, I highly doubt that the order of text taken directly from the game files will ever change, which makes the original method of keeping the buttons and their text – populated by hardcoded splitting – separate the safest option I could think of. Similarly, I considered whether my current method of passing a string to FontSurface::writeString to be cleaned was optimal, as it restricted TTS input to whenever this method is called. Perhaps catching the text earlier in the code could remove the need to split the string, as its fields would already be separate. Indeed, input to this method in nearly all cases can be found earlier, with many of its values split from it. Unfortunately, most of the text directly from the resources is already in a large block, which necessitates sorting anyway, but this time with the need to clean it as well (though I could perhaps recreate the cleaning process that exists in writeString, I wasn’t certain if I wanted to run the risk of missing an important step, and it seemed unnecessary). I also considered using the string’s formatting characters, but this seemed unreliable, as traits such as exact text position aren’t consistently indicative of the best voicing order, and it seemed easier to clean first and split later. Thus, I settled on many of my initial strategies, though I may change them in the future if a better idea arises.

Ultimately, working on MM was more challenging than previous engines, due to its immense amount of text, menus, and input. Its text is also full of formatting characters that shouldn’t be voiced, requiring a lot of cleaning, and its lack of standardization demanded a different process for each menu. I had to think of several different approaches, and pick which one seemed to be the most robust and effective. It was a very interesting and entertaining experience, and I’m quite happy with the TTS functionality for this engine. However, I’m not certain how well it will perform with Might and Magic 1 or other languages, or if my methods are optimal, and thus there may be more work to be done in the future.

Conclusion

This week, I finished TTS for MM, a larger engine that required more work. It was a rewarding experience, and I now have a PR opened for it. Next week, I’ll be working on SCUMM, and perhaps AGI if all goes well. These are the last two engines planned for my project. I hope to reach a stretch goal, but that depends on whether SCUMM and AGI have any surprises like MM did.

Introduction

This week, I mainly worked on adding text-to-speech to Efh, which posed a few unique challenges. I’ve opened a PR for it, and have begun work on adding TTS to MM.

Efh

Adding TTS to Efh was a little different from my experiences with other games. In most of the engines I’ve worked on, text appears on screen one piece at a time. Different sections of text rarely appear consecutively on screen, and if they do, they’re often part of the same string or part of a clickable menu. Escape from Hell, however, handles text differently, primarily because it has many menus full of numerous different pieces of non-interactable text, meaning it should be voiced as soon as it appears rather than as the user hovers over or clicks it. This text is usually split into many different strings, and may be displayed in the code in an order that doesn’t make sense for TTS – for example, the text that appears at the bottom of the menu may actually be displayed first in the code, so voicing it in the order that it displays, which is the easier route, would be awkward. In addition, very little text is displayed or fetched in methods that are only called once. Such an issue is common in many engines, but is further concerning in Efh because of the great variety of text: tracking the previously spoken text in a string, which is what I usually do in situations such as this one, won’t work easily, because each menu has many different strings of text. Fortunately, my solution to this problem was rather simple for the most part, as it involved keeping track of a “say menu” flag. The flag is turned on after user input, then turned off the instant the text is spoken. This worked for most menus, but didn’t work perfectly for the lower status menu that stays visible throughout most actions, as the display of the menu could consume the flag for other menus or be voiced in situations where it wasn’t appropriate. Therefore, I introduced a separate flag specifically for this menu as a solution.

After solving this problem, most of Efh was straightforward. Because a significant amount of the game’s text is hardcoded, it was very easy to find the majority of it. From that point on, most of the work on TTS for Efh involved delaying text on screen during combat until TTS finishes; speaking the user’s choices in menus and combat; and checking for when it’s appropriate to stop or interrupt speech, as the same methods may be used for different situations, some of which shouldn’t stop speech.

Ultimately, Efh had a lot of little details that were important to consider for the player’s ease of use, such as the best time to voice the player’s inventory or the best moments to interrupt text. Its abundance of text in the form of menus necessitated a slightly different approach from other engines, which was fun to explore. I was also able to identify and tentatively solve a few bugs in this engine while I tested my TTS implementation.

MM

After Efh, I began work on MM. Like Efh, MM games have an abundance of text in their menus, which is oftentimes displayed out of order. Fortunately, most of its text seems to be displayed in methods that are called only once, meaning that I rarely have to consider repeat voicing. Unfortunately, at least in Xeen, it has a wide variety of menus that each have their own specific code and considerations, including how buttons are displayed, how text is ordered, and how many times the display methods are called. In many cases, the text for an entire menu is in one string, which would usually be acceptable, but most of the menus display attributes and their values in separate blocks – for example, one string may be “Might, Intelligence, Gold, 10, 9, 800”, instead of “Might 10, Intelligence 9, Gold 800” – that makes voicing it directly awkward for the user. In addition, its text is full of different characters used to define traits, including the text’s coloration and alignment, which means it has to be thoroughly cleaned before TTS can voice it. I’ve found that FontSurface::writeString is responsible for cleaning and displaying most text, which makes it a great place to start, but it doesn’t always work to voice the text from there, as text may be displayed out of order. Thus, for situations such as these, I’ve decided on passing a string to it to be cleaned, then storing that string to be voiced at a later time. In each menu, this string can be split up and stitched back together in a manner that makes more sense for TTS, primarily by combining attributes and their respective values.

Buttons are another matter for this engine, as most buttons have text that is inherently separate from them in the code. In most menus, the text for buttons is combined with the text for the rest of the menu. To allow for more interactive voicing of buttons, I’ve currently settled on taking the cleaned text for the buttons, splitting it along newline characters, and storing this text in an array. Each button then has an integer index that corresponds to a string in the array. Putting the strings directly in the buttons was a possibility that I initially considered, but because the display order of the buttons often doesn’t match the order of the text, I found that it was easier to do it this way and not have to worry about reordering the divided text. However, TTS for MM is still in a relatively early stage, and I may change these methods if I find better solutions.

So far, MM’s variety of menus and buttons require a fair amount of TTS work. I’m looking forward to continuing work on it next week.

Conclusion

During this week of GSoC, I finished TTS for Efh, and began work on TTS for MM. It was an entertaining week, as I had a chance to work on a different type of game, which provided unique challenges compared to the games I’ve worked on up to this point. Next week, I’ll be continuing work on MM, and hopefully beginning work on SCUMM.

Introduction

This week, I worked on adding text-to-speech to Prince, which was a fun engine to work on. A PR has been opened for it, though more work may be needed in the future. In addition, I began work on adding TTS to Efh, which I plan to finish next week. My ADL PR was also merged this week, and I updated some of my earlier TTS PRs.

Prince

Most of my week was spent on adding TTS to Prince. Fortunately, the Prince and the Coward has no text in the form of images that I could find, which meant no hardcoded text was needed for this engine. However, Prince had some complexities in how it displays text. Its text is displayed rather simply in methods such as checkMob and printAt – which are either only called once or have means of tracking text changes by checking changes in indices, meaning there is no need to track the previously spoken text for this engine – but there are several exceptions to consider. For example, how the text in printAt should be voiced depends on several factors, including slot and location: slot 9 is generally subtitles, while slot 10 is often either subtitles or, if the location is the map, map text. Differentiating between these types of text is important because of the presence of the dub, which necessitates splitting TTS into several categories of subtitles, objects, and missing voiceovers. Since the Polish, German, and Russian translations all have dubs in their languages, subtitles should almost never be voiced for them, while the English and Spanish translations, which lack dubs in their languages, should only have subtitles voiced if the dub is muted. Thus, a fair amount of consideration had to be given to splitting up the text.

Prince also had a few other key exceptions. For one, when I worked on voicing the text of objects when they’re hovered over, I initially thought to use the _selectedMob variable, which keeps track of the mob that the player is hovering over: if, in checkMob, the selected mob doesn’t match the current mob number, then the user must be hovering over a new mob, meaning that the text should be voiced. However, I found that left clicking resets _selectedMob, which results in the text being awkwardly voiced again even though it hasn’t changed. This was easily fixed by introducing a new variable that tracks the selected mob, but is not reset upon left clicking. In addition, I worked on speaking missing voiceovers; solving the issue with the gambling merchants in the town, which constantly talk even as the player interacts with the environment and thus interrupt other TTS, requiring an exception for them that only voices their text if the player isn’t in dialog; and creating several custom encoding tables.

Another significant problem was changing voices. There appears to be no easy indicator that differentiates speaking characters in Prince: text colors are shared across several characters, mob numbers are not unique to certain characters and are instead specific to each location, and dialog seems to be controlled almost entirely by game scripts without any key character indicators. Therefore, my solution was to use a combination of several factors to determine the voice. The text color is enough to differentiate some characters, as the color is sometimes unique. For characters that share text colors, I opted to also check for the location number, since most characters don’t move locations, and those that do can be a catch-all for cases when the location number doesn’t match that of other characters. However, I found that this didn’t work in a few specific scenarios, such as the tavern with Arivald and the bard, who both have the same text color and are in the same location. For such exceptions, I decided to check for the mob number as well, as it differs between them. The result is different voices for each character, though I do wonder if there may be some other cleaner indicator I could use.

Ultimately, Prince was an entertaining engine to explore. It was neither particularly difficult nor particularly easy, as it had its own unique set of challenges, but none that were daunting.

Efh

After opening a PR for Prince, I started work on Efh. So far, Efh seems fairly simple, as much of its text is directly hardcoded, making displayed text very easy to find. However, the fact that its menus display many pieces of text at once every frame is different from most of the engines I’ve worked with, though I’ve currently solved the issue with a simple flag that toggles on after user input and is then toggled off after voicing occurs. Aside from that, there doesn’t seem to be much complexity with Efh, though I still have a fair amount of text left to voice, since I need to account for user input and ease of use.

Conclusion

During this week of GSoC, I opened a PR for adding TTS to Prince and started work on Efh, as well as updated some of my earlier PRs. It was an interesting week, since I enjoyed Prince. Next week, I’ll be continuing work on Efh, and possibly beginning MM if all goes well.

Introduction

During this week of GSoC, I opened PRs for adding text-to-speech to ADL and Parallaction. I thought that these engines would take me longer than they did, but neither of them were particularly challenging, which was a pleasant surprise.

ADL

I spent the beginning of this week finishing TTS for ADL. Fortunately, it needed very little additional work from last week: I only needed to add TTS to a few extra key presses and clean up the text, primarily by removing dashes that could interfere with voicing. Since ADL’s games are so simple, there wasn’t much to do with this engine, and I was able to make a PR early in the week.

Parallaction

After ADL, I worked on Parallaction, which has a level of complexity that is comparable to that of other engines I’ve worked with. To begin, Parallaction’s text is displayed through methods that are only called once, instead of every frame like most of the engines I’ve worked on. As a result, I didn’t have to track the previously said text for this engine, and it was rather simple to voice the text – although in certain cases, such as labels, I had to store the text to be voiced later, as all possible labels are initialized at once, and voicing them upon initialization would result in a great amount of invisible text being voiced. I was glad to find that voicing the text in these methods covered almost all instances of text, including dialogue options when hovered over.

Nonetheless, Parallaction did have a few exceptions to consider, mainly in the form of its credits and introduction. Most of the text in the introductory cutscene and the credits is in the form of an image, which required finding a good place to detect when these images are rendered. For example, the second and third opening credit lines appear after an animation plays, unlike the first line, which appears the instant the location switches. Therefore, I opted to voice this text in the on instruction opcode, as it is executed when these animations finish. A similar problem emerges with the end credits, which are in the form of images that slowly scroll upward. Trying to voice all of them the instant the credits begin results in the TTS speaking too quickly and being poorly synced with the actual credits. To fix this issue, I found that the instruction opcode inc is executed roughly when another credit moves onto screen. Thus, after voicing the first few credits at once – since inc isn’t executed for these – I voiced the credits one at a time in this opcode, resulting in much cleaner voicing. Finally, I resolved another syncing problem where the test results, displayed after picking a character, disappear too quickly for the TTS to keep up by delaying the loop that controls the location change while TTS is speaking. A similar strategy was used for some of the opening credits. The result is TTS that is better synced to the text for certain images and situations.

Another interesting task for Parallaction was switching voices for each character, which was doable because each character is differentiated by their .talk file. In most cases, a simple array with each file name was enough for switching voices during dialogue. However, some dialogue interactions, such as the one between all three playable characters at the end of the game, have more than one character per .talk file, differentiated by the mood value. This was resolved by updating the array to have extra entries, and adding the mood as an offset to the index for these special cases. Beyond that, the TTS voice needed to be switched when exiting a slide, showing a location comment (which I decided to use an extra “narrator” voice for), and switching characters, allowing for more dynamic character voicing.

Ultimately, Parallaction was an entertaining engine to add TTS to. It was mostly simple, with its exceptions – such as text in the form of an image, unique means of selecting a language, and password inputs – being not too hard to handle. Nonetheless, it’s currently only been tested with Nippon Safes, Inc. From the code, the Big Red Adventure seems to handle text similarly, allowing me to add tentative TTS to it, but I’m uncertain if it will work as well. Some translations also need verification.

Conclusion

I think that this week of GSoC was fairly successful, with PRs opened for adding TTS to ADL and Parallaction. ADL was easy, while Parallaction had very few surprises, making them both simpler than I expected. Next week, I’ll be working on adding TTS to Prince, which I’m looking forward to exploring.

Introduction

During this week of GSoC, I mostly focused on adding TTS to MADE, with some work on ADL as well. A PR has been made for MADE, though there may be more work for it in the future.

MADE

Most of my week was dedicated to adding TTS to MADE. MADE – or at least, Return to Zork – offered many new challenges. For one, rather than displaying text one piece at a time, like I’ve seen in most of the engines I’ve worked with, Return to Zork displays several pieces every frame across multiple channels. This meant that I couldn’t simply track the previously said text in a single variable to avoid speech loops, as it would be repeatedly overridden. To solve this problem, I tried several different approaches. First, I tried finding the function that actually sets most of the text, which I found in the form of sfReadMenu. This function does set the text one piece at a time, which would be good for voicing. However, the issue with it is that sfReadMenu is called several times when a new scene loads, even when the text that it’s setting isn’t visible, resulting in extraneous text being voiced. I tried to check for any flags that could identify this text as invisible, but I didn’t find anything good to use, so I scrapped this idea. I then thought of tracking several different previously said texts and their channels in an array, but this seemed unnecessarily cumbersome, especially because the channels are often changing. Eventually, I settled on adding a variable for the previously said text to the individual channels themselves, and using this variable to check if text should be voiced. This worked quite well in most cases, and after accounting for exceptions where the spoken text should be queued, it resulted in good TTS for much of the game.

MADE, however, had a few additional problems. For one, it seems to handle hovering over objects completely within the game scripts, which means that there’s no easy place to detect when the cursor hovers over buttons in Return to Zork’s save/load screen. After trying to search for possible conditions I could use, I eventually decided to recreate the click boxes for the save, load, and cancel buttons, as well as the text entry boxes for new save slots. Fortunately, none of the text for these buttons is in the form of an image, which meant I could find the IDs for each individual object or menu that displays them and use these IDs to get their text, a strategy that should work across languages.

Another issue arises with the tape recorder in Return to Zork. The text for the name and numbers on each tape recorder entry are displayed within channels, but the text that corresponds to these pieces – such as “name” or “trk” – are in sfDrawText, which results in awkward, out-of-order voicing (for example, when the track is 001 and the name is Wizard Trembyle, TTS would say “trk, name, 001, Wizard Trembyle”). Fortunately, this could easily be avoided by not voicing these pieces in sfDrawText, and instead fetching this text when voicing the channel text, resulting in cleaner voicing (such as “trk: 001, name: Wizard Trembyle”). Unfortunately, the tape recorder is interactive, which further complicates matters. Voicing each piece separately can result in issues if the user switches between entries where a number is the same, such as one entry having max track 001 and the other having max track 001, since the previously said text won’t change and thus it won’t be voiced at all. Getting around this involved voicing all pieces – name, track, and max track – whenever the name changes, as it should be unique for each entry, alongside voicing the track individually in specific situations to account for the user switching tracks. Voicing the time was another matter: if it’s voiced whenever it changes, then the TTS system will repeatedly try to voice it as it ticks upward, resulting in awkward interrupts or unnecessary lag. I eventually decided that it was best to try to voice it when the sound clip ends, so it’ll only be voiced at the moment it stops moving. These techniques involved keeping track of flags and the status of the previously said variables, since they’re modified and retrieved each frame, but it seems to have resulted in more responsive voicing of the tape recorder.

In conclusion, MADE was a little more challenging than expected, since Return to Zork has several simultaneous channels and handles much of its logic within the game scripts. Nonetheless, it was a fun experience adding TTS to it. It seems finished now, though it remains to be seen how well it will handle different games or translations, and it has a few hardcoded translations that need verification.

ADL

ADL has seemed quite easy so far. As an engine for rather simple text-based games, almost all of its text is handled by Display::printString and Display::printAsciiString. It doesn’t even require tracking the previously said text, since these methods to display text are called only once, and there are no buttons that require responsive voicing. Thus, I think I’ve almost finished TTS for it, though challenges may arise later.

Conclusion

Over week 4 of GSoC, I’ve completed more TTS implementations, with MADE finished and ADL mostly finished. MADE offered the most challenges, due to its different ways of handling text, but its lack of text in the form of images was welcome. It’s been an interesting week, and I’m excited to explore more engines. Next week, I’ll be looking to finish TTS for ADL and begin work on TTS for Parallaction.