No Cigar

As time to the deadline winds down, the pressure is building up. Since my focus has been on programming and dealing with the outstanding issues, I have neglected updating this blog as frequently as I would like.

As it stands, with the latest update, I believe the issues with the update cycle and timing system should be resolved. The previous revision had several problems due to the patterns or macros terminating prematurely or not stepping properly. With the revised system, song playback is handled as follows:

  • The TFMX files are loaded with a separate functions to load the data and samples. Once loaded, you call playSong() to start playback of a particular track. This loads the track data, starts PAULA and the interrupt cycle.
  • On the first interrupt, the trackstep data is read. If the trackstep line specifies a command, it is processed and the trackstep proceeds to the next line on the next interrupt. If the trackstep specifies patterns in one of the eight tracks, a flag specifying that a pattern is on is set to the specific track. The trackstep then cannot advance until all of the patterns have finished playing.
  • For the patterns that are on, one pattern command is processed per interrupt. In the case that the pattern specifies a note/macro, the pattern controller goes into a loop to process the macro until the macro specifies a wait which will cause it to hold until the next interrupt.

With the new system, pattern loading and macro loading is handled by separate functions. With this it is possible to independently load patterns or macros for testing purposes. I’m using this to my advantage and have designed a function to individually test macros for playback as is done in the original TFMX editor itself. Since I have been focusing on resolving the issues with the update cycles and timing system which have resulted in a total overhaul, I have not been able to successfully implement all the effects since they are inseparably tied to the timing system. With the test macro functionality in place, I will be able to finish implementing the effects and easily test them by playing back each of the Monkey Island macros.

As for the external interface to interact with; as stated, you simply make calls to individual load the data and sample files (these functions will be changed to accept parameters). You then simply call playSong to play one of the individual tracks. The track will automatically stop when complete or you can call a stop function at anytime to terminate the song prematurely. With these simple functions in place, it will be easy to add code to Monkey Island engine to playback TFMX audio. Hopefully, once the remaining effects are in place, TFMX audio will be ready for prime time. I hope to have them all in place over the next week and then proceed with the final step to linking to the Monkey Island engine.

Quick Update on Sample Playback

So the good news is that samples now playback like properly audible instruments. There is no longer a jumble of static and it turns out the instrument information is indeed stored as-is in signed 8-bit PCM samples. Lacking more information or documentation, I was initially unsure if the samples were stored with some sort of encoding scheme causing them to not playback correctly. This is not the case and after many headaches I’ve realized the problem lies in the timing system. I have to re-investigate some of the information presented regarding tempo numbers and interrupt speeds to track down why this is the case. As predicted, the issues with the timing system is still the major obstacles holding me back.

In the mean time, while I work on resolving this problem and possibly restructuring the timing system and update cycle, I will continue work on implementing the needed pattern and macro effects in an effort to have them ready for the end of the week.

The Big Push

It has been quite some time since my last update. Since the midterm evaluation period ended, I’ve been working on implementing the sample loading and playback and haven’t taken the time to post updates. It has been problematic to say the least. I can say that TFMX playback is functional… if you consider static noise to be music. As the big push towards the final deadlines and weeks of coding looms, I’ve devised a restructured plan to help myself stay on course and hopefully make up for my original schedule, which has pretty much gone out the window.

  • Objective: Implement sample loading and playback.
    This is the most critical portion of the entire TFMX project. The samples are stored in a separate smpl.* file from the rest of the TFMX data located in mdat.* and loaded via the macro commands tied to each note. They are then played back according to certain parameters set in the note and macro data. I’ve been working on implementing the specific macro commands which relate to sample playback but some issues are still in the way.
    Obstacles: The sample loading is relatively simple and has already been taken care of. There might be some useful optimizations or modifications to the loading structure that will take place later, but the current system of loading the whole sample file into memory will work for now. A handful of the macro commands relating to sample playback have also been implemented. Problems are occurring in the playback of the samples which could be related to speed or timing issues. I am working to solve these problems and also improve some of the other mechanisms related to the timing system. Completion of this task will solve one of the major lingering problems and allow me to finalize the TFMX system.
    Estimated completion time: Friday, July 25th.
  • Objective: Implement effects commands.
    After the sample playback is in place, I will implement the commands which apply effects to playback. These include note transposition, portamento, vibrato, envelope, and volume slides.
    Obstacles: Some of these effects introduce complex problems. The ones related to period manipulation will be more difficult to handle and will likely require some restructuring of playback routines to allow for the effect manipulation to take place. I anticipate there will be some problems here but that I will be able to rely on code used in the other tracker players to resolve many of my issues. I will also focus on only initially implementing the effects which are nessecary for Monkey Island playback.
    Estimated completion time: Thursday, July 31st.
  • Objective: Optimize and build Monkey Island interface
    Another crucial task coming up will be to optimize the code, fix any memory problems, and try to reduce the memory burden as much as possible. I will implement any withstanding pattern/macro commands and resolve any bugs or little quirks related to the Monkey Island TFMX files. The next step will be to implement the interface for the Monkey Island game to have access to the TFMX engine and playback the required audio during game play. At this stage, I’m sure other developers will have many recommendations for improvement or optimization as well.
    Obstacles: This should be a straight-forward cleanup task that should bring to light any remaining issues or problems. Building the interface for Monkey Island should also be simple as the I believe the playback routine code is already partially implemented. I believe enlisting the help of some of the other developers’ expertise on memory issues should help optimize the engine as well.
    Estimated completion time: Monday, August 4th.

At that point, we will be awfully close to the closing time so I cannot realistically say that I will a functional MaxTrax player on the table for the final deadline on August 18th. I will definitely begin working on it as soon as possible and continue to work on it until my commitment is completed, even if that takes us past the final deadline. I will also continue to improve the TFMX engine after the deadline as well. Right now my priorities are on finalizing TFMX and completing the required testing and debugging. After that is complete, I will reevaluate the goals for a MaxTrax player.

Objective 2 & 3 Update

The pattern and note reading functionality has been implemented as described in the previously laid out objectives. The pattern loading is tied to each track. Each track contains a pattern structure which is initialized when a new pattern is loaded. When the pattern is loaded, it is processed on each interrupt call. The pattern handling looks for commands or notes. The simple pattern commands involving sequencing and looping have been implemented. The more complex pattern commands handling note manipulation still need work. The note reading is primitive until the majority of the macro commands are implemented.

While the initial objectives were actually quite simple to achieve, they revealed other obstacles and problems that need to be handled. I plan to solve some of these problems over the next few days and my goal is to have a functional player in the next few days. The timing system still needs some work. I’m still trying to find the best method to handle the track tempo and wait commands which will also satisfy the requirements of some of the macro commands. The other major difficulty arises in the handling of some of the more complicated note effects. Once I can clear up some of the issues regarding the note processing and some of the more important macros, I will be able to have a simple player with limited capabilities. With the midterm evaluation looming, I expect to have this ready shortly.

Objective 1 Update

I’ve implemented the readTrackstep() functionality of the program. It simply reads through the trackstep of a song line by line and checks for either commands or pattern numbers. When it finds a pattern number, it passes on the handling to a separate readPattern function. The trackstep commands still need some work. There is a total of five commands where one is a simple stop command and another is not used in Monkey Island. The remaining three commands either play a selection, set the tempo, or start a volume slide. These three functions will be developed as the track handling structure evolves. The trackstep is handled via an 8 member array of track structures. The requirements of this structure is still changing as I refine the design. I still need to develop a method for handling the tempo/speed settings at which tracks are updated. I’m still not quite sure how to do this.

The readPattern function is also partially implemented with work in progress. Reading patterns involves identifying either commands or notes. The notes are handled by a separate function, leaving the command handling to be taken care of. The pattern commands are very much linked to the trackstep so developing the command reading functionality will help refine the required track structure. As the commands are implemented, I will go back to make the necessary changes and updates to the readTrackstep function.

Upcoming Tasks

My mentor, Jubanka, was able to help clarify some misunderstandings and put some hesitations to rest. With his help, I was able to get re-focused and back on track. The midterm review is coming up next week and I’m still planning on having a functional TFMX parser in place. The essential structure to read and interpreter TFMX files will be in place with the gaps being filled in afterwords.

A clean structure of the program was difficult to establish but a general design, which will likely be modified several times even over the next week, is in place. The TFMX interpreter loads the file and creates an array of songs. Each song corresponds to a starting and ending position in the trackstep, so the interpreter then loads the trackstep for a corresponding song. The interpreter then reads the trackstep to find the patterns or commands listed. It then either reads the pattern information or handles the command accordingly. The pattern information is then broken down into pattern commands or notes which the interpreter handles. The notes are then handled with their appropriate macros and samples. Well, in general, that’s what it does. It sounds easy I guess…

The midterm evaluations begin on July 7th, next Monday. Over the next week, certain objectives need to be handled to ensure the overall goal is met.

  • Objective 1: Develop the trackstep reader. This function will take a specified song number and be able to read the corresponding trackstep. When parsing the trackstep, it will look for either a pattern number or a trackstep command. The pattern numbers are looked up in table of pointers which give the offset to the corresponding pattern. When the reader sees trackstep commands, calls to external corresponding functions will be made. The trackstep commands will be handled separately.
    Obstacles: The biggest obstacle here is the handling of the trackstep commands. The pattern numbers are easy to read and the look-up procedure should be easy to develop. Some of the trackstep commands are not well documented and sometimes seem unnecessary or redundant. By separating the trackstep command handling into separate, individual functions, I believe this problem will be easier to overcome and leaves the possibility of developing only limited functionality open.
    Estimated completion time: Tuesday, July 1st.
  • Objective 2: Develop the pattern reader. This function will take a specified pattern number (or pattern address – haven’t decided if the look-up procedure will take place here or in the trackstep reader) and analyze it. Again here, the pattern information is broken down into notes or commands so the function must implement a system for determining what it is handling. When it finds a note, the note reading is then passed onto a separate function. The command handling is either done in house or again, to individual, external functions.
    Obstacles: The obstacles in this function are similar to those in the previous objective. The difficulty arises in handling the 15 or so different pattern commands. Again, I think the appropriate strategy here is to develop the command functionality separately. This will also be useful for certain overlapping functionality between the trackstep commands, the pattern commands, and the macro commands.
    Estimated completion time: Friday, July 4th.
  • Objective 3: Develop the note reader. The function will take analyze a note passed to it by the patten reader. This function needs to read the note and determine the note value, the volume, the channel number, the wait time, and most importantly, the macro used to play the note.
    Obstacles: The initials tasks of this objective should prove to be simple. The difficulty arises in the handling of the macros. Here again, I will have to diverge into a separate function for handling macros. I also need to clarify some concepts about the note values and the use of tables as look-ups.
    Estimated completion time: Sunday, July 6th.

By the time the midterm evaluation period rolls around, I should be developing the macro handling function and the corresponding commands. At the same time, the links between the TFMX class and the parent PAULA class will be forged. I will update throughout the week as each of the upcoming objectives gets completed.

Small Updates

With exams over and done with, I spent the weekend reintroducing myself to the project and attempting to get organized. I plan to shift to more frequent updates now that my schedule permits it and plan on working frantically over the next few weeks to make up for lost time due to exam stress. I have a meeting with my mentor Jubanka scheduled this week which should hopefully prove useful in ironing out many of the issues currently presenting themselves.

I’ve hacked up the main() function in order to allow for the ScummVM console to indefinitely delay while TFMX objects can be initialized and played. This has already proven useful in testing as the console can as an output for debugging. Some code for initialization has been put together hastily as well but will most likely be recycled as things get overhauled.

The analysis of the file format is complete and when time permits, as other pressing issues dissolve, I will compile the bulk of the information into a useful package for future TFMX hackers. This includes a detailed look at the patterns and macro commands. Also, a couple of minor issues regarding some of the oddities of TFMX will also be addressed.

Right now the focus is on hammering out design details and implementing proper strategies for the player so it is clean, readable, re-usable while remaining fully functional. Fumbling around with some of these previously implemented C players has been a nightmare and the last thing I want to do is simply rewrite bad code for a different backend. I believe overcoming some of these major challenges and having a better understanding of the “big picture” structure will allow me to better develop the smaller building blocks.

Challenges

To date, the most difficult undertaking of this project has been going from theoretical to practical. In essence, it is relatively simple to understand how an audio player should work and specifically how the TFMX format is built. With some documentation and by reverse engineering a few players, I have been able to build a solid understanding of the TFMX file structure. This pointer goes here, this command does that, and so forth but the real challenge arises in going from that initial understanding to building functional code in C++ (or any language for that matter). It is easy to logically read through a TFMX file and think, OK, this part here specifies what note to play, and so forth but our mind doesn’t think in C++ (unless you are like some people I know). We generally don’t think about things being broken down into if statements and for loops. So how do you go from those initial steps to designing code that incorporates variables, classes, and algorithms which need to systematically come together to represent that system that you understand only in your brain’s logic (and perhaps faulty logic)?

The answer obviously lies in practice and experience. If you don’t think like a computer, then it is going to take training to shape your brain into a computer. You will have to practice and scan through endless lines of source code. You will have to buy as many books as you can and when you run out of cash, head to the library. But one thing is for sure, it still will be challenging.

Now excuse my ranting…

Slow Moving

Well it definitely has been quite some time since my last update. I’m basically still slaving away here and haven’t had much time to provide progress updates. Things are moving slow and I’m still working on completing my previous list of objectives. I have a complex analysis final exam coming up this Wednesday so my attention has been diverted for the last week or so to prepare for my exam. Fortunately, I have decided not to take a second session summer course so I will be doubling the time and effort directed to the project to make up for the slow progress. I plan to further update my current goals and reorganize my list of objectives Wednesday after my exam is complete. With more time at my disposal, I’m still planning on having TFMX support implemented by the time midterm evaluations roll around in early July. Stay patient and more progress reports will start coming in by the end of the week while things get back on track.

Upcoming Tasks

So with coding official underway, it might be wise to take some time to organize some upcoming tasks and milestones. For now, I have a handful of goals which are critical to developing a functional TFMX replayer.

  • Complete further analysis of the TFMX file structure. I need to specifically look at the structure of pattern and macro data as well as the trackstep to interpret their function and commands. This is already underway. For now, focus is on the trackstep and pattern data with the macro implementation being ignored until later.
  • Develop structure in main() function for launching and experimenting with PAULA audio objects. This will later transform to a launcher for testing TFMX objects.
  • Design class and refine class structure for TFMX objects with basic function prototypes and member data.
  • Begin initial code to open TFMX files, perform some basic checks, and initialize necessary tables read from the TFMX file.

As of right now, I’m working through interpretation of the pattern data functions. This is a little bit tricky as the documentation available is not always clear and in some cases is just wrong. I’ve reworked through some of the other existing players to try to patch together the necessary information.