Pearls on a String

Hey all!

It’s been a while since I last blogged (whoops!).

But in the time since my last post, quite a bit has happened.

First off, I got the DS port working with loadable modules after reintegrating the thumb-interworking code into the DS port. It was quite simple, really, as the thumb relocations used were PC-relative (and thus needed little to no manual work). I tested a good number of the engines after getting this work done, including the SCUMM, SKY, LURE, QUEEN, and AGI engines. They all seemed to work fine (though I only played through about 10 minutes of each game and I’ve heard some, like Sam and Max, use more memory later on). The one exception in terms of successful testing is the CRUISE engine. That plugin had a single relocation type (R_ARM_TARGET1) that none of the other engines have. This type is supposed to be treated either the same as R_ARM_ABS32 or R_ARM_REL32 (how it should be treated can differ, even in the same file). Since I saw no easy way to determine how it should be treated during runtime, I added “–target1-abs” linker flags to the ds makefile (the flags specify that R_ARM_TARGET1 should be treated as R_ARM_ABS32 in all cases) and coded for the R_ARM_TARGET1 relocation accordingly, but to no avail (“Cruise for a Corpse” still crashes on bootup, showing only the Cruise cursor). As of now I can’t see the console (on the top screen of the ds) at the point of the crash, so debugging is difficult, but I’ll continue to work on this.

Another problem with the DS build is that (as mentioned before) I had to remove the “–gc-sections” linker flags since things were being garbage-collected in the plugins and main executable that referred to each other. This unfortunately resulted in a lot of bloat in the main executable. For some engines, this bloat is around 300kb, which is too much to be ignored. There’s been some discussion among the mentors concerning different ways to decrease this bloat, including building static builds with garbage collection and dumping the symbols then telling the linker to included only those symbols in the dynamic builds. In any case, this is something that will need to be worked on before the DS dynamic plugins work is reintegrated into the trunk.

Apart from the DS work, I did some work abstracting a more generic ELF-loader last week. It’s a little rough at the moment, but works for both the PS2 and DS ports, so at least I didn’t break anything :-). I put the methods dealing with relocations into their own files based on processor type (like arm-relocs.cpp and mips-relocs.cpp) and also split the shorts-segment-manager (made to effectively use the gp-relative section of MIPS processors) into its own file. There’s still a bit of work to be done for abstraction, including making subclasses of DLObject (like PS2DLObject, which could be a subclass of MIPSDLObject, etc.) and having the different plugin-providers use these different subclasses (right now, I just used “ifdefs” with different ports).

I’ve also worked on a bit of the plugin design change work. It took me a while of looking through the base code to understand how to best begin implementing the plugin design changes, but I added some new functions to the PluginManager class (a “loadFirstPlugin” and “loadNextPlugin”) as well as changing main.cpp and some functions in plugins.cpp to use these functions in a loop when “NEW_PLUGIN_DESIGN_FIRST_REFINEMENT” is enabled. I tested on a Windows build with dynamic plugins enabled and was able to successfully launch games with only one plugin ever loaded at a time!

So the plan for this week is to continue plugin design change, abstraction, Cruise for a Corpse, and DS memory problem work,

Tony

Last week, I restructured the DS loader to use a SeekableReadStream for file reading. Once that was in place and working, I restructured the PS2 loader to use a SeekableReadStream as well and tested it (this needed to be done anyway and it helped me ensure there was nothing inherently wrong with the way I was using SeekableReadStream in the DS loader).

I then had problems reading the symbol table in successfully. After some wasted time :P, I realized it was just a silly mistake. I had neglected to initialize _symtab_sect to anything in dsloader.h and the value of uninitialized ints is undefined in C++. Thus, the loader was detecting, from the value that _symtab_sect had, that the symbol table was already read in when it wasn’t.

At this point the loader was getting to the point where I had to deal with the relocations. I used consolePrintfs to make sure that the relocation types the loader was detecting were the same as the ones detected by arm-eabi-objdump (they were). I then began coding for these relocations. After a day of work on the relocations with no success as far as getting games to run with dynamic plugins enabled on the DS, I decided to disable the thumb-interworking on the ds temporarily so I could work on getting the DS with plugins in a runnable state without having to deal with the complication of thumb instructions/relocations.

I worked further on the relocations yesterday; I started dumping the instructions to be relocated and immediately noticed a MAJOR problem. Every instruction was 0x00000000! The addresses of the instructions seemed reasonable, though, so I suspected it had something to do with how the plugins were being built and not how I was retrieving target addresses. I dumped Mapfiles with “-Map” during plugin linking and discovered that whole input sections were being discarded, among other problems. After switching out the linker script for a modified version of the default “arm-eabi-ld” linker script, these problems seemed to be fixed, perhaps because the modified linker script I was using (based on the script used by the main executable) used the MEMORY command to map out different blocks of memory but I was trying to force the start address of the “.text” section to be 0. I am not sure whether switching out this linker script will cause problems.

I then did a fair bit of reading on the ARM instruction set and continued tweaking/testing the relocation code. Things still weren’t working like they should have. Yotam suggested comparing completely unrelocated code and semi-relocated code to see what ld was already doing for me, and through a few dumps, I realized I was trying to do A LOT that ld already did redundantly and thus messing up things that were already fixed! After removing this redundant code, I got Beneath a Steel Sky to run successfully with Dynamic Plugins enabled on the DS!

My next order of business, then, is to test the rest of the engines (some of which have relocation types I still haven’t dealt with). Then, I’ll reintegrate the thumb-interworking and deal with any major complications that brings (if any :D).

~Tony

Also, I plan to update my wiki schedule shortly: http://wiki.scummvm.org/index.php/User:Toneman

While working on changing how the loader opens and reads plugin files on the DS this weekend, I noticed that the “.plg” files produced in my builds were significantly smaller than expected (as in under a hundred kilobytes).

So I looked through verbose build output to investigate what the problem could be. The plugin linking output included all the necessary object files, so I knew the problem wasn’t there and decided to look closer at the plugin flags.

I worked through some build errors, switched my plugin.ld to use a modified version of the correct linker script and added a number of PLUGIN_LD_FLAGS (including “-mno-crt0 $(DEVKITPRO)/devkitARM/arm-eabi/lib/ds_arm9_crt0.o”), and plugins started linking. I had some trouble with plugin.ld syntax errors, but figured it out (I was doing things in SECTIONS like “{ contents } >region =fill :phdr” when it needed to be “{ contents } >region :phdr =fill”). Using objdump on the plugins, it seems most only have one relocation type to worry about (R_ARM_ABS32), but the cine and cruise engines also use R_ARM_THM_CALL and R_ARM_TARGET1. I’ll be working on completing dsloader.cpp (including the ARM-specific relocations) now, my first milestone being getting one of the engines that uses just R_ARM_ABS32 working.

Work on the DS dynamic module loader is progressing. In the tail end of last week, I put the basic loader code in for the DS and modified the default arm-eabi linker script to make a plugin linker for the DS. This week I found (in a .spec file included in one of the makefiles for the DS) that the DS port doesn’t use the default arm-eabi linker so I’ll have to modify the one it does use (ds_arm9.ld) instead. At the opening of this week, I took some time to try to understand the Makefiles for the DS (it’s a bit more complicated than the other ports I’ve worked with) and then started modifying them to use dynamic modules. I had a bit of trouble getting make to attempt to link the plugins, but after correcting some dependencies, linking of plugins is attempted (though not completed successfully, which hopefully will change once I switch to a modified version of ds_arm9.ld for plugin linking). I’ll post again once the plugins are successfully linking.

This week has not been the most productive thus far (though it’s only just begun). I’ve been trying to get engine plugins building for the ds (using the custom “plugin.ld” linker script) so I can begin work on the ARM-specific relocations needed for the DS, but the way I’ve modified the DS makefiles, engine plugins don’t build. (I currently get things like “warning: overriding commands for target ‘plugins/scumm.plg'”). I’ll be contacting Niel to better understand the DS make system shortly.

Check it out 🙂

http://wiki.scummvm.org/index.php/User:Toneman

So I was trying to figure out which relocation types I really need to worry about when it comes to the GP2x-Wiz today. After making a number of tweaks to the incomplete loader to get it to compile, I compiled and linked (with my modified ARM linker script) the game engines into plugins. I tried using “objdump” on these plugin files to figure out which relocation types were worth my time, but it kept returning all types as “unknown”. After a bit of research, I discovered another program “readelf” that successfully dumped relocation types. I noticed, however, that when I do use “objdump -R” which dumps a dynamic relocation table, I get output for both the main “scummvm.wiz” file and the plugin files. This leads me to believe that the binaries may be PIC, but since the version of objdump I’m using doesn’t seem to support ARM (as mentioned before, it couldn’t detect the relocation types), the reason it’s outputting a dynamic relocation table could be because it’s misreading the files, not because the files really are PIC. My goal for this weekend, then, is to become more certain about the nature of the generated files, and then to code for the relocations I may need to worry about, which are listed below (the names of some of these relocations also has me worried about PIC code):

• R_ARM_GLOB_DAT
• R_ARM_JUMP_SLOT
• R_ARM_COPY
• R_ARM_PLT32
• R_ARM_PC24 
• R_ARM_BASE_PREL
• R_ARM_GOT_BREL
• R_ARM_PC24
• R_ARM_ABS32

As of yesterday, the PS2 port would fail to compile with Dynamic Modules turned on unless I put nothing in the GP-relative section by adding a “-G0” flag to the Defines in the Makefile. Last night (after a discussion with Yotam) I made modifications to the linker scripts used when compiling with Dynamic Modules that got rid of these errors. In the linker script for the main engine, I had an extra zero appended to a hex value specifying an offset from the GP register. Fixing this typo (along with rearranging things in plugin.ld so that the .bss section was no longer assigned to the shorts segment, but rather to the plugin segment) seemed to eliminate the problem I was having with “relocation truncated to fit errors”.