Fixing the music in the PS2 version

After last week's progress, the sound in the PC version of EMI when run in ResidualVM was close to perfect. However, the PlayStation 2 version of the game still wasn't playing any music. Although I haven't focused much on support for the PS2 version so far I decided to look into the PS2 music issues, because I was ahead of schedule and this was one of my stretch goals.

PS2 version of EMI running in ResidualVM.

Much of the groundwork for PS2 support has already been done by previous contributors. For example, the SCX sound format used for the sounds in the PS2 version is supported (although the implementation currently doesn't stream from the disk). However, the music wasn't playing because a mapping from music state numbers to filenames was missing.

EMI (like other iMUSE games before, I presume) assigns an integer state number for each music track. The game scripts tell the engine which state should be playing at any given time by calling the engine function ImSetState, which takes a state number as parameter. In the engine, we then need to be able to map the state number to the correct music file.

In the PC version this mapping is stored in the file FullMonkeyMap.imt, but unfortunately in the PS2 version the mapping is hardcoded into the PS2 executable.

To locate the data inside the PS2 executable, I used the excellent PS2 disassembler tool ps2dis. Firstly, after starting up ps2dis and opening the executable, I invoked the static analyzer from the Analyzer menu. This will add function labels and track data references, making the disassembly more understandable.

Next, I opened the "Jump to Labeled" dialog with Ctrl+G. This dialog lists the strings stored in the binary. Scrolling the list down a bit revealed this:

The "Jump to Label" view in ps2dis.

The strings that end with ".scx" are the music filenames. If we jump to the location where one of the strings is stored, We can then use "Jump to Next Referer" with F3, which will take us to a location in code that references this data. For the music filenames there is only one location referencing them.

A section of code that references the music filenames.

This looks promising! The code here grabs the filename plus some unknown data and passes this information to a function. There are exactly 125 references to the filenames here, which is the same as the number of music states in EMI. My assumption is that the code here builds the state to filename mapping entry by entry counting up from state number 0.

With the state number to filename mapping figured out and implemented into ResidualVM, the PS2 version now plays music as well. These changes are now included in PR #972. With some additional fixes introduced in PR #975, the PS2 version is now fully playable, although some (mostly graphical) issues still remain.

Improving the music playback

For the past week I've been working on a number of improvements to the EMI sound system in ResidualVM. My change set in PR #972 improves especially the music playback in the PC version of the game.

Firstly, the music playback now starts at the correct position of the track, and the correct sub-section of the track is looped. This is because the music player now respects the cue point data which is stored in files with the file suffix ".jmm" alongside the actual MP3 music files.

The .jmm files may contain 'start', 'jump' and 'end' cues. The 'start' cue specifies a position in milliseconds where the music track should start playback. The 'jump' cue specifies two positions on the track. Once the first position is reached, the playback will jump to the second position. This cue is typically used to loop a sub-section of the music track. If the JMM file does not specify a 'jump' cue, the track does not loop (an example of this is the chapter screen music). The 'end' cue seems to always have a value of -1 in EMI, so I chose not to implement handling for it at this time.

Secondly, music syncing is now implemented. In some parts of the game, when the music track changes the new track should start from the same position where the previous track ended. Typically such synced tracks are slight variations of the same composition. For example, when Guybrush is close to the fountain in Melee town (set mel), the music changes to a version of the Melee town music with a sound of flowing water mixed in.

The information whether a music track should be synced with another track or not is stored in a file called FullMonkeyMap.imt. This file maps music track id numbers (which are referenced from the Lua scripts) to sound files, and specifies some parameters for each track. One of these parameters is called "sync", and has an integer value associated with it. A value of 0 seems to indicate that the track should never be synced. Otherwise, if the sync value matches the sync value of the track that was playing previously, the track will start from the same position where the previous track was at the time of transition.

Another improvement to music playback is when the music track changes, the music now transitions with a linear cross-fade, instead of instantly cutting to the new track. This makes the transitions sound less abrupt, and matches the behavior of the original game.

Correction to the shadow data format

After writing the previous blog post about shadows, I discovered that my shadow implementation caused a crash in the House of Sticks at Lucre island. The format of the shadow data in this particular set did not seem to match the format I described in the previous post.

It turned out that my interpretation of the shadow data was wrong. The shadow color is not part of the 'sector reference' structure as I thought before. Instead it is part of the 'shadow' structure. The color is stored after the array of 'shadow reference' structures.

The House of Sticks after correcting the issue.

It appears that in most cases a 'shadow' structure only contains one sector reference. This is why my initial implementation seemed to work. However, the House of Sticks set file contains a 'shadow' structure that references multiple sectors, and in this case my implementation tried read too much data, since it attempted to read a color for each sector reference.

This discovery indicates that a single shadow will always have a uniform color. However, if multiple shadows are drawn on top of each other, this can give the appearance of a single shadow with multiple colors. This happens for example in the Place of Prostheses.

A combination of two shadows with different colors can be seen in the Place of Prostheses.

Since the shadow color is uniform, rendering the shadows is simplified. PR #956 now contains much less changes to the renderer code, as the EMI shadow rendering is now very similar to that of Grim.

Implementing shadows

Feeling relaxed after a week of vacation I'm now finishing up work on implementing shadows for EMI. This work is now in PR #956.

EMI has two types of shadows: simple shadow sprites (internally called "lame shadows" or "dumb shadows" :) ) and planar projection shadows. The projection shadows look more realistic, but are only used in a few sets. If shadow quality is set to low in the options, only shadow sprites are used.

The screenshots below are taken from the original game. In the screenshot on the left, shadow sprites can be seen under Guybrush and Elaine. The screenshot on the right shows projection shadows cast by Guybrush and the brazier on the deck of the ship. Initially with ResidualVM both shadow types were not drawn at all.

After my previous sprite fixes, enabling the shadow sprites turned out to be very simple. Previously ResidualVM's actor drawing code contained a hack that skipped drawing the costume containing the shadow sprite if shadows were not explicitly activated for that actor with a call to ActorActivateShadow from Lua. After some testing in the original game I found out that the ActorActivateShadow function doesn't affect the shadow sprites at all in the original (instead it is used to toggle projection shadows). After removing the hack, the shadow sprites showed up as expected.

I then moved on to tackle the projection shadows. This turned out to be slightly trickier. Fortunately projection shadows have been implemented previously in ResidualVM's renderers for Grim Fandango, so most of the rendering code could be reused also for EMI. To render projection shadows we need a shadow color, the light position as well as the planes on which shadows are cast on. The first problem was that I didn't know where the original game got this information from.

In Grim, the shadow color, the light position and the shadow planes are set up through Lua code, but in EMI I couldn't find anything similar in the game's Lua scripts. The information had to be stored somewhere else.

The next place I decided to look in was the .setb files, which contain scene-specific information such as camera positions, sectors and lights encoded in binary form. The shadow color seems to change between sets, so it seemed like a fair assumption that the information I was looking for would be stored in the set data.

This led me to the discovery that for sets that have projected shadows enabled, there was indeed some data at the end of the .setb file that was not read by ResidualVM (marked with red in the pic above). This previously ignored block of data contains an array of structures that I call 'shadow' structures here. Each 'shadow' structure contains a name, the light position to be used when projecting the shadow and an array of 'sector reference' structures. The 'sector reference' structure contains the name of a sector within the set as well as the shadow color to be used when a shadow is projected onto that sector.

Once I discovered this data, it was fairly easy to enable the pre-existing planar projection shadow rendering code in ResidualVM also for EMI. The main difficulty was that previously the renderers only supported one global shadow color, but in EMI the shadow color can be set per-sector. I'll discuss this issue and how I resolved it in detail in the next blog post.

With all the issues resolved, it's very hard to tell ResidualVM and the original game apart in most scenes. The screenshots below are from ResidualVM. Compare them with the screenshots of the original above :)

Head tracking

For the past week I've been working on enabling head tracking. Head tracking allows actors to orient their head towards any point in the game world. This work is now included in PR #947.

The head tracking works by reorienting one of the bones of the skeleton after keyframe animation is applied. The bone is oriented to face the direction the actor wants to look towards. The look direction is interpolated, so the change in orientation is not instant.

The original Lua code uses the functions ActorSetHead and ActorSetHeadLimits to set up head tracking. The former takes as parameter the name of the bone that should be animated, while the latter takes three parameters that specify horizontal and vertical angle limits. The angle limits ensure that the head won't be oriented unrealistically.

My current implementation looks good in most cases, but I did find one case that looks noticeably different from the original. Mr. Cheese's head is at a weird angle when he looks towards Guybrush in the Scumm Bar. The screenshot on the left is ResidualVM, while the one on the right is the original.



The head tracking animates Mr. Cheese's neck bone, but the neck bone has children which are animated by keyframe animation, and the vertices of the head mesh follow the child bones. Although the neck bone is facing Guybrush, the head mesh is not, because of the animation applied to the child bones.

I spent a significant amount of time trying to figure out the difference between my implementation and the original during the week, but none of my solutions produced the same result as the original in this case.

One of my attempts was to override the animation of the neck bone as well as it's children bones completely whenever head tracking is active. This way the head was oriented exactly towards the point the actor looks at. That, however, didn't match the original game either. Mr. Cheese's head does seem to be affected by the animation also in the original.

For now I decided to leave this be. Mr. Cheese is currently the only case I'm aware of where the result is different from the original. I may return to this at a later stage once I've finished the other remaining tasks.