EMI is completable!

I finally got around to do a proper playthrough of EMI in ResidualVM with all the fixes and improvements made during GSoC. I'm happy to say the game can now be played from start to finish with only some minor issues :)

The biggest issue that remains right now is a bug with the lava boat puzzle on Monkey island. The boats on the lava field are not visible, which makes the puzzle difficult to complete. JoseJX is working on this issue though, so hopefully it will be resolved soon.

Other than that, there are a few graphical issues. Animations occasionally still seem to snap to the wrong frame. This happens for example when jumping out of the bank window on Lucre island, and when Guybrush gets a grog at the Micro-Groggery on Jambalaya island. Also, the text in the end credits flickers and the text color doesn't match the original.

Pathfinding is not perfect yet. Actors tend to zig-zag around before reaching their destination, which looks silly sometimes.

On the audio side, footstep sounds are sometimes wrong. For example, when walking on the Jambalaya island beach, it sounds like Guybrush is walking on wood. Also, voiceovers sometimes tend to get cut short, which is a bit annoying.

These issues will have to be resolved after GSoC though. I'm pleased that all of the issues I spotted are fairly minor, and the game is very much playable in the current state. I'm confident that support for EMI could be included in the stable release of ResidualVM fairly soon.

As GSoC soon comes to a close, I want to thank my mentors and the ScummVM/ResidualVM community for your support and for providing me this great opportunity to bring another classic adventure back to life. This has been a fun summer!

Bug squashing

With all the major feature implementation tasks completed, this week I focused mainly on fixing any remaining bugs I could find. Klusark was kind enough to do a full playthrough of the game in ResidualVM, which revealed a number of issues of varying severity. Some of these issues were blockers which prevented progression in the game at certain points. These should all be resolved now, and on current master the game should be completable from start to finish again.

Here's a brief summary of the issues I resolved this week:

• Guybrush got stuck when attempting to lure the fish on Lucre island into the scupperware box, preventing progression in the game
• The game got stuck when giving the earrings to the Dainty Lady figurehead
• Saving and then restoring a savegame caused lighting to break
• Saving and restoring caused a crash due to broken shadow initialization
• Guybrush faced the wrong way after the Voodoo Lady summons the items from the heirloom chest
• Some of the ships and the water at the harbor on Melee island were rendered incorrectly
• Guybrush was invisible in the heirloom cave, and the light shaft effects were drawn incorrectly

The last two issues were the most interesting ones, so I'll discuss those a bit next. Let's look at a before-and-after comparison from the Melee Island harbor.

Several issues can be seen in the screenshot of the old version on the left. The color of the waves is too dark, the water splash sprites are drawn over the ship and the rowboat, and the bottom of the ship is not clipped correctly.

In debugging these issues, I used Apitrace in order to figure out how exactly the original game forms the final picture. Since the game has an OpenGL renderer (activated with the parameter -gl), we can use Apitrace to extract the list of OpenGL function calls as well as the OpenGL state on each drawn frame. By examining the trace I found that ResidualVM was drawing some of the sprites in a different order than the original game, and for some sprites depth testing was enabled while it wasn't in ResidualVM. Also, the original game draws mask sprites to clip the ship and the rowboat to the correct shape, which were not drawn in ResidualVM at all.

The mask sprites visualized. With additive blending the black bits become transparent.

When actors (including sprites) are about to be drawn in EMI, they are first sorted into descending order by their sort order, which is a per-actor integer field that can be controlled from the game scripts. When an actor is attached to another, the actor's sort order will be overridden by the parent actor's sort order. This is what happens when the water splash sprite is attached to the ship sprite. However, after the sprite is attached, the scripts then set a new sort order for the sprite. In the original engine this value overrides whatever was derived from the parent, but in ResidualVM the sort order was not updated, which caused the sprites to be drawn in wrong order. Updating the sort order even if the actor is attached fixed this issue.

The rest of the issues were mostly related to incomplete parsing of sprite and model data files. I discovered flag fields in the sprite data, which control whether depth and alpha testing should be enabled for that sprite, and whether additive blending should be used or not. I also found a similar flag that controlled the blending mode in the model data. I discovered these mostly through trial-and-error, by changing suspicious-looking bits in the data files slightly and then loading up the original game to see how the changes effect the end result. Setting the correct blend mode for models fixed the water color.

To enable drawing of the mask sprites, I had to disable a piece of old, temporary code that simply skipped drawing of any sprites with "mask" in the name. I assume this was done because if incorrectly rendered, the mask sprites looked ugly and only cluttered the image (see the image above). Now with the blending and sort order fixes in place, the masks did produce the correct result, though.

This is the heirloom cave before and after my changes. The left one is fairly obviously broken: Guybrush is invisible, the light shaft effects look ugly, and shadows are missing. Fortunately the light shaft effect was corrected by the previous model blend mode fix.

Guybrush's invisibility was caused by another seemingly temporary hack in ResidualVM, which disabled drawing to the color buffer if an actor's sort order was greater or equal to 100. In the cave, Guybrush's sort order is set to 100 in the scripts, which caused him to become invisible. The original engine doesn't seem to do anything similar, so I concluded that this behavior was wrong.

The cave set also led me to discover another previously unknown field in the set shadow data. The field is a string, which specifies a name of a light in the set. When this string is present, the shadow projection point should be the light's position, instead of whatever is stored in the shadow data. After parsing the light name and implementing this behavior in ResidualVM, the shadows now work correctly in the cave also.

These fixes can currently be found in this branch.

Fixing the music in the PS2 version

After last week's progress, the sound in the PC version of EMI when run in ResidualVM was close to perfect. However, the PlayStation 2 version of the game still wasn't playing any music. Although I haven't focused much on support for the PS2 version so far I decided to look into the PS2 music issues, because I was ahead of schedule and this was one of my stretch goals.

PS2 version of EMI running in ResidualVM.

Much of the groundwork for PS2 support has already been done by previous contributors. For example, the SCX sound format used for the sounds in the PS2 version is supported (although the implementation currently doesn't stream from the disk). However, the music wasn't playing because a mapping from music state numbers to filenames was missing.

EMI (like other iMUSE games before, I presume) assigns an integer state number for each music track. The game scripts tell the engine which state should be playing at any given time by calling the engine function ImSetState, which takes a state number as parameter. In the engine, we then need to be able to map the state number to the correct music file.

In the PC version this mapping is stored in the file FullMonkeyMap.imt, but unfortunately in the PS2 version the mapping is hardcoded into the PS2 executable.

To locate the data inside the PS2 executable, I used the excellent PS2 disassembler tool ps2dis. Firstly, after starting up ps2dis and opening the executable, I invoked the static analyzer from the Analyzer menu. This will add function labels and track data references, making the disassembly more understandable.

Next, I opened the "Jump to Labeled" dialog with Ctrl+G. This dialog lists the strings stored in the binary. Scrolling the list down a bit revealed this:

The "Jump to Label" view in ps2dis.

The strings that end with ".scx" are the music filenames. If we jump to the location where one of the strings is stored, We can then use "Jump to Next Referer" with F3, which will take us to a location in code that references this data. For the music filenames there is only one location referencing them.

A section of code that references the music filenames.

This looks promising! The code here grabs the filename plus some unknown data and passes this information to a function. There are exactly 125 references to the filenames here, which is the same as the number of music states in EMI. My assumption is that the code here builds the state to filename mapping entry by entry counting up from state number 0.

With the state number to filename mapping figured out and implemented into ResidualVM, the PS2 version now plays music as well. These changes are now included in PR #972. With some additional fixes introduced in PR #975, the PS2 version is now fully playable, although some (mostly graphical) issues still remain.

Improving the music playback

For the past week I've been working on a number of improvements to the EMI sound system in ResidualVM. My change set in PR #972 improves especially the music playback in the PC version of the game.

Firstly, the music playback now starts at the correct position of the track, and the correct sub-section of the track is looped. This is because the music player now respects the cue point data which is stored in files with the file suffix ".jmm" alongside the actual MP3 music files.

The .jmm files may contain 'start', 'jump' and 'end' cues. The 'start' cue specifies a position in milliseconds where the music track should start playback. The 'jump' cue specifies two positions on the track. Once the first position is reached, the playback will jump to the second position. This cue is typically used to loop a sub-section of the music track. If the JMM file does not specify a 'jump' cue, the track does not loop (an example of this is the chapter screen music). The 'end' cue seems to always have a value of -1 in EMI, so I chose not to implement handling for it at this time.

Secondly, music syncing is now implemented. In some parts of the game, when the music track changes the new track should start from the same position where the previous track ended. Typically such synced tracks are slight variations of the same composition. For example, when Guybrush is close to the fountain in Melee town (set mel), the music changes to a version of the Melee town music with a sound of flowing water mixed in.

The information whether a music track should be synced with another track or not is stored in a file called FullMonkeyMap.imt. This file maps music track id numbers (which are referenced from the Lua scripts) to sound files, and specifies some parameters for each track. One of these parameters is called "sync", and has an integer value associated with it. A value of 0 seems to indicate that the track should never be synced. Otherwise, if the sync value matches the sync value of the track that was playing previously, the track will start from the same position where the previous track was at the time of transition.

Another improvement to music playback is when the music track changes, the music now transitions with a linear cross-fade, instead of instantly cutting to the new track. This makes the transitions sound less abrupt, and matches the behavior of the original game.

Correction to the shadow data format

After writing the previous blog post about shadows, I discovered that my shadow implementation caused a crash in the House of Sticks at Lucre island. The format of the shadow data in this particular set did not seem to match the format I described in the previous post.

It turned out that my interpretation of the shadow data was wrong. The shadow color is not part of the 'sector reference' structure as I thought before. Instead it is part of the 'shadow' structure. The color is stored after the array of 'shadow reference' structures.

The House of Sticks after correcting the issue.

It appears that in most cases a 'shadow' structure only contains one sector reference. This is why my initial implementation seemed to work. However, the House of Sticks set file contains a 'shadow' structure that references multiple sectors, and in this case my implementation tried read too much data, since it attempted to read a color for each sector reference.

This discovery indicates that a single shadow will always have a uniform color. However, if multiple shadows are drawn on top of each other, this can give the appearance of a single shadow with multiple colors. This happens for example in the Place of Prostheses.

A combination of two shadows with different colors can be seen in the Place of Prostheses.

Since the shadow color is uniform, rendering the shadows is simplified. PR #956 now contains much less changes to the renderer code, as the EMI shadow rendering is now very similar to that of Grim.

Implementing shadows

Feeling relaxed after a week of vacation I'm now finishing up work on implementing shadows for EMI. This work is now in PR #956.

EMI has two types of shadows: simple shadow sprites (internally called "lame shadows" or "dumb shadows" :) ) and planar projection shadows. The projection shadows look more realistic, but are only used in a few sets. If shadow quality is set to low in the options, only shadow sprites are used.

The screenshots below are taken from the original game. In the screenshot on the left, shadow sprites can be seen under Guybrush and Elaine. The screenshot on the right shows projection shadows cast by Guybrush and the brazier on the deck of the ship. Initially with ResidualVM both shadow types were not drawn at all.

After my previous sprite fixes, enabling the shadow sprites turned out to be very simple. Previously ResidualVM's actor drawing code contained a hack that skipped drawing the costume containing the shadow sprite if shadows were not explicitly activated for that actor with a call to ActorActivateShadow from Lua. After some testing in the original game I found out that the ActorActivateShadow function doesn't affect the shadow sprites at all in the original (instead it is used to toggle projection shadows). After removing the hack, the shadow sprites showed up as expected.

I then moved on to tackle the projection shadows. This turned out to be slightly trickier. Fortunately projection shadows have been implemented previously in ResidualVM's renderers for Grim Fandango, so most of the rendering code could be reused also for EMI. To render projection shadows we need a shadow color, the light position as well as the planes on which shadows are cast on. The first problem was that I didn't know where the original game got this information from.

In Grim, the shadow color, the light position and the shadow planes are set up through Lua code, but in EMI I couldn't find anything similar in the game's Lua scripts. The information had to be stored somewhere else.

The next place I decided to look in was the .setb files, which contain scene-specific information such as camera positions, sectors and lights encoded in binary form. The shadow color seems to change between sets, so it seemed like a fair assumption that the information I was looking for would be stored in the set data.

This led me to the discovery that for sets that have projected shadows enabled, there was indeed some data at the end of the .setb file that was not read by ResidualVM (marked with red in the pic above). This previously ignored block of data contains an array of structures that I call 'shadow' structures here. Each 'shadow' structure contains a name, the light position to be used when projecting the shadow and an array of 'sector reference' structures. The 'sector reference' structure contains the name of a sector within the set as well as the shadow color to be used when a shadow is projected onto that sector.

Once I discovered this data, it was fairly easy to enable the pre-existing planar projection shadow rendering code in ResidualVM also for EMI. The main difficulty was that previously the renderers only supported one global shadow color, but in EMI the shadow color can be set per-sector. I'll discuss this issue and how I resolved it in detail in the next blog post.

With all the issues resolved, it's very hard to tell ResidualVM and the original game apart in most scenes. The screenshots below are from ResidualVM. Compare them with the screenshots of the original above :)

Head tracking

For the past week I've been working on enabling head tracking. Head tracking allows actors to orient their head towards any point in the game world. This work is now included in PR #947.

The head tracking works by reorienting one of the bones of the skeleton after keyframe animation is applied. The bone is oriented to face the direction the actor wants to look towards. The look direction is interpolated, so the change in orientation is not instant.

The original Lua code uses the functions ActorSetHead and ActorSetHeadLimits to set up head tracking. The former takes as parameter the name of the bone that should be animated, while the latter takes three parameters that specify horizontal and vertical angle limits. The angle limits ensure that the head won't be oriented unrealistically.

My current implementation looks good in most cases, but I did find one case that looks noticeably different from the original. Mr. Cheese's head is at a weird angle when he looks towards Guybrush in the Scumm Bar. The screenshot on the left is ResidualVM, while the one on the right is the original.



The head tracking animates Mr. Cheese's neck bone, but the neck bone has children which are animated by keyframe animation, and the vertices of the head mesh follow the child bones. Although the neck bone is facing Guybrush, the head mesh is not, because of the animation applied to the child bones.

I spent a significant amount of time trying to figure out the difference between my implementation and the original during the week, but none of my solutions produced the same result as the original in this case.

One of my attempts was to override the animation of the neck bone as well as it's children bones completely whenever head tracking is active. This way the head was oriented exactly towards the point the actor looks at. That, however, didn't match the original game either. Mr. Cheese's head does seem to be affected by the animation also in the original.

For now I decided to leave this be. Mr. Cheese is currently the only case I'm aware of where the result is different from the original. I may return to this at a later stage once I've finished the other remaining tasks.