Stan's confusing coat

My recent sprite fixes PR includes a change that fixes the appearance of Stan (see the GitHub issue here). I found the method they used to produce the unmoving plaid effect quite interesting, so here's a quick summary.

This is the final composition for reference. Note Stan's trademark: the pattern on his jacket does not obey perspective. Although the effect looks simple, achieving this result with fixed-function OpenGL is actually not very straightforward. Let's see how this image is produced.

If we render the first background layer only, we can make an interesting observation. A rectangle covered with the check pattern of Stan's jacket is part of the background. It covers all of Stan's screen-space area. Stan moves back and forth during dialogue, but never leaves the area covered by this rectangle.

Here is a render with only the character models on a black background. We can immediately see that the jacket is not part of the model. Instead, Stan has a fully transparent texture where the jacket should be, so we can see through him. The transparent jacket is still drawn to the depth buffer, though.

If we combine the first background layer and the character models we get a result like above. Since Stan's model is partly transparent we can see the check-patterned background through him. Obviously something still needs to be done about the rest of the rectangle surrounding him.

Above is an additional render with the rest of the background layers enabled. The desk is part of a background layer that is drawn over Stan, covering his legs. This image is similar to what ResidualVM produced before my fixes.

Here's the trick. A sprite is rendered behind Stan, but over the background layer. I highlighted the sprite with red color in the above screenshot to make it stand out from the rest of the background. The sprite contains a chunk of the "real" background without the check-patterned rectangle. Since the sprite is drawn with depth test enabled, the sprite only covers Stan's surroundings but not Stan himself.

This sprite was previously not drawn at all in ResidualVM, so the check-patterned rectangle was not replaced by the correct background around Stan.

This method works nicely since the backgrounds are static and the camera never moves. Other than a bit of overdraw, there's also no additional rendering cost.

The most obscure bug fix yet

After finishing the lighting task, I had some time to focus on fixing bugs before moving on to my after mid-term tasks.

One of my GSoC mentors, chkr brought up a bug regarding the candles in Governor's mansion. In ResidualVM, the candle sprites didn't show up at all. Look at the chandelier in the screenshots below to see the difference. ResidualVM on the left, original on the right.

 This seemed simple enough to quickly fix before moving on to other tasks I thought, so I got to debugging.

When looking at the original Lua code that sets up the candles, my first observation was that the candles were first being attached to the chandelier actor and their position was set afterwards. Perhaps something was going wrong with the offsetting of attached actors even after JoseJX's recent fixes? This turned out to be a false lead when I noticed that no calls were being made to sprite drawing functions at all for the candle sprites. Even if the sprites were positioned out of view, the current ResidualVM implementation should attempt to draw them. Later it turned out that the candle actors were in fact in the correct positions.

In the EMI engine, sprites are components of chores. A chore is basically a track of instructions for the components that are executed in sequence. Only the components of active, playing chores are updated and drawn. Chores are components of costumes, which in turn are components of an actor. I found out the candle actors and their costumes were present like they should be, but the costumes were not playing any chores with sprite components, so no sprites were drawn.

This seemed weird to me. The Lua code is supposed to start the "flame12fps" chore for the candles, but the call to start the chore was never made to ResidualVM. The next step was to figure out why.

The original Lua only starts the chore in the case if no chores are already playing. But surely there shouldn't be any chores playing if the actor was just created? Well, it turned out that when the game is run in ResidualVM, there is.

By adding a breakpoint to the IsChorePlaying engine opcode that the check in Lua calls for each chore, I found out that the opcode returns true because the "wear_default" chore of the costume "dumbshadow" is playing. Because of this, the sprite animation is never started.

In the original engine the chore is not playing. This is easy to check by running custom Lua code with Ctrl+E after enabling the hidden debug functions in the game (I could explain how to do this in a later blog post). If the chore is forced to play, IsChorePlaying will also return true in the original engine, so there is no special handling in the engine to always return false for this particular chore.

Ok, so what is the "wear_default" chore of "dumbshadow" and why is it started then? The "dumbshadow" costume is used to produce fake shadow effects such as the ones that can be seen under Guybrush and Elaine in the above screenshot of the original game. The shadows currently don't show up in ResidualVM, but even if they did, we probably wouldn't want the candles to have shadows. Thus it seems likely that the correct behavior would be to not activate the shadow, like the original engine seems to do.

The Lua code that starts the shadow chore can be found in _actors.lua. The chore is started as soon as the actor is put in a set, if certain conditions are fulfilled. Among these conditions is one peculiar one. If the opcode GetActorPuckVector returns nil, the shadow won't be activated. When GetActorPuckVector is run on the candles in the original engine, the returned value is nil. In ResidualVM the opcode always returns a non-nil value.

Finally we're getting to the gist of the bug, after seemingly drifting so far. GetActorPuckVector is an opcode used to get the forward vector of an actor projected to the actor's current sector (or walkplane) to be used for movement. It appears that in the original engine the GetActorPuckVector opcode always return nil until a call is made to SetActorFollowBoxes, which constrains the actor's movement to the walkable sectors. Most likely the purpose of the nil-check is to only activate shadows for actors that are constrained to walkable sectors.

Replicating this behavior in ResidualVM's implementation of GetActorPuckVector fixes the candles, as expected. Phew!

The FASTDYN light mode

One more quick post about the lighting modes before I move on to other subjects.

Looking at the comparison shots in the previous post gave me an idea about the FASTDYN lighting mode. The FASTDYN lighting appears smoother when compared to NORMDYN, and it seems to also light the back side of models slightly. These properties reminded me of the Half Lambert lighting technique used in the Half-Life games.

I decided to quickly implement the technique to see how it compares with the original. Surprisingly in some cases the result is almost an exact match, but in others the shading seems too bright. For example, see Otis  in the screenshots below. Left is the original renderer (FASTDYN mode), and on the right is my implementation with the Half Lambert technique.

The lighting is close to the original when Otis is at about a 45 degree angle to the light coming from the windows. However, if we rotate Otis slightly this no longer is the case.

Otis is now at about a 90 degree angle to the light, but the shading seems much darker in the original. If we rotate Otis 45 degrees further, the lighting is once again very close to my Half Lambert implementation.

Otis is now facing away from the light coming from the direction of the windows.

This behavior seems fairly strange to me. There must be some additional term in the FASTDYN lighting model that I'm not aware of yet.

I may return to this at a later time, but right now I'm happy with my current implementation that matches the NORMDYN mode of the original game exactly. This was just recently merged to ResidualVM's master, so give it a try.

Lighting comparisons

I submitted a pull request for lighting in EMI. I'm currently well ahead of my GSoC schedule, so I now have some time for ironing out any last remaining issues related to animation and lighting implementation before moving on to post mid-term tasks. See my roadmap on the ResidualVM wiki.

Regarding lighting, I spent some time trying to figure out the FASTDYN lighting mode I discussed briefly in the previous post, but I wasn't able to replicate the look of the the original. The FASTDYN lighting seems less precise and was probably used only for performance optimization, so I decided it wasn't really worth it to spend more time on it. Modern machines can handle the more accurate per-vertex lighting calculations for all actors just fine, and at least to my eye the results look better.

Here are a few more comparison shots, this time without any cheating. Original on the left, ResidualVM on the right.

Mr. Cheese uses the FASTDYN lighting mode in the original, so minor differences can be seen above.

The lawyers use FASTDYN mode. There also seems to be a ResidualVM-specific bug visible here, the lawyers are missing their eyes!

The voodoo lady uses FASTDYN mode which causes minor differences. The lack of head turning is also quite distracting here. I'll work on that after the GSoC mid-term.

Lighting: first results

My work on animations has been merged back to ResidualVM's master branch, so I've now moved on to the next big thing on my task list, which is lighting. So far I have a basic implementation working that supports ambient, point and directional lights. Below is a comparison showcasing the lighting in the original game and my implementation in ResidualVM. Can you spot the difference?

(To be completely honest I did cheat a bit in the above image, though. EMI has several different lighting modes that can be set per-actor. So far I've only focused on the LIGHT_NORMDYN mode, but in addition there is also LIGHT_FASTDYN, which is faster but the lighting is less accurate. Otis and Carla use the LIGHT_FASTDYN mode in the original so the lighting looks a bit different. For the screenshot of the original game I forced LIGHT_NORMDYN for Otis and Carla by executing custom Lua code in the original engine.)

My first attempt was to try to tweak the parameters of the standard OpenGL fixed function lighting model in order to match the lighting in the original EMI. I soon realized it was impossible to get an exact match with this approach, though. For example, EMI uses a simple attenuation model where each light has a minimum and a maximum falloff range. Vertices within the minimum range will be fully lit. Between the minimum and maximum range the light falls off linearly, and at the maximum range vertices will be unlit. OpenGL uses a different attenuation model that can only approximate the model used in EMI.

How did they do the lighting in the original game, then? Back then, GPUs with a programmable pipeline were not yet common, so shaders were out of question. Running apitrace on the original game reveals that no calls are made to glLight functions, so it looks like the original game does not make use of the fixed function lighting either. Instead, it seems the original game calculates shading in software, and the shading is then baked in to the vertex data that is passed on to OpenGL for rendering.

Isn't this inefficient? Yeah, but not really. The models in EMI are fairly simple, and there is only a few of them in view at once. Many actors use the LIGHT_STATIC lighting mode, which means that lighting only needs to be calculated once for them. Only a fraction of actors use the dynamic lighting mode, in which case the lighting must be recalculated each frame. On modern hardware the main bottleneck is perhaps the data transfer from CPU to GPU, as the vertex data needs to be updated every time the lighting changes.

I decided to implement a software shading solution similar to the original game, and the result can be seen in the image above. This is ideal for the software and fixed function OpenGL renderers in ResidualVM, but certainly isn't the best solution on modern hardware. For the modern shader-based OpenGL renderer it's a better idea to use a shader instead, at least for dynamically lit models.

My current lighting hacks can be found be found here.

Animation blending

In EMI, just like in many other 3D games the motion of a 3D model displayed on sceen may be a combination of several different keyframe animations playing simultaneously. For example, the animators may have supplied a 'walk' animation and a 'hold object' animation for a character in the game. The game engine may produce a 'walk while holding an object' animation by combining the lower body part of the 'walk' animation with the upper body part of the 'hold object' animation. This can be achieved with the animation priorities I described in the previous post. In addition, the engine can animate the transition between different animation states such as 'walk' and 'stand' by interpolating between them. This interpolation is what I refer to as animation blending.

With animation blending, the final animation that is displayed on the screen is the weighted sum of all simultaneously active animations. The weight of an animation (the "blend weight") determines how much contribution the animation has in the final result. The total sum of weights should equal 1. For example, we could animate the transition from 'stand' to 'walk', by linearly interpolating a value α from 0 to 1 over time and setting the blend weight of 'walk' to α and the blend weight of 'stand' to (1-α) at each step in time. The interpolated transition may not look completely realistic, but in most cases it looks much better than instantly snapping to another animation.

In EMI, the game may request certain animations to be faded in or faded out. To support this in ResidualVM, I store a 'fade' value for all active animations. When requested, the value is linearly interpolated between 0 and 1. If animations had equal priority, we could assign weight=fade for all animations and simply divide the intermediate weights by the total sum of weights to get the final normalized blend weights. However, with prioritized animations this changes a bit.

For higher priority animations we want to assign more weight than for lower priority animations. An animation with weight 100% will always completely override anything with a lower priority. If the animation has weight 60%, lower priority animations will only get to distribute the remaining 40% of weight among themselves.

How is this implemented? The way I'm doing it now is I first collect priority-specific accumulated contribution to animation "layers". Each animation layer contains the accumulated contribution of animation with a certain priority. For example, layer 0 contains the contribution of animation with priority 0, layer 1 contains the contribution of animation with priority 1, and so on. Within a layer we can assign weights for the animations in the simple fashion described before, with the exception that we'll only divide the weights by the total sum if the sum exceeds 1. I also assign a fade value to animation layers, which is simply the sum of weights.

Once the animation contribution is split into layers, we'll get the final result by blending the layers together. The blend weights are assigned so that the highest priority animation will contribute the most, and the remaining weight is distributed to lower priority animations. To be exact, the layer weights are calculated as follows for n layers:

weight_n = fade_n
weight_n-1 = fade_n-1 * (1 - fade_n)
weight_n-2 = fade_n-2 * (1 - fade_n-1) * (1 - fade_n)
...
weight_1 = fade_1 * (1 - fade_2) * ... * (1 - fade_n-1) * (1 - fade_n)

The end result is a system where each animation can be independently faded in and out, while still respecting animation priorities.

Animation progress

The Summer of Code is finally here, and the first week is already almost over! I'm sorry that I haven't updated the blog more, but I've been busy working on EMI's animations this week and I've made some excellent progress.

Some background: right now there is a basic implementation of EMI animations in ResidualVM, but the implementation is still far from perfect. I'm currently focusing on two major issues that I've identified. Firstly, the current implementation prioritizes animations in the order in which they were started, so the animation that is started last always overrides any previously applied animation (although there are specialized workarounds for some cases). Secondly, unlike in the original game, all animation transitions are instant due to the lack of animation blending. In this post I'm focusing mainly on the prioritization of animations.

If play order shouldn't matter, then how should it be chosen which animation takes precedence over another? We could get some ideas by looking at Grim Fandango, since it is based on an older version of the same engine. In Grim, each keyframe animation contains a priority field (actually two of them, but let's keep it simple) that control the order in which animations are applied. A higher priority animation always takes precedence over a lower priority one.

One might ask what happens if two animations with the same priority play at the same time. The answer is they are blended together, and the result is an average between the two animations. Again, the result is the same regardless of the order in which the animations were applied. I'll describe blending in more detail in a later post.

Knowing how the priorization of animations was done in Grim, the first thing I did of course was to try to find a similar priority field in EMI's .animb animation data format. Perhaps unsurprisingly, it turns out there is one! However, the catch is that the priority field in EMI is bone-specific. In other words, an animation may specify a different priority value for each bone that the animation is affecting. (Note: EMI uses skeletal animation.)

For example, we could have an animation of Guybrush waving both of his arms. For the left hand the priority value could be 1, and for the right hand it could be 3. In addition, we could have a standing idle animation with a constant priority of 2 for all bones. Now, if both of these animations were applied at the same time, the result would be that Guybrush would wave his right arm, but the rest of his body would follow the standing idle animation.

Typical real priority values I've seen so far are 0 for base poses like standing idle, 1 for walk and run, 2 for holding an object in hands, and so on.

Without animation blending, adding support for the priority value is fairly straightforward. When applying the animation to the bones of the skeleton, we can keep track of the currently applied priority for each bone. If the animation's priority is higher than the bone's current priority, the animation replaces the current bone transform completely. Otherwise the animation is skipped. Using this method we can apply the animations in arbitrary order and the highest priority animation is always displayed. Of course this simplistic approach is still dependent on the order in which the animations are applied in the case where animation priorities are equal.

Things start to get more complicated once animation blending is added into the mix, though. With blending, a higher priority animation may not completely replace a lower priority one if it has a blend weight lower than 100%. In that case some of the lower priority animation will "show through". In order to implement this properly animations can no longer be applied in arbitrary order. Instead, we need to calculate the transformation for each bone by applying the animations in descending priority order, taking the blend weights into account at each step. This is complicated by the fact that since priorities are bone-specific, the order in which animations should be applied may be different for each bone.

I'll go into details on how I solved this in the next post. In the meantime you can check out my progress at https://github.com/Akz-/residual/commits/animations.

Surprise GIF