Anti-Ghosting with Temporal Anti-Aliasing

Game developers like to customize their technology to deliver the experiences they want to share with players. The Grand Tour Game was no exception. Starting with many of Lumberyard’s rendering features, we added new innovations to the engine to achieve the results we wanted. You can download Lumberyard here and try implementing this technique yourself.

For The Grand Tour Game, making the show’s beautiful vehicles the star of the game was a chief concern for the art team. Because vehicles tend to have highly curved surfaces and high reflectance, they introduce additional rendering challenges, such as specular and edge aliasing. We decided that temporal anti-aliasing (TAA) would be the best solution to solve these aliasing issues, and Lumberyard already supports TAA. Unfortunately, we encountered a lot of ghosting as a result of using TAA with a fast moving camera. Ghosting occurs when portions of a moving object across multiple frames combine incorrectly to give the appearance of a second copy of the object. Through a combination of new techniques that we developed for The Grand Tour Game and previous advances by other studios and researchers, we were able to eliminate most ghosting without sacrificing performance.

We decided on TAA for The Grand Tour Game because it tends to produce a softer, more photorealistic image in both static and moving scenes. FXAA (Fast Approximate Anti-Aliasing) and SMAA (Subpixel Morphological Anti-Aliasing) work well for static scenes, but still produce artifacts for moving scenes. Lumberyard’s deferred lighting pipeline does not support MSAA (Multisample Anti-Aliasing). Like MSAA, TAA uses multiple samples per pixel to provide anti-aliasing. The difference is that with temporal anti-aliasing, the samples are spread across multiple frames. It uses a frame history buffer and a per-pixel velocity buffer to reproject each pixel to gather the additional sample. For each pixel, we use the per-pixel velocity as an offset, as well as the previous frame’s view projection matrix, to determine where to query the frame history buffer. Modifying the camera’s projection matrix with a sub-pixel jitter each frame allows us to produce anti-aliased results even in scenes where there is no camera motion. With fast rotation or linear motion, the history pixel (the sample retrieved from the frame history buffer after pixel reprojection) may correspond to a location with vastly different lighting conditions or to an entirely separate object. This history mismatch, if unaddressed, causes severe ghosting, as shown below.

Neighborhood clamping is one way to mitigate this artifact. For each current frame pixel, neighborhood clamping determines the minimum and maximum value per color channel for its 3×3 pixel neighborhood, and rejects history contribution if the history pixel as a channel with intensity that is outside of that range.

This clamping helps alleviate ghosting for certain surfaces as seen above, but noisy surfaces will have a neighborhood range that nearly spans the entire color range which makes the clamp ineffective. Many of the road surfaces in the The Grand Tour Game are noisy asphalt, so this issue occurred frequently with both linear motion and camera rotation. In the images below, note the ghosting behind the moving car and to the left of the car when the camera rotates:

By visualizing the neighborhood clamping (below, red pixels are rejecting history due to neighborhood clamping), we can see that much of it occurs behind the vehicle – the 3×3 neighborhood defines a narrow enough range. The asphalt has some clamping, but it is much less common. This is because the asphalt is noisier and more likely to accept pixel histories erroneously, causing the ghosting seen above.

For Uncharted 4, Naughty Dog introduced a technique to help mitigate this kind of ghosting (see “Temporal Antialiasing in Uncharted 4” from SIGGRAPH 2016). Their technique uses the stencil buffer to tag key objects such as characters, and then disables history blending if the current frame and history pixels have different stencil values. Additionally, a 3×3 Gaussian blur is used to help avoid oversharpening from newly revealed pixels that have no other samples to average.

Even though this technique adds two additional texture fetches and some instructions to the TAA shader to perform the blur, the initial results were very promising. Additionally, we already perform a stencil resolve pass, so generating the stencil buffers were very cheap, aside from the extra memory cost of storing a history stencil buffer. For our usage, we tag all materials that constitute the vehicles in our stencil buffer:

This technique nearly eliminates the ghosting caused by camera rotation. In the animation below, red indicates pixels that reject history contribution due to the stencil values for the current frame and history frame differing. If you compare this to the earlier animation of the ghosting from camera rotation, you will notice that the pixels that were ghosting are fully within the area that is being discarded now.

While Naughty Dog’s technique largely solves ghosting caused by camera rotation around key objects, ghosting can still occur for fast linear motion. In The Grand Tour Game, shadows behind or underneath the car still produced ghosting artifacts. The stencil testing correctly determined that neither the current nor history pixels belong to the car, but the history pixel is in shadow while the current pixel is in direct light. This caused the edge of the car’s shadow to produce ghosting, as seen below:

One solution would be to add car shadows to the stencil test. However, this would be overly invasive to an otherwise unrelated rendering system, may have negatively impacted framerate, and likely wouldn’t be a full solution, since shadows may not be the only remaining contributor to severe ghosting. To kick off an alternate solution, we worked with the art team to establish the tenets for each situation:

1. If the player vehicle is stationary, the entire frame should appear smooth and anti-aliased.
2. If the player vehicle is moving, the player vehicle and mid-range to distant environment should appear smooth and anti-aliased.
3. If the player vehicle is moving, the near environment and road should not reveal any ghosting.

Based on these criteria, we developed a new anti-ghosting TAA technique that uses a blend of pixel depth and pixel motion to determine how to blend the pixel history.

To begin, any pixel closer to the camera than a specified depth threshold is considered for disabling blending with history pixels. We chose our depth threshold empirically such that the entire player car and a short length of road behind it would be within the range. This satisfies the most common scenarios where ghosting occurred – player motion and motion of a competing car that the player is approaching. We cannot simply disable history blending outright for pixels in this depth range, as it causes artifacts as seen below due to the sub-pixel camera jitter:

As we already have pixel velocity as a shader input, we leverage it to determine if a pixel is moving above a speed threshold from one frame to the next. For static or slow moving pixels, we continue using TAA as normal. As pixel motion increases, we can begin rejecting the history contribution because the jitter is not as noticeable while moving, and other post effects such as motion blur help to alleviate artifacts. As we always want the player vehicle to look great, one final adjustment is made – if the current frame pixel is stencil tagged, do not consider this anti-ghosting technique.

This new technique has little to no impact on performance – we already compute most of what was needed to find the history location, so this only costs us a few additional shader instructions. The below code sample is the crux of the technique, but other changes had to be made to shader setup and execution outside of this snippet.

half currentStencil = tex2D(StencilCurrentSrv, screenUV).r;
half historyStencil = tex2D(StencilHistorySrv, historyUV).r;
float pixelMovement = length(motionVector * PerFrame_ScreenSize.xy);

bool bStencilDifferent = (currentStencil != historyStencil);
if (AntiGhostingMethod >= c_AntiGhostingComplex) // complex = 2
{
    // Complex anti-ghosting is based on stencil tagging, pixel depth, and pixel motion. This // greatly mitigates ghosting from both camera rotation and fast camera motion. Key // objects (such as character models) are tagged in their material and get marked in the // stencil buffer. If the current and history stencils differ, then reject the history as // the pixel is newly uncovered. If the pixel is not marked (meaning it is typically // environment such as roads or terrain) and the pixel has moved above a threshold, perform // depth-based anti-ghosting to mitigate ghosting near the camera where it is most // egregious. This motion threshold should be kept relatively low, as if it is too high, // you will notice the TAA history blend pop out from behind the car as the camera speeds // up or slows down. A value of 1 seems good, 5 is too high.
    float worldDepth = current.depth * PerView_NearFarClipDist.y;
    bool bStencilDepthMotion = (currentStencil == 0 && pixelMovement > MotionThresholdMin
        && worldDepth < DepthThresholdMax);
    if (!bStencilDifferent && bStencilDepthMotion)
    {
        // Lerp toward using full current frame data, using a smoothstep of a min/max depth. // This allows pixels near the camera to use full new frame weight if they passed the // above checks, but avoids a sharp cutoff when approaching standard history blending.
        float depthBlend = 1.0f - smoothstep(DepthThresholdMin, DepthThresholdMax, worldDepth);
        frameWeight = lerp(frameWeight, 1.0f, depthBlend);
        // To mitigate sharpness in pixels using frame weight closer to 1, lerp to a 3x3 // Gaussian blurred color of the current pixel neighborhood. Without this, newly // revealed or anti-ghosted pixels appear much sharper than they should.
        currentColor = lerp(currentColor, current.colorBlurred, depthBlend);
    }
}
if (AntiGhostingMethod >= c_AntiGhostingSimple) // simple = 1
{
    // Simple anti-ghosting is based on stencil tagging. This greatly mitigates ghosting from // camera rotation. Key objects (such as character models) are tagged in their material // and get marked in the stencil buffer. If the current and history stencils differ, // then reject the history as the pixel is newly uncovered.
    if (bStencilDifferent)
    {
        frameWeight = 1.0f;
        // To mitigate sharpness in newly revealed pixels, lerp to a 3x3 Gaussian blurred color // of the current pixel neighborhood. Without blurring, these pixels appear // much sharper than they should.
        currentColor = current.colorBlurred;
    }
}

To help illustrate the technique, the visualization below uses red to indicate pixels that are moving fast enough and are close enough to the camera to trigger the anti-ghosting. Note that since pixel motion is measured in screen space, pixels near the edges of the screen trigger anti-ghosting sooner than pixels in the center of the screen. This helps to break up the boundary between normal TAA and our anti-ghosting technique.

Since implementing this, there have been a few times where we’ve made custom adjustments to the minimum and maximum depth thresholds for anti-ghosting, based on where the player vehicle is located in cutscenes. In the example below, the camera was overhead and further away from the car than usual, causing noticeable ghosting behind the car. Fortunately, this data was already exposed outside of the shader, so it was simple to make adjustments for different cameras. Having a more holistic way to identify the focus of the camera and automatically adjust the depth thresholds would be beneficial.

Overall, the combination of Naughty Dog’s prior improvements and our new anti-ghosting temporal anti-aliasing exceeded our expectations. On Xbox One, this added about 0.1ms to our TAA shader for a total runtime cost of about 1.6ms on the GPU. See the results for yourself by checking out The Grand Tour Game, available now on PlayStation 4 and Xbox One.

About the author

Steve Karolewics is a Senior Graphics Engineer at Amazon Game Studios Seattle. He led the rendering team for The Grand Tour Game and worked on split-screen multiplayer, visual improvements, and performance optimizations.