RT shadows: foliage shimmers under FSR with the half-res shadow mask; full res fixes it but costs ~1.3 GB at 4K #625

New issue

Open

opened 2026-10-10 03:37:42 +00:00 by sky_line · 0 comments

sky_line commented

2026-10-10 03:37:42 +00:00

Owner

Likely cause

The half-res mask is upsampled into the lighting pass with a depth-aware filter. On foliage whose detail is smaller than a mask texel, each frame's sub-pixel jitter changes which shell / blade wins a full-res pixel, while the shadow value comes from a mask texel that covered different geometry, so the per-pixel shadow flickers frame to frame and the temporal upscaler sees it as shimmer.

Options

Temporal super-resolution of the mask: trace at half res with a per-frame sub-pixel offset (checkerboard / jittered) and accumulate into a full-res result, the way FSR / DLSS upscale colour. Full-res quality, half-res rays.
Slimmer mask storage, so full res is affordable: 185 B per texel is a lot (history slots, RGBA32F depth+gradient, moments, ids, projector layers); pack depth and gradient as half floats, share history images, drop redundant copies with ping-pong, quantize visibility to 8-16 bits.
Better upsample: select the mask sample whose depth / normal best matches the full-res pixel (nearest-depth or bilateral with a normal term) and feed foliage pixels with a low-confidence upsample into the reactive mask instead of letting them shimmer.
Consider mask scale by tier: full res on cards with plenty of VRAM, half (with the above) elsewhere.

Related: #85 (RT denoiser), the grass shadow band (shadow-ray alpha mip mismatch, clean_shadow_rt.glsl:123), #615 / #614 (VRAM).

With ray-traced sun shadows, dense alpha-mask foliage (shell grass: stacked horizontal alpha planes with sparse dots) shimmers under FSR (and likely TAA) at the default half-resolution RT shadow mask (RenderVKRTShadowMaskScale 0.5). Setting the mask to full resolution (1.0) removes the shimmer, but at 4K the mask then takes ~1.34 GB of video memory (it is ~185 B per mask pixel across 13 images; 354 MB at half res). ## Likely cause The half-res mask is upsampled into the lighting pass with a depth-aware filter. On foliage whose detail is smaller than a mask texel, each frame's sub-pixel jitter changes which shell / blade wins a full-res pixel, while the shadow value comes from a mask texel that covered different geometry, so the per-pixel shadow flickers frame to frame and the temporal upscaler sees it as shimmer. ## Options - **Temporal super-resolution of the mask:** trace at half res with a per-frame sub-pixel offset (checkerboard / jittered) and accumulate into a full-res result, the way FSR / DLSS upscale colour. Full-res quality, half-res rays. - **Slimmer mask storage**, so full res is affordable: 185 B per texel is a lot (history slots, RGBA32F depth+gradient, moments, ids, projector layers); pack depth and gradient as half floats, share history images, drop redundant copies with ping-pong, quantize visibility to 8-16 bits. - **Better upsample:** select the mask sample whose depth / normal best matches the full-res pixel (nearest-depth or bilateral with a normal term) and feed foliage pixels with a low-confidence upsample into the reactive mask instead of letting them shimmer. - Consider mask scale by tier: full res on cards with plenty of VRAM, half (with the above) elsewhere. Related: #85 (RT denoiser), the grass shadow band (shadow-ray alpha mip mismatch, clean_shadow_rt.glsl:123), #615 / #614 (VRAM).

sky_line added the

labels

2026-10-10 03:37:42 +00:00

sky_line added this to the Slipstream project

2026-10-10 03:37:42 +00:00

sky_line referenced this issue