RT shadows: shadow terminator artifacts on smooth-shaded low-poly mesh (triangles fill stepwise) -- Hanika offset for all RT shadow rays #664

New issue

Closed

opened 2026-10-11 04:00:16 +00:00 by sky_line · 2 comments

sky_line commented

2026-10-11 04:00:16 +00:00

Owner

Ray-traced shadows on smooth-shaded low-poly mesh (avatar legs, rigged bodies, sculpties) step along the triangles. Near the terminator, where the surface turns away from the light, whole triangles fill with shadow one at a time, from one side to the other. Dotted lines also appear along triangle edges from the soft-shadow samples. Seen in-world on a pony avatar's legs lit by an RT projector.

Cause: the shadow terminator problem

Shading uses the interpolated vertex normal, i.e. a smooth curve.
The shadow ray starts on the flat triangle. Near the terminator, a ray leaving one flat facet toward the light hits the neighbouring facet, which bulges above the flat plane, and reports shadow.
We already have sun-only mitigations: RenderVKSunTerminatorSoften (Chiang, Li & Burley 2019) and RenderVKRTShadowTerminatorFloor, which lifts soft-disc rays near the terminator. Projector RT shadows, and probably RT point/probe rays, appear to have neither.

Fix: Hanika, "Hacking the Shadow Terminator" (Ray Tracing Gems II, ch. 4)

Before tracing, move the shadow-ray origin from the flat triangle onto the smooth surface implied by the vertex normals (projection onto the tangent planes at the three vertices, blended barycentrically). It's cheap, needs no tuning, and is independent of light type. Apply it to sun / moon, projector and any other RT shadow rays (RT shadow mask, the lit pass's ray-query path, god rays if relevant), for both rigged (post-skin positions and normals) and static geometry. Keep the existing sun tweaks only where they still add something, and keep an A/B setting.

Related: #648 (AO facet lines from the per-facet geometric normal). It's the same smooth-vs-faceted mismatch, but in a different pass.

Ray-traced shadows on **smooth-shaded low-poly mesh** (avatar legs, rigged bodies, sculpties) step along the triangles. Near the terminator, where the surface turns away from the light, whole triangles fill with shadow one at a time, from one side to the other. Dotted lines also appear along triangle edges from the soft-shadow samples. Seen in-world on a pony avatar's legs lit by an RT **projector**. ## Cause: the shadow terminator problem - Shading uses the interpolated vertex normal, i.e. a smooth curve. - The shadow ray starts on the **flat** triangle. Near the terminator, a ray leaving one flat facet toward the light hits the neighbouring facet, which bulges above the flat plane, and reports shadow. - We already have sun-only mitigations: `RenderVKSunTerminatorSoften` (Chiang, Li & Burley 2019) and `RenderVKRTShadowTerminatorFloor`, which lifts soft-disc rays near the terminator. Projector RT shadows, and probably RT point/probe rays, appear to have neither. ## Fix: Hanika, "Hacking the Shadow Terminator" (Ray Tracing Gems II, ch. 4) Before tracing, move the shadow-ray origin from the flat triangle onto the smooth surface implied by the vertex normals (projection onto the tangent planes at the three vertices, blended barycentrically). It's cheap, needs no tuning, and is independent of light type. Apply it to sun / moon, projector and any other RT shadow rays (RT shadow mask, the lit pass's ray-query path, god rays if relevant), for both rigged (post-skin positions and normals) and static geometry. Keep the existing sun tweaks only where they still add something, and keep an A/B setting. Related: #648 (AO facet lines from the per-facet geometric normal). It's the same smooth-vs-faceted mismatch, but in a different pass.

sky_line added the

rendering

bug

labels

2026-10-11 04:00:16 +00:00

sky_line added this to the Slipstream project

2026-10-11 04:00:16 +00:00

sky_line commented

2026-10-11 04:00:16 +00:00

Author

Owner

Agent dispatched (Opus).

sky_line added the

status

in-progress

label

2026-10-11 04:00:16 +00:00