Move projector-shadow ray queries out of the main lit shader #849

New issue

Open

opened 2026-10-18 21:26:49 +00:00 by sky_line · 1 comment

sky_line commented

2026-10-18 21:26:49 +00:00

Owner

Deferred until after 1.3. Tracking only for now.

Problem

With RT on, the main lit fragment shader gets a ray-query variant (lit rayquery, about 186k SPIR-V words) so it can trace projector shadows (and sun/moon when RT shadows are on) inline. That makes it the most expensive pipeline we compile, by a wide margin, on drivers without hardware BVH traversal:

AMD (RDNA2, Windows proprietary driver): ray-query traversal is shader code that gets inlined at every ray-query site. In one report the lit ray-query pipelines peaked at 9-11 GB of compiler memory each (about 3 GB attributed per pipeline) and took 15-16 s each to compile.
Mesa (Intel ANV, RADV): large uber-shaders are expensive to compile as well (SIMD-width variants, whole-function passes). Compile memory on a 16 GB Arc machine was enough to get the viewer OOM-killed (now mitigated by the 25% compile-memory cap, 3df839bf30).
NVIDIA: hardware traversal makes the variant cheap there, which is why this didn't show up in our testing.

The variant is also another full specialization set of the biggest shader (plain + ray-query, for the rigged, capture and particle lit pipelines too), which doubles compile time and cache size for those.

Proposal

Trace projector shadows the same way sun/moon RT shadows already work: in a separate compute pass (like the RT shadow mask) that writes per-pixel visibility, which the lit shader then samples. The lit shader then needs no ray-query variant at all.

One mask per projector is too much. Options to evaluate: a packed per-pixel visibility for the top N projectors affecting each tile (the light clustering already selects these), or a per-light visibility atlas at reduced resolution plus the existing denoiser.
Particles (PartLightRt) and the capture pass (FragCapRt) have their own ray-query variants. Decide whether they read the same mask, keep shadow maps, or keep a small dedicated ray-query path.
Bonus: projector RT shadows get the RT shadow mask's temporal denoiser instead of the single hard trace per pixel.

Measure first

Before designing it, rank pipelines by learned compile cost per driver (vkpipecache learned costs, VKPipe batch reports) to confirm the lit ray-query variants dominate on AMD and Mesa, and check the per-frame cost of a separate projector trace pass against the inline trace on NVIDIA and on RDNA2.

**Deferred until after 1.3.** Tracking only for now. ## Problem With RT on, the main lit fragment shader gets a ray-query variant (lit `rayquery`, about 186k SPIR-V words) so it can trace projector shadows (and sun/moon when RT shadows are on) inline. That makes it the most expensive pipeline we compile, by a wide margin, on drivers without hardware BVH traversal: - **AMD (RDNA2, Windows proprietary driver):** ray-query traversal is shader code that gets inlined at every ray-query site. In one report the lit ray-query pipelines peaked at 9-11 GB of compiler memory each (about 3 GB attributed per pipeline) and took 15-16 s each to compile. - **Mesa (Intel ANV, RADV):** large uber-shaders are expensive to compile as well (SIMD-width variants, whole-function passes). Compile memory on a 16 GB Arc machine was enough to get the viewer OOM-killed (now mitigated by the 25% compile-memory cap, 3df839bf30). - **NVIDIA:** hardware traversal makes the variant cheap there, which is why this didn't show up in our testing. The variant is also another full specialization set of the biggest shader (plain + ray-query, for the rigged, capture and particle lit pipelines too), which doubles compile time and cache size for those. ## Proposal Trace projector shadows the same way sun/moon RT shadows already work: in a separate compute pass (like the RT shadow mask) that writes per-pixel visibility, which the lit shader then samples. The lit shader then needs no ray-query variant at all. - One mask per projector is too much. Options to evaluate: a packed per-pixel visibility for the top N projectors affecting each tile (the light clustering already selects these), or a per-light visibility atlas at reduced resolution plus the existing denoiser. - Particles (`PartLightRt`) and the capture pass (`FragCapRt`) have their own ray-query variants. Decide whether they read the same mask, keep shadow maps, or keep a small dedicated ray-query path. - Bonus: projector RT shadows get the RT shadow mask's temporal denoiser instead of the single hard trace per pixel. ## Measure first Before designing it, rank pipelines by learned compile cost per driver (`vkpipecache` learned costs, VKPipe batch reports) to confirm the lit ray-query variants dominate on AMD and Mesa, and check the per-frame cost of a separate projector trace pass against the inline trace on NVIDIA and on RDNA2.

sky_line added the

labels

2026-10-18 21:26:49 +00:00

sky_line added this to the Slipstream project

2026-10-18 21:26:49 +00:00

sky_line referenced this issue

2026-10-19 02:26:45 +00:00

Hybrid reflections: ray-traced fallback where SSR misses (off-screen / occluded content) #859