Device lost: GPU hang in the RT reflection trace (async compute), seen while toggling reflection probes #1049

New issue

Open

opened 2026-10-25 05:07:03 +00:00 by sky_line · 1 comment

sky_line commented

2026-10-25 05:07:03 +00:00

Owner

Recurring. The graphics device is lost through a GPU hang (TDR), not a page fault: the device-fault report says "no fault address (instruction pointers only): GPU hang / timeout".

Breadcrumbs:

Async compute queue: last started c_defer_rtrefl_trace, last finished c_defer_rtrefl_sel, so the RT reflection trace never completed.
Graphics queue: waiting at pr_early.

Setup: dev 1.5 (b2761279e3), RTX 5090, NVIDIA 615.78.08, Linux, DLSS Ray Reconstruction on (RenderVKUpscaler 11), RT on, ReflectionProbeDetail 3, about 5 minutes into the session, VRAM plentiful.

Context: the owner was turning reflection probes on and off over a few minutes, though nothing changed in the moments right before the hang. RT reflection misses sample the probes, so a probe resource or cluster table that is stale or half-rebuilt while the reflection trace reads it (for example, a probe list walk with a garbage count) is a prime suspect. So is an invalid ray or a non-terminating walk in the reflection trace, the same family as the Xid 109 fix (ab53e12cc9).

Done when: the hang cause is found and fixed. The probe toggle stress (probes on/off and detail changes during RT reflections) and long sessions run without device loss, clean under sync validation.

**Recurring.** The graphics device is lost through a GPU hang (TDR), not a page fault: the device-fault report says "no fault address (instruction pointers only): GPU hang / timeout". **Breadcrumbs:** - Async compute queue: last started `c_defer_rtrefl_trace`, last finished `c_defer_rtrefl_sel`, so the RT reflection trace never completed. - Graphics queue: waiting at `pr_early`. **Setup:** dev 1.5 (b2761279e3), RTX 5090, NVIDIA 615.78.08, Linux, DLSS Ray Reconstruction on (RenderVKUpscaler 11), RT on, ReflectionProbeDetail 3, about 5 minutes into the session, VRAM plentiful. **Context:** the owner was turning reflection probes on and off over a few minutes, though nothing changed in the moments right before the hang. RT reflection misses sample the probes, so a probe resource or cluster table that is stale or half-rebuilt while the reflection trace reads it (for example, a probe list walk with a garbage count) is a prime suspect. So is an invalid ray or a non-terminating walk in the reflection trace, the same family as the Xid 109 fix (ab53e12cc9). **Done when:** the hang cause is found and fixed. The probe toggle stress (probes on/off and detail changes during RT reflections) and long sessions run without device loss, clean under sync validation.

sky_line added the

rendering

bug

labels

2026-10-25 05:07:03 +00:00

sky_line added this to the Slipstream project

2026-10-25 05:07:04 +00:00

sky_line commented

2026-10-25 05:30:19 +00:00

Author

Owner

Investigation result (dev commit f0e2c154a2): root cause not proven. The hang path is now strictly bounded and instrumented.

Proven from the log and code:

The hung kernel is the fused RT reflection list trace (split mode was off). The forward pass right after it shares the same checkpoint and is also covered.
Every loop in that path is already bounded by a constant or a clamped count.
Acceleration-structure content was valid at the loss (TLAS→BLAS OK, no non-finite transforms).
Probe detail never changed during the session, and turning probes off only writes a header count.

New lead (inferred): the render resolution flipped between 1280x696 and 3840x2088 six times in the 2.5 minutes before the hang, the last one 63 s before. That was the upscaler being switched for the #1051 test. The lifetime of the RT tile / ray lists across those re-creations wasn't audited and is the next suspect.

Added:

A stricter rtRaySane: world-bounded origin, finite direction and tmax.
A per-ray ray-query budget of 48 in the reflection kernels.
Always-on counters for budget cuts, refused rays and candidate-budget overruns. They warn on first occurrence, appear in the VKLitGuard summary and in the DEVICE_LOST report.

The next occurrence should therefore say which guard fired. release/1.4 shares the same path; a cherry-pick needs small context fixes.

Investigation result (dev commit f0e2c154a2): **root cause not proven.** The hang path is now strictly bounded and instrumented. **Proven from the log and code:** - The hung kernel is the fused RT reflection list trace (split mode was off). The forward pass right after it shares the same checkpoint and is also covered. - Every loop in that path is already bounded by a constant or a clamped count. - Acceleration-structure content was valid at the loss (TLAS→BLAS OK, no non-finite transforms). - Probe detail never changed during the session, and turning probes off only writes a header count. **New lead (inferred):** the render resolution flipped between 1280x696 and 3840x2088 six times in the 2.5 minutes before the hang, the last one 63 s before. That was the upscaler being switched for the #1051 test. The lifetime of the RT tile / ray lists across those re-creations wasn't audited and is the next suspect. **Added:** - A stricter rtRaySane: world-bounded origin, finite direction and tmax. - A per-ray ray-query budget of 48 in the reflection kernels. - Always-on counters for budget cuts, refused rays and candidate-budget overruns. They warn on first occurrence, appear in the VKLitGuard summary and in the DEVICE_LOST report. The next occurrence should therefore say which guard fired. release/1.4 shares the same path; a cherry-pick needs small context fixes.

sky_line referenced this issue from a commit

2026-10-25 06:21:59 +00:00

#1049: RT reflection hang guard -- per-ray ray-query budget, world-bounded rays, always-on counters

sky_line added the

status

testing

label

2026-10-25 06:22:07 +00:00

sky_line referenced this issue from a commit

2026-10-25 09:08:08 +00:00

#1039: visibility queue shaders (M2): every hit ray leaves the lighting kernels

sky_line referenced this issue from a commit