sky_line/Slipstream

Fork 1

GPU hang (device lost) in the RT reflection trace on async compute, twice ~1300 s into a session #1508

New issue

Open

opened 2026-11-09 03:19:54 +00:00 by TheSlowly · 2 comments

TheSlowly commented

2026-11-09 03:19:54 +00:00

Collaborator

Two GPU hangs (TDR, VK_ERROR_DEVICE_LOST from vkWaitForFences, no fault address) tonight on a test build of main e6107ec629 + our open PRs; the viewer logs out after each. Both reports attached.

	1	2
When (UTC)	02:54:57	03:16:58
Since start	1297 s (frame 33137)	1302 s (frame 23819)
Window	background for 168 s	focused

Same breadcrumbs both times (NV checkpoints):

graphics: TOP g_gi_window, BOTTOM g_gi_window   (waiting)
async:    TOP c_defer_rtrefl_trace, BOTTOM c_gi_hitmat

So the async queue started c_defer_rtrefl_trace (the RT reflection trace) and never finished it; graphics sat after g_gi_window. The #1049 / #1128 hang guards report nothing cut (0 rays budget-cut, 0 refused, 0 candidate-budget walks); the ray list for the frame is bounded (7124 rays, 112 groups). Registry: VisQueue 0, RT reflections 1 (split 0), GI 1 (split kernels 1, shade split 1).

Both at ~1300 s after start with different frame counts: possibly something time-based, possibly chance.

GPU: RTX 5070 Ti, NVIDIA 610.88. Settings of note: RenderVKRayTracing=1, RenderVKUpscaler=8 with RenderVKDlssRayReconstruction=2, async compute on, RenderVKGIEarlyTrace=1, RenderFarClip=4096, BackgroundFPSLimit=25.

The previous test build (main fb2032d7a4 + the same PRs) had no device loss; the last one before tonight was 2026-10-16. Renderer changes on main since fb2032d7a4 that touch the RT scene or reflections: 7a6eb2c9ac (RenderVKBlendHitLighting), 0ce20c1493 (inverted-hull outlines leave RT), a5a85effaf (rigged pose skip capped by footprint), 10248bab42 / e3810729d0 (forward GI fallback). Our side: #1472 (rigged tint cull in llcleanrtvk.cpp) was rebased onto 0ce20c1493 for this build, so the two RT-scene culls now meet in the same place; worth ruling out too.

Next: rerun with one of these off to bisect (RenderVKBlendHitLighting=0, then RenderVKAsyncCompute=0 to see if the hang moves to the graphics queue) and see whether it repeats near 1300 s.

Two GPU hangs (TDR, `VK_ERROR_DEVICE_LOST` from vkWaitForFences, no fault address) tonight on a test build of main e6107ec629 + our open PRs; the viewer logs out after each. Both reports attached. | | 1 | 2 | |---|---|---| | When (UTC) | 02:54:57 | 03:16:58 | | Since start | 1297 s (frame 33137) | 1302 s (frame 23819) | | Window | background for 168 s | focused | Same breadcrumbs both times (NV checkpoints): ``` graphics: TOP g_gi_window, BOTTOM g_gi_window (waiting) async: TOP c_defer_rtrefl_trace, BOTTOM c_gi_hitmat ``` So the async queue started `c_defer_rtrefl_trace` (the RT reflection trace) and never finished it; graphics sat after `g_gi_window`. The #1049 / #1128 hang guards report nothing cut (0 rays budget-cut, 0 refused, 0 candidate-budget walks); the ray list for the frame is bounded (7124 rays, 112 groups). Registry: VisQueue 0, RT reflections 1 (split 0), GI 1 (split kernels 1, shade split 1). Both at ~1300 s after start with different frame counts: possibly something time-based, possibly chance. GPU: RTX 5070 Ti, NVIDIA 610.88. Settings of note: RenderVKRayTracing=1, RenderVKUpscaler=8 with RenderVKDlssRayReconstruction=2, async compute on, RenderVKGIEarlyTrace=1, RenderFarClip=4096, BackgroundFPSLimit=25. The previous test build (main fb2032d7a4 + the same PRs) had no device loss; the last one before tonight was 2026-10-16. Renderer changes on main since fb2032d7a4 that touch the RT scene or reflections: 7a6eb2c9ac (RenderVKBlendHitLighting), 0ce20c1493 (inverted-hull outlines leave RT), a5a85effaf (rigged pose skip capped by footprint), 10248bab42 / e3810729d0 (forward GI fallback). Our side: #1472 (rigged tint cull in llcleanrtvk.cpp) was rebased onto 0ce20c1493 for this build, so the two RT-scene culls now meet in the same place; worth ruling out too. Next: rerun with one of these off to bisect (RenderVKBlendHitLighting=0, then RenderVKAsyncCompute=0 to see if the hang moves to the graphics queue) and see whether it repeats near 1300 s.

gpu_lost_1_0254Z.txt

22 KiB

gpu_lost_2_0316Z.txt

22 KiB

gpu_lost_3_0329Z.txt

22 KiB

TheSlowly added the

rendering

bug

labels

2026-11-09 03:19:54 +00:00

TheSlowly added this to the Slipstream project

2026-11-09 03:19:54 +00:00

TheSlowly commented

2026-11-09 03:30:59 +00:00

Author

Collaborator

Third one, same signature (async TOP c_defer_rtrefl_trace, BOTTOM c_gi_hitmat; graphics after g_gi_window; TDR, no fault address), report attached:

03:29:47 UTC, 542 s after start (frame 14656), so not time-based; the ~1300 s twice was chance.
Window in the background for 27 s. That's 2 of 3 in the background (BackgroundFPSLimit=25).

The hung kernel is the split RT reflection TRACE (mRtSplitPipe[RTS_TRACE], clean_deferred_light.comp). Shader changes on main since fb2032d7a4 (the last build without hangs) in what it runs: 5426f54488 (shared RT light walk headers), 31270a9950 + f891ae0fee (own-fixture re-trace / skip walk in clean_vis_walk.glsl), 8ee88d7506 (GI hit side), 7a6eb2c9ac (blend hit lighting), cf83137229 (#1493 water reflection troughs), 3d5e9f57bd (positional shadow facing cull). The walks look bounded on paper (<= 3 re-traces, candidate budgets), but the #1049 / #1128 guards counted nothing cut in any of the three.

Workaround while this is open: RenderVKRTReflections=0 (probe fallback) should keep the trace from running; we'll report whether it holds.

Third one, same signature (async TOP `c_defer_rtrefl_trace`, BOTTOM `c_gi_hitmat`; graphics after `g_gi_window`; TDR, no fault address), report attached: - 03:29:47 UTC, **542 s** after start (frame 14656), so not time-based; the ~1300 s twice was chance. - Window in the background for 27 s. That's 2 of 3 in the background (BackgroundFPSLimit=25). The hung kernel is the split RT reflection TRACE (`mRtSplitPipe[RTS_TRACE]`, clean_deferred_light.comp). Shader changes on main since fb2032d7a4 (the last build without hangs) in what it runs: 5426f54488 (shared RT light walk headers), 31270a9950 + f891ae0fee (own-fixture re-trace / skip walk in clean_vis_walk.glsl), 8ee88d7506 (GI hit side), 7a6eb2c9ac (blend hit lighting), cf83137229 (#1493 water reflection troughs), 3d5e9f57bd (positional shadow facing cull). The walks look bounded on paper (<= 3 re-traces, candidate budgets), but the #1049 / #1128 guards counted nothing cut in any of the three. Workaround while this is open: RenderVKRTReflections=0 (probe fallback) should keep the trace from running; we'll report whether it holds.

TheSlowly added the

crash

priority

high

labels

2026-11-09 03:35:26 +00:00

TheSlowly commented

2026-11-09 18:13:46 +00:00

Author

Collaborator

No hang since the third one (2026-11-09 03:29 UTC):

One session ran with RenderVKRTHangBeacon=2 from 05:57 UTC for at least 1 h 40 min (until 07:36 at least; it later closed normally). RT reflections stayed on; c_defer_rtrefl_trace measured ~0.5 to 1.2 ms per frame (RenderVKGpuTimers), so the hangs were not a slow pass creeping up. For part of it RenderVKBlendHitLighting was off.
Two later sessions (17:32 to 17:46, 18:01 onward) without the beacon: no hang so far.

So it is rare enough that three in ~1.5 h last night looks like a bad spot or bad luck, or the beacon twin kernel hides it. A read of every loop the list-trace kernel reaches found them all capped (48 queries x 64 or 256 candidates per ray), which points at a single rayQueryProceedEXT traversal that never returns rather than a shader loop. Of the changes since fb2032d7a4, only 3d5e9f57bd changes what this kernel does: it adds the sun's facing cull to the reflection hit's lamp shadow rays (rtrProjRay). clean_rt_common.glsl:96-99 records a similar NVIDIA hang in this kernel family from a value-identical edit (690930d47c).

No hang since the third one (2026-11-09 03:29 UTC): - One session ran with `RenderVKRTHangBeacon=2` from 05:57 UTC for at least 1 h 40 min (until 07:36 at least; it later closed normally). RT reflections stayed on; `c_defer_rtrefl_trace` measured ~0.5 to 1.2 ms per frame (RenderVKGpuTimers), so the hangs were not a slow pass creeping up. For part of it RenderVKBlendHitLighting was off. - Two later sessions (17:32 to 17:46, 18:01 onward) without the beacon: no hang so far. So it is rare enough that three in ~1.5 h last night looks like a bad spot or bad luck, or the beacon twin kernel hides it. A read of every loop the list-trace kernel reaches found them all capped (48 queries x 64 or 256 candidates per ray), which points at a single `rayQueryProceedEXT` traversal that never returns rather than a shader loop. Of the changes since fb2032d7a4, only 3d5e9f57bd changes what this kernel does: it adds the sun's facing cull to the reflection hit's lamp shadow rays (`rtrProjRay`). clean_rt_common.glsl:96-99 records a similar NVIDIA hang in this kernel family from a value-identical edit (690930d47c).

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

sky_line/Slipstream#1508

No description provided.

Rows
Columns