GPU hang (Xid 109) in the RT reflection trace (c_defer_rtrefl_trace) -- persists past ab53e12cc9

sky_line commented

2026-10-27 05:22:44 +00:00

Owner

GPU hang (device lost) in the ray-traced reflection trace. Same signature as #995 / #996 / #997, and it persists past ab53e12cc9 ("RT reflections: GPU hang, Xid 109", rtRaySane: every rayQueryInitialize validated). Owner rule: this stays open until the cause is proven (a decisive repro + fix, not a plausible guess) -- it has survived several fixes.

Occurrence (owner, 2026-10-27 05:17:56 UTC): RTX 5090, Linux, NVIDIA 615.78.08, dev f3c24ffab5 (desktop, not VR). Camera very close to the own avatar's hair (static, unrigged, opaque mesh), pose tool open, night, rain. Not a page fault; VRAM 4.6 / 21.5 GB.

Kernel: NVRM: Xid 109, CTX SWITCH TIMEOUT (a hang, the TDR class).
VK_ERROR_DEVICE_LOST from vkWaitForFences (vk_swapchain.cpp:258); VK_EXT_device_fault: no fault address (instruction pointers only).
NV checkpoints, async compute queue: stage 0x1 last=c_defer_rtrefl_trace, stage 0x2000 last=c_defer_rtrefl_sel -- the RT reflection TRACE dispatch started and never finished (the mark is recorded after the trace dispatch).
Graphics queue: last=g_gi_window at both stages -- graphics was waiting at the GI early-trace join (asyncJoinUpTo) for the async segment; it is the waiter, not the hang.
Settings: RenderVKRayTracing=1 RenderReflectionProbeDetail=3 RenderShadowDetail=2 RenderVKSunShadows=2 RenderVKProjectorShadows=2 RenderVKUpscaler=11 RenderVKDlssRayReconstruction=1 RenderVKRenderScale=1 RenderVKTAA=0 RenderVKAsyncComputeQueue=1 RenderVKAsyncCompute=-1 RenderVKAsyncDeferredLighting=1 RenderVKGIEarlyTrace=1 RenderVKGILatencyFrame=0 RenderVKMirrors=1 RenderFarClip=4096 RenderQualityPerformance=6.

Related history: #995/#996/#997 (RTX 4070 Windows, breadcrumbs incl. g_defer_rtrefl, 2-3 s frames = TDR); triage notes cluster 4. The AMD RX 9070 XT RT-mask hang (#874/#889/#945/#973) is the same kind (ray-query dispatch that never returns) in a different pass.

Plan: root-cause pass on the reflection trace (kernel loops, indirect dispatch sizing / tile counts, any loop whose bound comes from data, what changed in this path since ab53e12cc9 incl. the visibility-queue work), a bounded-iteration guard that turns a runaway into a counted, logged event instead of a TDR, and a decisive repro. Close only with proof.

GPU hang (device lost) in the ray-traced reflection trace. Same signature as #995 / #996 / #997, and it persists past `ab53e12cc9` ("RT reflections: GPU hang, Xid 109", rtRaySane: every rayQueryInitialize validated). **Owner rule: this stays open until the cause is proven** (a decisive repro + fix, not a plausible guess) -- it has survived several fixes. **Occurrence (owner, 2026-10-27 05:17:56 UTC):** RTX 5090, Linux, NVIDIA 615.78.08, dev `f3c24ffab5` (desktop, not VR). Camera very close to the own avatar's hair (static, unrigged, opaque mesh), pose tool open, night, rain. Not a page fault; VRAM 4.6 / 21.5 GB. - Kernel: `NVRM: Xid 109, CTX SWITCH TIMEOUT` (a hang, the TDR class). - `VK_ERROR_DEVICE_LOST` from vkWaitForFences (vk_swapchain.cpp:258); VK_EXT_device_fault: no fault address (instruction pointers only). - NV checkpoints, async compute queue: stage 0x1 last=`c_defer_rtrefl_trace`, stage 0x2000 last=`c_defer_rtrefl_sel` -- the RT reflection TRACE dispatch started and never finished (the mark is recorded after the trace dispatch). - Graphics queue: last=`g_gi_window` at both stages -- graphics was waiting at the GI early-trace join (asyncJoinUpTo) for the async segment; it is the waiter, not the hang. - Settings: RenderVKRayTracing=1 RenderReflectionProbeDetail=3 RenderShadowDetail=2 RenderVKSunShadows=2 RenderVKProjectorShadows=2 RenderVKUpscaler=11 RenderVKDlssRayReconstruction=1 RenderVKRenderScale=1 RenderVKTAA=0 RenderVKAsyncComputeQueue=1 RenderVKAsyncCompute=-1 RenderVKAsyncDeferredLighting=1 RenderVKGIEarlyTrace=1 RenderVKGILatencyFrame=0 RenderVKMirrors=1 RenderFarClip=4096 RenderQualityPerformance=6. **Related history:** #995/#996/#997 (RTX 4070 Windows, breadcrumbs incl. `g_defer_rtrefl`, 2-3 s frames = TDR); triage notes cluster 4. The AMD RX 9070 XT RT-mask hang (#874/#889/#945/#973) is the same *kind* (ray-query dispatch that never returns) in a different pass. **Plan:** root-cause pass on the reflection trace (kernel loops, indirect dispatch sizing / tile counts, any loop whose bound comes from data, what changed in this path since ab53e12cc9 incl. the visibility-queue work), a bounded-iteration guard that turns a runaway into a counted, logged event instead of a TDR, and a decisive repro. Close only with proof.

sky_line added the

labels

2026-10-27 05:22:44 +00:00

sky_line self-assigned this

2026-10-27 05:22:44 +00:00

sky_line added this to the Slipstream project

2026-10-27 05:22:44 +00:00

sky_line referenced this issue

2026-10-27 05:22:44 +00:00

[viewer] [crash] Crash: first my crash on Jetstream #995

sky_line referenced this issue

2026-10-27 05:22:44 +00:00

[viewer] [crash] Crash: DLSS dergading #996

sky_line referenced this issue

2026-10-27 05:22:44 +00:00

[viewer] [crash] Crash: Crash again after settings changed #997

sky_line commented

2026-10-27 06:48:17 +00:00

Author

Owner

Second occurrence, different kernel (owner, 2026-10-27 06:47:35 UTC, RTX 5090 Linux, driver 615.78.08, dev 674219b7bb = the #1128 guard + ray-list header build, beacon OFF, RenderVKVisQueue 30 set live ~2 min earlier for an occupancy A/B):

Kernel: NVRM: Xid 109, CTX SWITCH TIMEOUT again; no fault address.
Async NV checkpoints: stage 0x1 last=c_defer_shadowsunwalk, stage 0x2000 last=c_vq_prod -- with VisQueue bit 16 the frame-level walk wave runs every queued consumer's walk back to back; the GPU hung in that walk phase (sun-shadow walk or the walks right after), NOT in the RT reflection list trace. Graphics again only waiting at g_gi_window (GI join).
Ray-list header (#1128 instrument): slots 0/1 bounded (~38.5k rays, 602-604 groups); the newest slot (the hung frame) "copy not executed" -- the reflection SEL had not run yet, consistent with the hang being earlier in the segment.
rtRaySane refused 0 rays (float form passed 0) in every guard window this session incl. the three rings at the loss -- no non-finite rays caught (the rings are the frames before the hung one).

Reading: two different ray-query kernels (reflection list trace; VQ shadow/visibility walk) have now hung the same way with no bad ray caught. Points at what they share -- the acceleration structures / driver traversal -- rather than one kernel's logic; the visibility queue (back-to-back walks, heavier async segment) makes it easier to hit. Next: extend the hang beacon to the VQ walk kernels so a hang in either names its stuck ray, and compare stuck rays' TLAS instance / region.

**Second occurrence, different kernel** (owner, 2026-10-27 06:47:35 UTC, RTX 5090 Linux, driver 615.78.08, dev `674219b7bb` = the #1128 guard + ray-list header build, beacon OFF, `RenderVKVisQueue 30` set live ~2 min earlier for an occupancy A/B): - Kernel: `NVRM: Xid 109, CTX SWITCH TIMEOUT` again; no fault address. - Async NV checkpoints: stage 0x1 last=`c_defer_shadowsunwalk`, stage 0x2000 last=`c_vq_prod` -- with VisQueue bit 16 the frame-level walk wave runs every queued consumer's walk back to back; the GPU hung in that walk phase (sun-shadow walk or the walks right after), NOT in the RT reflection list trace. Graphics again only waiting at `g_gi_window` (GI join). - Ray-list header (#1128 instrument): slots 0/1 bounded (~38.5k rays, 602-604 groups); the newest slot (the hung frame) "copy not executed" -- the reflection SEL had not run yet, consistent with the hang being earlier in the segment. - `rtRaySane refused 0 rays (float form passed 0)` in every guard window this session incl. the three rings at the loss -- no non-finite rays caught (the rings are the frames before the hung one). **Reading:** two different ray-query kernels (reflection list trace; VQ shadow/visibility walk) have now hung the same way with no bad ray caught. Points at what they share -- the acceleration structures / driver traversal -- rather than one kernel's logic; the visibility queue (back-to-back walks, heavier async segment) makes it easier to hit. Next: extend the hang beacon to the VQ walk kernels so a hang in either names its stuck ray, and compare stuck rays' TLAS instance / region.

sky_line referenced this issue from a commit

2026-10-27 11:44:08 +00:00

#1128 RT reflection hang: fold-proof ray guard (with its own proof counter) + a hang beacon that names the stuck ray

sky_line referenced this issue from a commit

2026-10-27 11:44:08 +00:00

#1128 hang beacon: a one-frame deliberate-hang self test to validate the instrument

sky_line referenced this issue from a commit

2026-10-27 11:44:08 +00:00

#1128 hang beacon on every ray-query kernel; shadow-core query budget; recording notes + vendor crash dump in the device-lost report

TheSlowly commented

2026-10-27 21:35:03 +00:00

Collaborator

Two more occurrences, same signature (TheSlowly, 2026-10-27; RTX 5070 Ti, Windows, NVIDIA 610.88; test build = main a152d3c727 + #1167 / #1168, neither touches RT):

	19:28:24 UTC	21:05:33 UTC
Window	background, 59 s	background, 199 s
Graphics NV checkpoints	`g_gi_window` / `g_gi_window`	`g_gi_window` / `g_gi_window`
Async NV checkpoints	`c_defer_rtrefl_trace` / `c_defer_rtrefl_sel`	`c_defer_rtrefl_trace` / `c_defer_rtrefl_sel`

Both times the viewer was unfocused (BackgroundFPSLimit=25, camera static). That's a pattern worth checking: it never happened while I was actively playing today.
21:05: TDR, no fault address. Every guard ring was clean (rtRaySane refused 0, query budget cut 0, candidate budget hit 0).
The ray list was tiny. The newest slot read 425 rays / 7 groups / 87 tiles (copy not executed for the hung frame); the slots before it were 431 and 414. So it hung on a ~430-ray dispatch, which looks like a traversal that never ends rather than too much work.
Settings: RenderVKRayTracing=1 RenderReflectionProbeDetail=3 RenderShadowDetail=2 RenderVKSunShadows=2 RenderVKProjectorShadows=2 RenderVKUpscaler=2 RenderVKDlssRayReconstruction=2 RenderVKAsyncComputeQueue=1 RenderVKAsyncCompute=-1 RenderVKAsyncDeferredLighting=-1 RenderVKGIEarlyTrace=1 RenderVKGILatencyFrame=0 RenderVKFramePipelining=0 RenderVKGodRays=3 RenderVKMirrors=1 RenderFarClip=4096 RenderQualityPerformance=6 RenderFPSLimit=0 BackgroundFPSLimit=25.
Side effect (#1069): after the device-lost logout the process crashed again at exit, after "Goodbye!" (access violation reading 0x16ee8 in static teardown; minidump saved).

I have the full 21:05 report (slipstream_gpu_lost_last.txt, submission list and async segments) if you want it attached.

**Two more occurrences, same signature** (TheSlowly, 2026-10-27; RTX 5070 Ti, Windows, NVIDIA 610.88; test build = main `a152d3c727` + #1167 / #1168, neither touches RT): | | 19:28:24 UTC | 21:05:33 UTC | |---|---|---| | Window | **background, 59 s** | **background, 199 s** | | Graphics NV checkpoints | `g_gi_window` / `g_gi_window` | `g_gi_window` / `g_gi_window` | | Async NV checkpoints | `c_defer_rtrefl_trace` / `c_defer_rtrefl_sel` | `c_defer_rtrefl_trace` / `c_defer_rtrefl_sel` | - Both times the viewer was **unfocused** (BackgroundFPSLimit=25, camera static). That's a pattern worth checking: it never happened while I was actively playing today. - 21:05: TDR, no fault address. Every guard ring was clean (`rtRaySane refused 0`, query budget cut 0, candidate budget hit 0). - The ray list was tiny. The newest slot read 425 rays / 7 groups / 87 tiles (copy not executed for the hung frame); the slots before it were 431 and 414. So it hung on a ~430-ray dispatch, which looks like a traversal that never ends rather than too much work. - Settings: RenderVKRayTracing=1 RenderReflectionProbeDetail=3 RenderShadowDetail=2 RenderVKSunShadows=2 RenderVKProjectorShadows=2 RenderVKUpscaler=2 RenderVKDlssRayReconstruction=2 RenderVKAsyncComputeQueue=1 RenderVKAsyncCompute=-1 RenderVKAsyncDeferredLighting=-1 RenderVKGIEarlyTrace=1 RenderVKGILatencyFrame=0 RenderVKFramePipelining=0 RenderVKGodRays=3 RenderVKMirrors=1 RenderFarClip=4096 RenderQualityPerformance=6 RenderFPSLimit=0 BackgroundFPSLimit=25. - Side effect (#1069): after the device-lost logout the process crashed again at exit, after "Goodbye!" (access violation reading 0x16ee8 in static teardown; minidump saved). I have the full 21:05 report (`slipstream_gpu_lost_last.txt`, submission list and async segments) if you want it attached.

sky_line referenced this issue from a commit

2026-10-27 22:03:47 +00:00

Ray-traced reflection fallback (RenderVKRTReflections) off by default

sky_line referenced this issue from a commit

2026-10-27 22:03:58 +00:00

Ray-traced reflection fallback (RenderVKRTReflections) off by default

sky_line referenced this issue from a commit

2026-10-27 22:56:29 +00:00

Ray-traced reflection fallback (RenderVKRTReflections) off by default

sky_line commented

2026-10-29 05:07:25 +00:00

Author

Owner

Third owner occurrence, same signature (2026-10-29 05:01:44 UTC; RTX 5090, Linux, NVIDIA 615.78.08; test build ae5a243897 = main f58a6c9619 + the current test batch; window focused, 787 s into the session):

Kernel: NVRM: Xid 109, CTX SWITCH TIMEOUT; VK_ERROR_DEVICE_LOST from vkWaitForFences; no fault address.
NV checkpoints, async compute: stage 0x1 last=c_defer_rtrefl_trace, stage 0x2000 last=c_defer_rtrefl_sel. Graphics: g_gi_window at both stages (only waiting at the GI join).
Ray list header (#1128 instrument): slots 690 / 688 / 702 rays, 11 groups each, about 30 tiles, all bounded. So it hung on a dispatch of about 700 rays: a traversal that never ends, not too much work.
New context: the owner was firing a scripted weapon in third person until about 5 s before the loss. The weapon rezzes many short-lived, non-physical temporary objects. The static BLAS count was churning (19081 to 19111 per 5 s log window).
Settings as in the original report: RT on, DLSS RR, projector shadows 2, async deferred lighting on, GI early trace on.

Working hypothesis (unproven): an acceleration-structure lifetime problem. A BLAS, its compaction source, or memory freed when an object is deleted gets released or reused while a TLAS still in flight on the async queue references it, so the traversal loops. It would explain why validating rays (ab53e12cc9) did not help. A rezzing weapon may turn this into a repeatable repro, and the owner is trying that now. Source audit of every BLAS free / compaction / retire path against the async frames-in-flight fence is in progress.

User reports with the same signature:

#1060 and #1102: RTX 5070 Ti, Windows, 1.5.0.1 / 1.5.0.2, both while flying around the same busy event region. Breadcrumbs c_defer_rtrefl_trace / c_defer_rtrefl_sel, graphics pr_early. #1060's guard: 0 rays refused, candidate budget hit 601154 walks.
#941: 1.4.0.5, before the reflection trace moved to async compute: graphics g_defer_rtrefl_fwd / g_defer_rtrefl_trace.
#995 / #996 / #997 are already linked above.

**Third owner occurrence, same signature** (2026-10-29 05:01:44 UTC; RTX 5090, Linux, NVIDIA 615.78.08; test build `ae5a243897` = main `f58a6c9619` + the current test batch; window focused, 787 s into the session): - Kernel: `NVRM: Xid 109, CTX SWITCH TIMEOUT`; `VK_ERROR_DEVICE_LOST` from vkWaitForFences; no fault address. - NV checkpoints, async compute: stage 0x1 last=`c_defer_rtrefl_trace`, stage 0x2000 last=`c_defer_rtrefl_sel`. Graphics: `g_gi_window` at both stages (only waiting at the GI join). - Ray list header (#1128 instrument): slots 690 / 688 / 702 rays, 11 groups each, about 30 tiles, all bounded. So it hung on a dispatch of about 700 rays: a traversal that never ends, not too much work. - **New context:** the owner was firing a scripted weapon in third person until about 5 s before the loss. The weapon rezzes many short-lived, non-physical temporary objects. The static BLAS count was churning (19081 to 19111 per 5 s log window). - Settings as in the original report: RT on, DLSS RR, projector shadows 2, async deferred lighting on, GI early trace on. **Working hypothesis (unproven):** an acceleration-structure lifetime problem. A BLAS, its compaction source, or memory freed when an object is deleted gets released or reused while a TLAS still in flight on the async queue references it, so the traversal loops. It would explain why validating rays (`ab53e12cc9`) did not help. A rezzing weapon may turn this into a repeatable repro, and the owner is trying that now. Source audit of every BLAS free / compaction / retire path against the async frames-in-flight fence is in progress. **User reports with the same signature:** - #1060 and #1102: RTX 5070 Ti, Windows, 1.5.0.1 / 1.5.0.2, both while flying around the same busy event region. Breadcrumbs `c_defer_rtrefl_trace` / `c_defer_rtrefl_sel`, graphics `pr_early`. #1060's guard: 0 rays refused, candidate budget hit 601154 walks. - #941: 1.4.0.5, before the reflection trace moved to async compute: graphics `g_defer_rtrefl_fwd` / `g_defer_rtrefl_trace`. - #995 / #996 / #997 are already linked above.

sky_line referenced this issue from a commit

2026-10-29 07:47:38 +00:00

#1128: Initialize RT fallback tables before BLAS address publication

sky_line referenced this issue from a commit

2026-10-29 07:47:38 +00:00

#1128: Gate AS builds on coherent input publication and rebuild on recovery

sky_line referenced this issue from a commit

2026-10-29 07:47:38 +00:00

#1128: Keep RT fallback initialization allocation failures soft

sky_line referenced this issue from a commit

2026-10-29 09:17:56 +00:00

#1128 RT reflection hang: fold-proof ray guard (with its own proof counter) + a hang beacon that names the stuck ray