M4: RT, GI, DLSS / FSR, frame generation, effects and weather come back #1550

Open
opened 2026-11-15 06:01:14 +00:00 by sky_line · 3 comments
Owner
  • Rebuild the RT structures, GI, denoisers, DLSS / FSR, frame generation (NGX re-initialised on the new device), effects and weather after the basics are up.
  • Every feature returns as it was before the loss.

Done when each feature works after a simulated loss, including NGX features, with no NGX handle left bound to the old device.

Part of #1545.

- Rebuild the RT structures, GI, denoisers, DLSS / FSR, frame generation (NGX re-initialised on the new device), effects and weather after the basics are up. - Every feature returns as it was before the loss. **Done when** each feature works after a simulated loss, including NGX features, with no NGX handle left bound to the old device. Part of #1545.
sky_line added this to the Slipstream project 2026-11-15 06:01:14 +00:00
Author
Owner

First real in-session GPU loss on the owner's RTX 5090 (2026-11-18 18:55): a TDR hang (graphics g_gi_window, async c_defer_rtrefl_sel).

What recovered (12.2 s): RT, async compute, frame generation and the scene all came back.

What did not:

  1. The lost device's VRAM was never returned. The breakdown shows unattributed 9594 MB and the process grew to 15.2 GB from 9.9 GB, so the viewer now hits the VRAM budget and drops resources. The reaper's "(released)" verdict was wrong: 9901 -> 9644 MB.
  2. DLSS stayed unavailable. NGX answered NotImplemented on the new device.

The owner considers this unacceptable for recovery. Brief: agents/recovery_vram_leak_brief.md (free the epoch's allocations and objects before vkDestroyDevice, fix the verdict, re-init NGX).

**First real in-session GPU loss on the owner's RTX 5090** (2026-11-18 18:55): a TDR hang (graphics `g_gi_window`, async `c_defer_rtrefl_sel`). **What recovered (12.2 s):** RT, async compute, frame generation and the scene all came back. **What did not:** 1. **The lost device's VRAM was never returned.** The breakdown shows `unattributed 9594 MB` and the process grew to 15.2 GB from 9.9 GB, so the viewer now hits the VRAM budget and drops resources. The reaper's "(released)" verdict was wrong: 9901 -> 9644 MB. 2. **DLSS stayed unavailable.** NGX answered NotImplemented on the new device. The owner considers this unacceptable for recovery. Brief: agents/recovery_vram_leak_brief.md (free the epoch's allocations and objects before vkDestroyDevice, fix the verdict, re-init NGX).
Author
Owner

Second real loss, 2026-11-18 20:10 (build 981cba947c, which predates the reaper fix ae1f063eb7).

The loss: a TDR during a DLSS mode switch (vkQueueWaitIdle). Graphics last ran rrGuides; async compute last ran c_gi_hitmat.

Recovery, logged as successful:

  • took 1.2 s;
  • DLSS unavailable again;
  • VRAM 16918 -> 16789 MB, wrongly logged as "released".

Then a crash: SIGSEGV 26 s later at 20:10:37, inside libnvidia-glcore.so.615.78.08 (+0xe1fa0e / +0xcfbb09), called from the main thread. There was no crash dialog, and the core dump exists (pid 1621190). Likely a call into the driver with a lost-epoch handle.

To verify: the reaper fix should cover this. Run the forced-recovery test on ae1f063eb7 and check that nothing crashes in the minutes after recovery.

Pattern across both real losses today: async compute was running RT GI or RT reflections (c_defer_rtrefl_sel, c_gi_hitmat) while graphics ran other work. Cross-queue sync is the suspect for the hangs themselves.

**Second real loss, 2026-11-18 20:10** (build 981cba947c, which predates the reaper fix ae1f063eb7). **The loss:** a TDR during a DLSS mode switch (vkQueueWaitIdle). Graphics last ran `rrGuides`; async compute last ran `c_gi_hitmat`. **Recovery, logged as successful:** - took 1.2 s; - DLSS unavailable again; - VRAM 16918 -> 16789 MB, wrongly logged as "released". **Then a crash:** SIGSEGV 26 s later at 20:10:37, inside `libnvidia-glcore.so.615.78.08` (+0xe1fa0e / +0xcfbb09), called from the main thread. There was no crash dialog, and the core dump exists (pid 1621190). Likely a call into the driver with a lost-epoch handle. **To verify:** the reaper fix should cover this. Run the forced-recovery test on ae1f063eb7 and check that nothing crashes in the minutes after recovery. **Pattern across both real losses today:** async compute was running RT GI or RT reflections (`c_defer_rtrefl_sel`, `c_gi_hitmat`) while graphics ran other work. Cross-queue sync is the suspect for the hangs themselves.
Author
Owner

1.5.6 ships the lost-device storage reclaim and NGX discovery preservation (9239e99ddd). Leaving open: this milestone's remaining scope is tracked here.

1.5.6 ships the lost-device storage reclaim and NGX discovery preservation (9239e99ddd). Leaving open: this milestone's remaining scope is tracked here.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
sky_line/Slipstream#1550
No description provided.