M1: a simulated loss brings back an empty scene with working UI; the old device is freed #1547

Open
opened 2026-11-15 06:01:14 +00:00 by sky_line · 2 comments
Owner
  • On RenderVKDebugSimulateDeviceLost (and on a real driver-reported loss), keep the SL session and main loop running, set the old epoch aside, and build a new device, swapchain and UI on the same instance.
  • Show the last frame frozen with a small "recovering" overlay until the new renderer draws; then post a system message that Slipstream prevented a graphics crash.
  • Destroy the old swapchain and device on a reaper thread once the loss is confirmed, so its VRAM is released.
  • Discard the instant replay ring.
  • Repeat-loss policy: a third loss in a short window falls back to today's Restart / Quit dialog.

Done when 50 simulated losses in a row each return to a working UI with an empty or partial scene, with no leak of device memory (checked with the memory log) and no validation errors.

Part of #1545.

- On `RenderVKDebugSimulateDeviceLost` (and on a real driver-reported loss), keep the SL session and main loop running, set the old epoch aside, and build a new device, swapchain and UI on the same instance. - Show the last frame frozen with a small "recovering" overlay until the new renderer draws; then post a system message that Slipstream prevented a graphics crash. - Destroy the old swapchain and device on a reaper thread once the loss is confirmed, so its VRAM is released. - Discard the instant replay ring. - Repeat-loss policy: a third loss in a short window falls back to today's Restart / Quit dialog. **Done when** 50 simulated losses in a row each return to a working UI with an empty or partial scene, with no leak of device memory (checked with the memory log) and no validation errors. Part of #1545.
sky_line added this to the Slipstream project 2026-11-15 06:01:14 +00:00
Author
Owner

First real in-session GPU loss on the owner's RTX 5090 (2026-11-18 18:55): a TDR hang (graphics g_gi_window, async c_defer_rtrefl_sel).

What recovered (12.2 s): RT, async compute, frame generation and the scene all came back.

What did not:

  1. The lost device's VRAM was never returned. The breakdown shows unattributed 9594 MB and the process grew to 15.2 GB from 9.9 GB, so the viewer now hits the VRAM budget and drops resources. The reaper's "(released)" verdict was wrong: 9901 -> 9644 MB.
  2. DLSS stayed unavailable. NGX answered NotImplemented on the new device.

The owner considers this unacceptable for recovery. Brief: agents/recovery_vram_leak_brief.md (free the epoch's allocations and objects before vkDestroyDevice, fix the verdict, re-init NGX).

**First real in-session GPU loss on the owner's RTX 5090** (2026-11-18 18:55): a TDR hang (graphics `g_gi_window`, async `c_defer_rtrefl_sel`). **What recovered (12.2 s):** RT, async compute, frame generation and the scene all came back. **What did not:** 1. **The lost device's VRAM was never returned.** The breakdown shows `unattributed 9594 MB` and the process grew to 15.2 GB from 9.9 GB, so the viewer now hits the VRAM budget and drops resources. The reaper's "(released)" verdict was wrong: 9901 -> 9644 MB. 2. **DLSS stayed unavailable.** NGX answered NotImplemented on the new device. The owner considers this unacceptable for recovery. Brief: agents/recovery_vram_leak_brief.md (free the epoch's allocations and objects before vkDestroyDevice, fix the verdict, re-init NGX).
Author
Owner

1.5.6 ships the lost-device storage reclaim and NGX discovery preservation (9239e99ddd). Leaving open: this milestone's remaining scope is tracked here.

1.5.6 ships the lost-device storage reclaim and NGX discovery preservation (9239e99ddd). Leaving open: this milestone's remaining scope is tracked here.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
sky_line/Slipstream#1547
No description provided.