GPU-loss recovery: the session survives a graphics device loss #1545

Open
opened 2026-11-15 06:01:13 +00:00 by sky_line · 1 comment
Owner

A graphics device loss (VK_ERROR_DEVICE_LOST: a Windows TDR, an NVIDIA Xid 109 context timeout such as the RT reflections hang #1128, a driver reset) must not end the session. The SL session stays connected; the Vulkan renderer rebuilds in place on a new device and the user keeps playing.

Today (#135) a loss writes the GPU-lost report, logs out and exits (Restart / Quit). The report stays; the exit goes.

How it works

  • Device epochs. Each device generation (device, swapchain, renderer, services, the per-device state of the singletons) is an epoch. On a loss, epoch N is set aside and never touched again except to be destroyed; epoch N+1 is built on the same VkInstance and physical device.
  • The lost flag is per epoch, looked up by device / queue handle, so a stray call on an old handle is skipped. Tables keyed by raw Vulkan handles are cleared at each new epoch (a new device can reuse handle values).
  • The old device's VRAM is freed: epoch N is destroyed on a reaper thread once the driver has reported it lost.
  • Scene data lives outside the renderer (instance / mesh / material / skin stores, the disk caches), so a new renderer reloads from them; textures come back through the existing eviction path (BC disk cache, J2C cache, HTTP).
  • Progressive rebuild: device + swapchain + UI first; a raster-only renderer; then near-to-far re-upload driven by the existing priority systems (texel demand, texture-prep priority classes, self / HUD first, nearest-first terrain); shadows, probes, RT, GI, DLSS / FSR, frame generation, effects, weather and VR last.

Decisions

  • No watchdog for now: recovery starts on a driver-reported loss. A watchdog that starts recovery before the driver's own timeout may come later, only if the driver's reset can be kept from killing the new device.
  • Every feature comes back as it was. A real crash trace (not a TDR / timeout) may later justify disabling a feature or saving a report.
  • What the user sees: the last frame frozen with a small "recovering" overlay; when the rebuilt renderer draws, a system message says Slipstream prevented a graphics crash.
  • VR: the headset runtime's own anti-hang room covers the gap; the XR session is rebuilt in place.
  • Instant replay: the pre-loss ring is discarded.
  • Real-hang tests: deliberate hangs on the GTX 1080 test box first, then an RTX 5090.
  • Repeated losses: a third loss in a short window (or a lost physical device) falls back to today's Restart / Quit dialog, which also stays under Windows' TDR limit.

Main risks

  • ~120 vkDeviceIsLost and ~200 LLVulkanContext::instance() call sites must each be classified (current epoch vs a specific epoch).
  • A thread stuck inside a driver call on the dead device cannot be recovered until the driver returns.
  • The window can only have one live swapchain; the old one is destroyed once the loss is confirmed.
  • NGX (DLSS / frame generation) has never been re-initialised in-process.
  • GPU readbacks after a loss must never reach the disk caches or the server (bakes, snapshots, map captures).

Milestones

Child tickets below; each is testable with RenderVKDebugSimulateDeviceLost 1/2/3.

A graphics device loss (VK_ERROR_DEVICE_LOST: a Windows TDR, an NVIDIA Xid 109 context timeout such as the RT reflections hang #1128, a driver reset) must not end the session. The SL session stays connected; the Vulkan renderer rebuilds in place on a new device and the user keeps playing. Today (#135) a loss writes the GPU-lost report, logs out and exits (Restart / Quit). The report stays; the exit goes. ## How it works - **Device epochs.** Each device generation (device, swapchain, renderer, services, the per-device state of the singletons) is an epoch. On a loss, epoch N is set aside and never touched again except to be destroyed; epoch N+1 is built on the same VkInstance and physical device. - **The lost flag is per epoch**, looked up by device / queue handle, so a stray call on an old handle is skipped. Tables keyed by raw Vulkan handles are cleared at each new epoch (a new device can reuse handle values). - **The old device's VRAM is freed**: epoch N is destroyed on a reaper thread once the driver has reported it lost. - **Scene data lives outside the renderer** (instance / mesh / material / skin stores, the disk caches), so a new renderer reloads from them; textures come back through the existing eviction path (BC disk cache, J2C cache, HTTP). - **Progressive rebuild:** device + swapchain + UI first; a raster-only renderer; then near-to-far re-upload driven by the existing priority systems (texel demand, texture-prep priority classes, self / HUD first, nearest-first terrain); shadows, probes, RT, GI, DLSS / FSR, frame generation, effects, weather and VR last. ## Decisions - No watchdog for now: recovery starts on a driver-reported loss. A watchdog that starts recovery before the driver's own timeout may come later, only if the driver's reset can be kept from killing the new device. - Every feature comes back as it was. A real crash trace (not a TDR / timeout) may later justify disabling a feature or saving a report. - What the user sees: the last frame frozen with a small "recovering" overlay; when the rebuilt renderer draws, a system message says Slipstream prevented a graphics crash. - VR: the headset runtime's own anti-hang room covers the gap; the XR session is rebuilt in place. - Instant replay: the pre-loss ring is discarded. - Real-hang tests: deliberate hangs on the GTX 1080 test box first, then an RTX 5090. - Repeated losses: a third loss in a short window (or a lost physical device) falls back to today's Restart / Quit dialog, which also stays under Windows' TDR limit. ## Main risks - ~120 `vkDeviceIsLost` and ~200 `LLVulkanContext::instance()` call sites must each be classified (current epoch vs a specific epoch). - A thread stuck inside a driver call on the dead device cannot be recovered until the driver returns. - The window can only have one live swapchain; the old one is destroyed once the loss is confirmed. - NGX (DLSS / frame generation) has never been re-initialised in-process. - GPU readbacks after a loss must never reach the disk caches or the server (bakes, snapshots, map captures). ## Milestones Child tickets below; each is testable with `RenderVKDebugSimulateDeviceLost 1/2/3`.
sky_line added this to the Slipstream project 2026-11-15 06:01:13 +00:00
Author
Owner

Milestones: #1546, #1547, #1548, #1549, #1550, #1551, #1552, #1553, #1554

Milestones: #1546, #1547, #1548, #1549, #1550, #1551, #1552, #1553, #1554
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
sky_line/Slipstream#1545
No description provided.