Survive a graphics device loss: rebuild the Vulkan renderer instead of logging out #1069

New issue

Open

opened 2026-10-25 19:41:35 +00:00 by sky_line · 0 comments

sky_line commented

2026-10-25 19:41:35 +00:00

Owner

Problem

On VK_ERROR_DEVICE_LOST the viewer writes the device-loss report, logs out and closes (onGraphicsDeviceLost). A device loss is often not our fault: on Linux a GPU reset caused by ANY process (#1063: Chromium's GPU process in the media plugin faulted, amdgpu did a mode1 reset, "VRAM is lost") kills every Vulkan device on the card. Windows TDRs from other apps behave similarly.

Goal

Recover in-session: tear down the lost VkDevice and everything created from it, create a new device, re-upload/rebuild GPU state, and keep the session (no logout).

Scope sketch

Destroy path: every VK object owner must release without touching the dead device beyond destroy calls (already mostly true for shutdown); stop worker threads that submit (async compute, compile workers, uploads).
Re-create: device + queues, swapchain, renderer init (pipelines from the pipeline cache), descriptor heaps / bindless tables, RT foundation.
Re-populate: textures (re-upload from CPU copies / refetch from cache), geometry (re-register objects), RT BLAS/TLAS, probes, particles, UI atlases, FX. Treat it like a full re-ingest; correctness over speed.
Loop guard: if the device is lost again within N seconds of a rebuild (our own hang, not an external reset), fall back to today's behaviour (report + logout), possibly with the safe-settings path.
Keep the device-loss report and breadcrumbs exactly as today, written before the rebuild.
Related: #1049 (RT reflection hang), #1063 (external GPU reset), the existing relaunch-after-device-loss path.

## Problem On `VK_ERROR_DEVICE_LOST` the viewer writes the device-loss report, logs out and closes (`onGraphicsDeviceLost`). A device loss is often not our fault: on Linux a GPU reset caused by ANY process (#1063: Chromium's GPU process in the media plugin faulted, amdgpu did a mode1 reset, "VRAM is lost") kills every Vulkan device on the card. Windows TDRs from other apps behave similarly. ## Goal Recover in-session: tear down the lost VkDevice and everything created from it, create a new device, re-upload/rebuild GPU state, and keep the session (no logout). ## Scope sketch - Destroy path: every VK object owner must release without touching the dead device beyond destroy calls (already mostly true for shutdown); stop worker threads that submit (async compute, compile workers, uploads). - Re-create: device + queues, swapchain, renderer init (pipelines from the pipeline cache), descriptor heaps / bindless tables, RT foundation. - Re-populate: textures (re-upload from CPU copies / refetch from cache), geometry (re-register objects), RT BLAS/TLAS, probes, particles, UI atlases, FX. Treat it like a full re-ingest; correctness over speed. - Loop guard: if the device is lost again within N seconds of a rebuild (our own hang, not an external reset), fall back to today's behaviour (report + logout), possibly with the safe-settings path. - Keep the device-loss report and breadcrumbs exactly as today, written before the rebuild. - Related: #1049 (RT reflection hang), #1063 (external GPU reset), the existing relaunch-after-device-loss path.

sky_line added the

rendering

bug

labels

2026-10-25 19:41:35 +00:00

sky_line added this to the Slipstream project

2026-10-25 19:41:35 +00:00

sky_line referenced this issue

2026-10-25 19:46:22 +00:00

[viewer] [crash] Crash: graphics driver crash on intense load (?) #1063

sky_line self-assigned this

2026-10-26 02:58:26 +00:00