Vulkan: GPU breadcrumbs for device-lost diagnosis (NV checkpoints / AMD buffer markers) #136

New issue

Closed

opened 2026-09-28 02:29:23 +00:00 by sky_line · 1 comment

sky_line commented

2026-09-28 02:29:23 +00:00

Owner

Related: #107, #135.

When the GPU hangs and the device is lost, the CPU-side dump only shows where we noticed (the next fence wait), not which pass hung. Add GPU breadcrumbs:

NVIDIA: VK_NV_device_diagnostic_checkpoints — vkCmdSetCheckpointNV at our existing gpuSubMark points (c_cull, c_lights, g_prepass, g_ssao, g_pass1, pass2, p_bloom, p_dof, p_aa, p_final, UI...); on DEVICE_LOST read vkGetQueueCheckpointDataNV and log the last executed/started markers.
AMD: VK_AMD_buffer_marker (vkCmdWriteBufferMarkerAMD into a host-visible buffer) with the same marker set.
Fallback for other vendors: VK_EXT_device_fault where available (vkGetDeviceFaultInfoEXT).
Setting to force it on (cheap; could default on for beta).
Output goes into the log so tester logs identify the hanging pass.

Related: #107, #135. When the GPU hangs and the device is lost, the CPU-side dump only shows where we noticed (the next fence wait), not which pass hung. Add GPU breadcrumbs: - NVIDIA: `VK_NV_device_diagnostic_checkpoints` — `vkCmdSetCheckpointNV` at our existing gpuSubMark points (c_cull, c_lights, g_prepass, g_ssao, g_pass1, pass2, p_bloom, p_dof, p_aa, p_final, UI...); on DEVICE_LOST read `vkGetQueueCheckpointDataNV` and log the last executed/started markers. - AMD: `VK_AMD_buffer_marker` (`vkCmdWriteBufferMarkerAMD` into a host-visible buffer) with the same marker set. - Fallback for other vendors: `VK_EXT_device_fault` where available (`vkGetDeviceFaultInfoEXT`). - Setting to force it on (cheap; could default on for beta). - Output goes into the log so tester logs identify the hanging pass.

sky_line referenced this issue

2026-09-28 02:29:24 +00:00

Windows: abrupt exit with no dump or log error (fail-fast?) — tracking #107

sky_line referenced this issue from a commit

2026-09-28 20:10:33 +00:00

Vulkan: GPU breadcrumbs for device-lost diagnosis

sky_line commented

2026-09-30 06:19:49 +00:00

Author

Owner

Done: GPU breadcrumbs are in (NV device-diagnostic checkpoints / AMD buffer markers). The log prints which one is active ([vkgfx] GPU breadcrumbs: ...), and gpuSubMark/gpuTimerMark drop a crumb per pass.

Done: GPU breadcrumbs are in (NV device-diagnostic checkpoints / AMD buffer markers). The log prints which one is active (`[vkgfx] GPU breadcrumbs: ...`), and gpuSubMark/gpuTimerMark drop a crumb per pass.

sky_line closed this issue

2026-09-30 06:19:49 +00:00

sky_line referenced this issue

2026-09-30 19:40:28 +00:00

GPU device lost on GTX 1080 Ti (Windows 10, driver 582.66) while the viewer was out of focus #209

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

sky_line/Slipstream#136

No description provided.

Rows
Columns