VRAM edge cases at heavy venues: pre-flight all large allocations, rigged resize double-hold, #561 floor vs live budget, ladder never recovers #563

New issue

Closed

opened 2026-10-08 19:08:46 +00:00 by sky_line · 1 comment

sky_line commented

2026-10-08 19:08:46 +00:00

Owner

Follow-ups from the 2026-10-08 PEAK Lounge session (70 heavy human avatars, RTX 5090, heap budget ~22.9 GB, default settings):

Driver segfaults instead of OOM. NVIDIA 615.71.09 crashes inside vkAllocateMemory when a large request pushes the device-local heap past its VK_EXT_memory_budget budget. It happened twice: in ensureTemporal (1.66 GB skinHistPositions) and in ensureRiggedBuffers at 20.9/22.9 GB.
- Shipped: soft allocations ≥ 64 MB are pre-flighted against the live heap budget (commit "VK: pre-flight large soft buffer allocations…").
- To do: apply the same pre-flight to images and to non-soft large allocations, or convert those callers to soft paths.
Rigged resize double-holds memory. The new buffers are created before the old ones are freed, so a resize briefly needs old + new. Free first where nothing in flight uses the old buffer (after a device idle), or resize in smaller steps.
The #561 budget floor can push VRAM to the edge. The floor is protected demand +30%, up to 40% of the VRAM budget. At that venue it plus everything else reached 20.9/22.9 GB. The floor should account for headroom against the live heap budget, not just a fraction of it.
The VRAM ladder never recovers at heavy venues. Restore needs usage < 0.80·B and usage + re-add < 0.85·B (llvrambudgetvk.cpp ~455–470), but textures (all-full demand ~17.5 GB) refill whatever is freed, so the ladder parks at rung ~27/39. Needed:
- importance-ordered rungs (far textures, RT and far rigged before near rigged);
- textures must not refill the headroom a recovery is waiting for.

**Follow-ups from the 2026-10-08 PEAK Lounge session** (70 heavy human avatars, RTX 5090, heap budget ~22.9 GB, default settings): 1. **Driver segfaults instead of OOM.** NVIDIA 615.71.09 crashes inside `vkAllocateMemory` when a large request pushes the device-local heap past its `VK_EXT_memory_budget` budget. It happened twice: in `ensureTemporal` (1.66 GB `skinHistPositions`) and in `ensureRiggedBuffers` at 20.9/22.9 GB. - **Shipped:** soft allocations ≥ 64 MB are pre-flighted against the live heap budget (commit "VK: pre-flight large soft buffer allocations…"). - **To do:** apply the same pre-flight to images and to non-soft large allocations, or convert those callers to soft paths. 2. **Rigged resize double-holds memory.** The new buffers are created before the old ones are freed, so a resize briefly needs old + new. Free first where nothing in flight uses the old buffer (after a device idle), or resize in smaller steps. 3. **The #561 budget floor can push VRAM to the edge.** The floor is protected demand +30%, up to 40% of the VRAM budget. At that venue it plus everything else reached 20.9/22.9 GB. The floor should account for headroom against the live heap budget, not just a fraction of it. 4. **The VRAM ladder never recovers at heavy venues.** Restore needs usage < 0.80·B and usage + re-add < 0.85·B (`llvrambudgetvk.cpp` ~455–470), but textures (all-full demand ~17.5 GB) refill whatever is freed, so the ladder parks at rung ~27/39. Needed: - importance-ordered rungs (far textures, RT and far rigged before near rigged); - textures must not refill the headroom a recovery is waiting for.

sky_line added the

labels

2026-10-08 19:08:46 +00:00

sky_line added this to the Slipstream project

2026-10-08 19:08:46 +00:00