VRAM: suballocate textures from shared blocks (~0.9 GB of 64 KiB page padding); bonus: group spatially related textures

sky_line commented

2026-10-05 06:04:01 +00:00

Owner

Part of #405. Follow-up from #407.

Problem

Every texture gets its own vkAllocateMemory, and the driver rounds every allocation up to its page granularity: 64 KiB on NVIDIA (measured by the #407 probe on an RTX 5090 / 615.71; the tracker now measures it at device creation). Most SL textures are small: floor-size and low-mip images, UI and icons. So in a busy region:

overhead 978 (pad 956 + driver 39 + untracked 0 + unattributed 0) | ... 16357 allocs @64K
tex 374 MB

~956 MB of VRAM is padding, more than double the actual texture bytes (374 MB). About 10,000 of the ~14,000 texture allocations are tiny (floor size), and each wastes ~60 KB. Measured on the probe:

1000 × 4×4 images: 0.5 MB requested, 62.5 MB used.
2000 × 100×100 RGBA8: 109 MB requested, 125 MB used.

Huge allocation counts also hurt on drivers with a low maxMemoryAllocationCount. Some report 4096; NVIDIA has no practical limit. The bug report now shows the limit when there is one.

Goal

Suballocate textures (and other small images/buffers) out of large shared device-memory blocks instead of one allocation each. That recovers most of the padding (~0.9 GB here) and cuts the allocation count ~3×.

Design notes

Block allocator: large blocks (e.g. 64-256 MiB; also check bufferImageGranularity and the alignment from vkGetImageMemoryRequirements), grouped by memory type, with a suballocator: size classes / buddy, or TLSF-style for mixed sizes. VMA's algorithms are a reference, but we own the allocator (no new dependency unless decided).
What goes in blocks: small and medium textures (e.g. ≤ 1-4 MiB); dedicated allocations stay for big images, render targets and anything the driver prefers dedicated (VkMemoryDedicatedRequirements / prefersDedicatedAllocation).
Residency changes: mip-ups and downgrades reallocate an image at a new size. Free-list reuse plus occasional defragmentation/compaction (copy live images into a fresh block during a quiet frame, update bindless descriptors) keeps blocks from turning into Swiss cheese. Respect the budget: a half-empty block is waste too, so release empty blocks promptly and report per-block occupancy.
Accounting: the #407 tracker keeps working per category. Report block slack (block size − live suballocations) as slack, not pad, and keep pad meaning driver rounding of the blocks themselves (≈ 0).
Interaction with the ladder: texture mip-downs free suballocations; empty blocks are freed so the VRAM actually drops. Make sure the texture pressure steps still show real savings.

When placing textures in blocks, prefer putting spatially related or co-used textures in the same block (and near each other) when they fit:

textures of the same object / linkset / region cell, or the same avatar's outfit (body, head, clothing layers, attachments);
the material set of one face (base colour, normal, ORM / specular, emissive).

Possible benefits:

Locality in the GPU's memory system (TLB / page reach, fewer page-table walks when a draw touches several textures of the same object).
Cheaper eviction and defrag: whole groups leave together when an object or region unloads, which keeps fragmentation down.
Simpler streaming bookkeeping per object or region.

Measure, don't assume: the GPU win is hardware-dependent and may be small. Compare frame time and TLB-related counters (Nsight / RGP) for a busy scene with grouped vs ungrouped placement before committing to the grouping logic. The padding fix alone is the main goal.

Done when

Busy-region padding drops from ~950 MB to under ~50 MB, and the allocation count drops substantially.
No visual regressions; bindless descriptors stay correct through defrag.
Ladder texture steps still free real VRAM (empty blocks released).
The breakdown/readout show block occupancy and slack.
(Bonus) Grouped placement measured, and kept only if it helps.

Decided plan (owner, 2026-10-05)

Allocator: VMA

Use AMD's Vulkan Memory Allocator (VMA): a single header (vk_mem_alloc.h), MIT license, vendored. It replaces a hand-rolled TLSF and defrag, and gives us:

pools and TLSF suballocation;
dedicated-allocation heuristics;
VK_EXT_memory_budget integration;
statistics;
a defragmentation API for later.

The #407 tracker stays. VMA's device-memory callbacks report every block it allocates or frees, so driver-level accounting keeps working, and per-category texture bytes come from suballocations.

Measured on the RTX 5090 (615.71)

vkGetImageMemoryRequirements, full mip chains, optimal tiling, sampled + transfer-dst + mutable format. Alignment is 1 KiB for every case; bufferImageGranularity is 1 KiB. The 64 KiB is only the driver's per-vkAllocateMemory rounding, so suballocation recovers almost all of it.

Size	BC7 / BC3 / BC5	BC1	RGBA8
4-16 px	1.5-2.5 KiB	1.5-2.5 KiB	1.5-3 KiB
32²	3.5 KiB	3 KiB	7 KiB
64²	7.5 KiB	5 KiB	23 KiB
128²	23.5 KiB	13 KiB	87 KiB
256²	87.5 KiB	45 KiB	343 KiB
512²	343 KiB	173 KiB	1.33 MiB
1024²	1.33 MiB	685 KiB	5.3 MiB

Size classes, by the image's memory-requirement size

Small, ≤ 64 KiB (BC7 ≤ 128², BC1 ≤ 256², RGBA8 ≤ 64²: floor textures, low mips, icons, thumbnails)
- Today each pays 64 KiB (62-98% waste). This is almost all of the ~950 MB padding.
- Use size-class slabs: 2, 3, 4, 6, 8, 12, 16, 24, 32, 48 and 64 KiB, in 16-64 MiB blocks.
- Any free slot fits the next texture of its class, so there's no external fragmentation. Internal waste is ≤ ~33% worst case and typically < 10%.
Medium, 64 KiB - 4 MiB (BC7 256²-1024², BC1 512²-2048², RGBA8 128²-512²)
- Today's rounding waste is up to ~42% at the low end and < 5% from 1 MiB up.
- Use one TLSF pool (VMA default) with 64-128 MiB blocks and fullest-block-first placement (min-memory strategy).
Large, > 4 MiB, plus all render targets, probes, shadow maps, RT and anything the driver prefers dedicated: stays dedicated, as today (< 1.5% padding).

The 64 KiB threshold comes from the granularity the tracker measures at device creation (#407), not a constant, so drivers with a smaller rounding unit shift the small class automatically. AMD and Intel numbers come from bug reports.

Expected (busy region at 4K):

Small: ~10k textures × ~60 KiB → ~0.6 GB recovered (≈ 50 MB slab slack total).
Medium: a few thousand × ~30 KiB → ~0.1 GB.

Phasing

Phase 1: vendor VMA; small-class slabs and the medium TLSF pool for world textures (and UI textures where they share the path); dedicated stays for large textures.
- Opportunistic compaction: textures are re-created on every mip up/down, so each new image goes into the fullest block that fits. Sparse blocks drain naturally without a defrag pass.
- Empty blocks are released promptly.
- Occupancy and slack go in the readout and the VRAM breakdown line.
Phase 2 (only if numbers say so): a VMA defragmentation pass (we do the copies, the descriptor updates and the frame-in-flight retirement), if block occupancy stays low with significant slack in long single-location sessions.
Bonus: group related textures (per-region / per-avatar pools or placement hints). Measure TLB / frame time first.

Unload hitches

With one allocation per texture, unloading a region or a ladder texture step issues thousands of vkFreeMemory calls at once. Those are slow driver calls and a likely contributor to unload hitches. With suballocation, freeing a texture is a cheap free-list operation; vkFreeMemory only happens when a whole block empties. Measure the hitch before and after with the existing hitch counters.

Part of #405. Follow-up from #407. ## Problem Every texture gets its own `vkAllocateMemory`, and the driver rounds every allocation up to its page granularity: **64 KiB** on NVIDIA (measured by the #407 probe on an RTX 5090 / 615.71; the tracker now measures it at device creation). Most SL textures are small: floor-size and low-mip images, UI and icons. So in a busy region: ``` overhead 978 (pad 956 + driver 39 + untracked 0 + unattributed 0) | ... 16357 allocs @64K tex 374 MB ``` **~956 MB of VRAM is padding**, more than double the actual texture bytes (374 MB). About 10,000 of the ~14,000 texture allocations are tiny (floor size), and each wastes ~60 KB. Measured on the probe: - 1000 × 4×4 images: 0.5 MB requested, 62.5 MB used. - 2000 × 100×100 RGBA8: 109 MB requested, 125 MB used. Huge allocation counts also hurt on drivers with a low `maxMemoryAllocationCount`. Some report 4096; NVIDIA has no practical limit. The bug report now shows the limit when there is one. ## Goal Suballocate textures (and other small images/buffers) out of large shared device-memory blocks instead of one allocation each. That recovers most of the padding (~0.9 GB here) and cuts the allocation count ~3×. ## Design notes - **Block allocator:** large blocks (e.g. 64-256 MiB; also check `bufferImageGranularity` and the alignment from `vkGetImageMemoryRequirements`), grouped by memory type, with a suballocator: size classes / buddy, or TLSF-style for mixed sizes. VMA's algorithms are a reference, but we own the allocator (no new dependency unless decided). - **What goes in blocks:** small and medium textures (e.g. ≤ 1-4 MiB); dedicated allocations stay for big images, render targets and anything the driver prefers dedicated (`VkMemoryDedicatedRequirements` / `prefersDedicatedAllocation`). - **Residency changes:** mip-ups and downgrades reallocate an image at a new size. Free-list reuse plus occasional **defragmentation/compaction** (copy live images into a fresh block during a quiet frame, update bindless descriptors) keeps blocks from turning into Swiss cheese. Respect the budget: a half-empty block is waste too, so release empty blocks promptly and report per-block occupancy. - **Accounting:** the #407 tracker keeps working per category. Report block slack (block size − live suballocations) as `slack`, not `pad`, and keep `pad` meaning driver rounding of the blocks themselves (≈ 0). - **Interaction with the ladder:** texture mip-downs free suballocations; empty blocks are freed so the VRAM actually drops. Make sure the `texture pressure` steps still show real savings. ## Bonus: pack related textures together When placing textures in blocks, prefer putting **spatially related or co-used textures** in the same block (and near each other) when they fit: - textures of the same object / linkset / region cell, or the same avatar's outfit (body, head, clothing layers, attachments); - the material set of one face (base colour, normal, ORM / specular, emissive). Possible benefits: - **Locality** in the GPU's memory system (TLB / page reach, fewer page-table walks when a draw touches several textures of the same object). - **Cheaper eviction and defrag:** whole groups leave together when an object or region unloads, which keeps fragmentation down. - Simpler **streaming bookkeeping** per object or region. Measure, don't assume: the GPU win is hardware-dependent and may be small. Compare frame time and TLB-related counters (Nsight / RGP) for a busy scene with grouped vs ungrouped placement before committing to the grouping logic. The padding fix alone is the main goal. ## Done when - [ ] Busy-region padding drops from ~950 MB to under ~50 MB, and the allocation count drops substantially. - [ ] No visual regressions; bindless descriptors stay correct through defrag. - [ ] Ladder texture steps still free real VRAM (empty blocks released). - [ ] The breakdown/readout show block occupancy and slack. - [ ] (Bonus) Grouped placement measured, and kept only if it helps. --- ## Decided plan (owner, 2026-10-05) ### Allocator: VMA Use **AMD's Vulkan Memory Allocator (VMA)**: a single header (`vk_mem_alloc.h`), MIT license, vendored. It replaces a hand-rolled TLSF and defrag, and gives us: - pools and TLSF suballocation; - dedicated-allocation heuristics; - `VK_EXT_memory_budget` integration; - statistics; - a defragmentation API for later. The #407 tracker stays. VMA's device-memory callbacks report every block it allocates or frees, so driver-level accounting keeps working, and per-category texture bytes come from suballocations. ### Measured on the RTX 5090 (615.71) `vkGetImageMemoryRequirements`, full mip chains, optimal tiling, sampled + transfer-dst + mutable format. **Alignment is 1 KiB for every case; `bufferImageGranularity` is 1 KiB.** The 64 KiB is only the driver's per-`vkAllocateMemory` rounding, so suballocation recovers almost all of it. | Size | BC7 / BC3 / BC5 | BC1 | RGBA8 | |---|---|---|---| | 4-16 px | 1.5-2.5 KiB | 1.5-2.5 KiB | 1.5-3 KiB | | 32² | 3.5 KiB | 3 KiB | 7 KiB | | 64² | 7.5 KiB | 5 KiB | 23 KiB | | 128² | 23.5 KiB | 13 KiB | 87 KiB | | 256² | 87.5 KiB | 45 KiB | 343 KiB | | 512² | 343 KiB | 173 KiB | 1.33 MiB | | 1024² | 1.33 MiB | 685 KiB | 5.3 MiB | ### Size classes, by the image's memory-requirement size - **Small, ≤ 64 KiB** (BC7 ≤ 128², BC1 ≤ 256², RGBA8 ≤ 64²: floor textures, low mips, icons, thumbnails) - Today each pays 64 KiB (62-98% waste). This is almost all of the ~950 MB padding. - Use **size-class slabs**: 2, 3, 4, 6, 8, 12, 16, 24, 32, 48 and 64 KiB, in 16-64 MiB blocks. - Any free slot fits the next texture of its class, so there's no external fragmentation. Internal waste is ≤ ~33% worst case and typically < 10%. - **Medium, 64 KiB - 4 MiB** (BC7 256²-1024², BC1 512²-2048², RGBA8 128²-512²) - Today's rounding waste is up to ~42% at the low end and < 5% from 1 MiB up. - Use one **TLSF pool** (VMA default) with 64-128 MiB blocks and fullest-block-first placement (min-memory strategy). - **Large, > 4 MiB, plus all render targets, probes, shadow maps, RT and anything the driver prefers dedicated**: stays dedicated, as today (< 1.5% padding). The 64 KiB threshold comes from the **granularity the tracker measures at device creation** (#407), not a constant, so drivers with a smaller rounding unit shift the small class automatically. AMD and Intel numbers come from bug reports. **Expected (busy region at 4K):** - Small: ~10k textures × ~60 KiB → ~0.6 GB recovered (≈ 50 MB slab slack total). - Medium: a few thousand × ~30 KiB → ~0.1 GB. ### Phasing 1. **Phase 1:** vendor VMA; small-class slabs and the medium TLSF pool for world textures (and UI textures where they share the path); dedicated stays for large textures. - **Opportunistic compaction:** textures are re-created on every mip up/down, so each new image goes into the fullest block that fits. Sparse blocks drain naturally without a defrag pass. - Empty blocks are released promptly. - **Occupancy and slack** go in the readout and the VRAM breakdown line. 2. **Phase 2 (only if numbers say so):** a VMA defragmentation pass (we do the copies, the descriptor updates and the frame-in-flight retirement), if block occupancy stays low with significant slack in long single-location sessions. 3. **Bonus:** group related textures (per-region / per-avatar pools or placement hints). Measure TLB / frame time first. ### Unload hitches With one allocation per texture, unloading a region or a ladder texture step issues thousands of `vkFreeMemory` calls at once. Those are slow driver calls and a likely contributor to unload hitches. With suballocation, freeing a texture is a cheap free-list operation; `vkFreeMemory` only happens when a whole block empties. Measure the hitch before and after with the existing hitch counters.

sky_line added the

rendering

perf

labels

2026-10-05 06:04:01 +00:00

sky_line added a new dependency

2026-10-05 06:04:01 +00:00

#405 Epic: RAM/VRAM management -- one budget, tiered texture storage, run on a stated minimum

sky_line referenced this issue

2026-10-05 06:04:01 +00:00

Epic: RAM/VRAM management -- one budget, tiered texture storage, run on a stated minimum #405