Memory-safe, fast shader/pipeline compilation: per-pipeline measurement + memory-aware compile scheduler #471

New issue

Closed

opened 2026-10-05 23:08:05 +00:00 by sky_line · 1 comment

sky_line commented

2026-10-05 23:08:05 +00:00

Owner

Our cold-compile data (NVIDIA, 47 threads): 111 pipelines requested, 64 unique compiles, 283 s of total compile work, 26 s wall time. One pipeline alone takes 25.9 s. A warm cache takes about 30 ms.

Plan

Per-pipeline measurement:
- Log each pipeline's name, compile time and process memory growth during the compile (Windows: commit / PrivateUsage; Linux: RSS).
- Log a top-10 per batch, and add a summary to bug reports.
Memory-aware scheduler:
- Each compile reserves an estimated cost before it starts, and only starts if reserved + estimate fits in live free memory (Windows: commit headroom) minus a reserve.
- Estimates are learned from step 1 and saved per GPU/driver next to the pipeline cache.
- Compile the heaviest pipelines first. Small pipelines fill the spare threads; heavy ones are serialized when memory is tight.
- Pause while the memory governor is tight or critical.
Later: split the work into tiers (first-frame pipelines first, ray-query variants only when ray tracing is in use), shrink the 26 s pipeline, test the bindless-table-size theory, a "Preparing graphics…" progress UI, and graphics pipeline libraries.

Goals: peak memory bounded on every driver; cold-start wall time near the slowest single pipeline on machines with enough memory; warm starts unchanged.

Follow-up to #462 / #458: pipeline compiles at the login screen used about 30–49 GB of committed memory on AMD's Windows driver. That memory is driver-internal; our own Vulkan allocations were about 420 MB. Beta 3's static cap fixed the reporter's case, but it guesses (about 2 GB per compile). **Our cold-compile data (NVIDIA, 47 threads):** 111 pipelines requested, 64 unique compiles, 283 s of total compile work, 26 s wall time. **One pipeline alone takes 25.9 s.** A warm cache takes about 30 ms. ## Plan 1. **Per-pipeline measurement:** - Log each pipeline's name, compile time and process memory growth during the compile (Windows: commit / PrivateUsage; Linux: RSS). - Log a top-10 per batch, and add a summary to bug reports. 2. **Memory-aware scheduler:** - Each compile reserves an estimated cost before it starts, and only starts if reserved + estimate fits in live free memory (Windows: commit headroom) minus a reserve. - Estimates are learned from step 1 and saved per GPU/driver next to the pipeline cache. - Compile the heaviest pipelines first. Small pipelines fill the spare threads; heavy ones are serialized when memory is tight. - Pause while the memory governor is tight or critical. 3. **Later:** split the work into tiers (first-frame pipelines first, ray-query variants only when ray tracing is in use), shrink the 26 s pipeline, test the bindless-table-size theory, a "Preparing graphics…" progress UI, and graphics pipeline libraries. **Goals:** peak memory bounded on every driver; cold-start wall time near the slowest single pipeline on machines with enough memory; warm starts unchanged.

sky_line added the

labels

2026-10-05 23:08:05 +00:00

sky_line referenced this issue from a commit

2026-10-06 09:43:22 +00:00

VK: name + measure every batched pipeline compile (time, attributed memory, concurrency)

sky_line referenced this issue from a commit

2026-10-06 09:43:22 +00:00