Investigate: why AMD Windows driver needs GBs of host memory per pipeline compile (OUT_OF_HOST_MEMORY) -- our shaders or driver bug? #673

New issue

Open

opened 2026-10-11 06:55:58 +00:00 by sky_line · 0 comments

sky_line commented

2026-10-11 06:55:58 +00:00

Owner

Investigation. Why does AMD's Windows driver (RX 6600 XT, 32.0.21045.5002) use so much host memory compiling our pipelines that vkCreateGraphicsPipelines returns VK_ERROR_OUT_OF_HOST_MEMORY? Our compile scheduler already assumes 3 GB per unknown pipeline on AMD, which is itself extreme. See #667 / #669 and the crash-workaround ticket.

Questions

Which pipelines are expensive? Measure per-pipeline peak host memory and compile time on AMD (and on RADV and NVIDIA for comparison), for example with the learned costs in vk_pipeline_costs.txt. Do the large ray-query lit variants (LIT_*_FragRt, the mask) or giant uber-shaders dominate?
What in the SPIR-V drives it?
- huge unrolled loops
- large constant arrays
- deeply inlined functions
- big switch statements
- ray-query code (GL_EXT_ray_query) with many candidate paths
- excessive specialization
- nonuniformEXT bindless indexing everywhere
- very long shaders produced by our include structure
Try bisecting by stripping features from one expensive pipeline.
Known AMD issues: Has AMD documented anything about pipeline-compile memory blow-ups with ray query or large shaders? Check driver release notes, GPUOpen and other engines' bug trackers. Would VK_EXT_pipeline_robustness, disabling robustness, pipeline libraries / VK_EXT_graphics_pipeline_library, or VK_KHR_pipeline_binary change it?
Workarounds on our side if it's in our shaders:
- split uber-shaders
- move rarely-used paths to separate pipelines
- reduce unrolling ([[dont_unroll]])
- reduce specialization constants
- precompile and ship pipeline caches per driver
If it's an AMD driver bug: document it, file it with AMD (GPUOpen / developer support) with a minimal repro, and keep the workarounds.

Deliverable

Findings with numbers. Then either shader changes, or a documented driver bug plus a confirmed workaround configuration.

**Investigation.** Why does AMD's Windows driver (RX 6600 XT, 32.0.21045.5002) use so much host memory compiling our pipelines that `vkCreateGraphicsPipelines` returns `VK_ERROR_OUT_OF_HOST_MEMORY`? Our compile scheduler already assumes **3 GB per unknown pipeline** on AMD, which is itself extreme. See #667 / #669 and the crash-workaround ticket. ## Questions 1. **Which pipelines are expensive?** Measure per-pipeline peak host memory and compile time on AMD (and on RADV and NVIDIA for comparison), for example with the learned costs in `vk_pipeline_costs.txt`. Do the large ray-query lit variants (`LIT_*_FragRt`, the mask) or giant uber-shaders dominate? 2. **What in the SPIR-V drives it?** - huge unrolled loops - large constant arrays - deeply inlined functions - big switch statements - ray-query code (`GL_EXT_ray_query`) with many candidate paths - excessive specialization - `nonuniformEXT` bindless indexing everywhere - very long shaders produced by our include structure Try bisecting by stripping features from one expensive pipeline. 3. **Known AMD issues:** Has AMD documented anything about pipeline-compile memory blow-ups with ray query or large shaders? Check driver release notes, GPUOpen and other engines' bug trackers. Would `VK_EXT_pipeline_robustness`, disabling robustness, pipeline libraries / `VK_EXT_graphics_pipeline_library`, or `VK_KHR_pipeline_binary` change it? 4. **Workarounds on our side if it's in our shaders:** - split uber-shaders - move rarely-used paths to separate pipelines - reduce unrolling (`[[dont_unroll]]`) - reduce specialization constants - precompile and ship pipeline caches per driver 5. **If it's an AMD driver bug:** document it, file it with AMD (GPUOpen / developer support) with a minimal repro, and keep the workarounds. ## Deliverable Findings with numbers. Then either shader changes, or a documented driver bug plus a confirmed workaround configuration.

sky_line added the

platform

perf

labels

2026-10-11 06:55:58 +00:00

sky_line added this to the Slipstream project

2026-10-11 06:55:58 +00:00

sky_line referenced this issue

2026-10-11 06:55:58 +00:00

[viewer] [crash] Crash: Crash on Log In #667

sky_line referenced this issue

2026-10-11 06:55:58 +00:00