Return pipeline-compile memory to the OS after each compile batch (OOM on 16 GB after first launch) #1103

New issue

Open

opened 2026-10-26 05:25:53 +00:00 by sky_line · 0 comments

sky_line commented

2026-10-26 05:25:53 +00:00

Owner

Pipeline compile memory is not returned to the OS after a compile batch.

Seen in #1059 (Intel Arc B580, Mesa 26.2.5, 16 GB RAM + zram): first launch after an update discards the pipeline cache, so the viewer cold-compiles ~200 pipelines on up to 15 threads (~500 s of compile time). The session then sits at 8.6-9.9 GB resident (vs ~4 GB on a larger scene on the owner's NVIDIA box with a warm cache), swaps heavily into zram, and is OOM-killed after ~6 minutes. The reporter cannot reproduce it on later launches (warm cache). The memory is freed but retained by the allocator: glibc keeps per-thread arenas, and the driver compiler allocates heavily on every compile thread.

Fix:

Return freed memory after each pipeline batch (world, post effects, spec sets): malloc_trim(0) when a batch completes (Linux/glibc; no-op elsewhere).
Bound the arenas the compile threads can create (mallopt(M_ARENA_MAX, ...) or an equivalent per-thread policy), so 15 compile threads cannot spread memory over 15 arenas.
The compile budget follows live memory: today it is computed once at startup ("compile budget now 7002 MB"); it must shrink while the memory governor is tight/critical, and assume heavier compiles on Mesa.
Log resident memory before/after each batch (and the trim's result) so reports show whether it worked.

Verify: cold-cache start (delete vk_pipeline_cache.bin + vk_spec_*.bin) vs warm start; resident memory after the batch should drop to the warm-start level.

Pipeline compile memory is not returned to the OS after a compile batch. **Seen in #1059** (Intel Arc B580, Mesa 26.2.5, 16 GB RAM + zram): first launch after an update discards the pipeline cache, so the viewer cold-compiles ~200 pipelines on up to 15 threads (~500 s of compile time). The session then sits at 8.6-9.9 GB resident (vs ~4 GB on a larger scene on the owner's NVIDIA box with a warm cache), swaps heavily into zram, and is OOM-killed after ~6 minutes. The reporter cannot reproduce it on later launches (warm cache). The memory is freed but retained by the allocator: glibc keeps per-thread arenas, and the driver compiler allocates heavily on every compile thread. **Fix:** 1. Return freed memory after each pipeline batch (world, post effects, spec sets): `malloc_trim(0)` when a batch completes (Linux/glibc; no-op elsewhere). 2. Bound the arenas the compile threads can create (`mallopt(M_ARENA_MAX, ...)` or an equivalent per-thread policy), so 15 compile threads cannot spread memory over 15 arenas. 3. The compile budget follows live memory: today it is computed once at startup ("compile budget now 7002 MB"); it must shrink while the memory governor is tight/critical, and assume heavier compiles on Mesa. 4. Log resident memory before/after each batch (and the trim's result) so reports show whether it worked. **Verify:** cold-cache start (delete `vk_pipeline_cache.bin` + `vk_spec_*.bin`) vs warm start; resident memory after the batch should drop to the warm-start level.

sky_line added the

labels

2026-10-26 05:25:53 +00:00

sky_line self-assigned this

2026-10-26 05:25:53 +00:00

sky_line added this to the Slipstream project

2026-10-26 05:25:53 +00:00