RAM: retained CPU texture copies overshoot the allowance on big-region teleports (18 GB vs 4 GB) -- size the limit to available memory and evict undriven decoded copies #636

New issue

Open

opened 2026-10-10 10:36:43 +00:00 by sky_line · 0 comments

sky_line commented

2026-10-10 10:36:43 +00:00

Owner

What happened (2026-10-10, measured)

Teleport into a large region not visited in a long time (Canterlot), on a 256 GB machine, build b475586aaf + fixes:

Time (UTC)	RSS	Retained CPU texture copies	Decode in flight
10:32:37	~6 GB	~1.4 GB	~8 MB
10:33:52	15.9 GB	10.8 GB	4.1 GB
10:34:22	22.7 GB	18.3 GB	0
10:34:27	9.9 GB	6.3 GB	0
10:34:32	9.2 GB	5.2 GB	0

The breakdown at the peak was mostly "never VK-driven" copies (~10 GB): textures LL fetched and decoded for objects in view that no Vulkan material referenced yet, because ingest was still catching up on ~22k objects. We only release a CPU copy once a VK image covers it, so they stayed until ingest registered their objects, then dropped all at once.
The memory governor's retained allowance was 4096 MB, but retention reached 18 GB: denials only stop new decodes (up to ~380 denied/s), and nothing evicts copies that are already decoded.
On a 16 GB machine this teleport would likely have run out of memory.
Decoding itself was healthy: OpenJPEG 2.5.4, zero failures, ~14k fetches decoded in ~90 s, and the decode queue drained.

Do

Size the limit to the machine. Derive the retained-copy allowance (and the decode in-flight budget) from available physical memory and the memgov reserve, with sane floor and ceiling. For example, a fraction of avail - reserve, clamped to something like 512 MB to 4 GB, with the exact numbers justified in the PR. Recompute live as availability changes (other apps, PSI). A user setting can override it. This follows the soft-degrade rule: temporary caps, settings untouched.
Enforce the limit on what is already decoded, not only on new decodes. When retained copies exceed the allowance, drop the decoded CPU copies of textures no VK material uses yet ("never VK-driven" / undriven / off-screen first, LRU by last demand), keeping the compressed bytes (raw-bytes cache / disk cache). When ingest later needs one, re-decode it; decode is cheap per the numbers above.
Optional, measure first: don't let LL fetch and decode at full priority for objects ingest hasn't registered yet, so fetch can't run far ahead of what we can use.
Stats: the VKMem RAM: line gets allowance (and how it was derived), retained, evicted-undriven MB/count and re-decodes, plus the peak per teleport/login in MemTimeline.

Done when

The same teleport keeps retained copies at or under the computed allowance, and peak RSS drops accordingly. Check the MemTimeline[teleport] lines.
No texture stays blurry or blank because its copy was evicted: re-decode works, with no fetch churn (re-fetch only if raw bytes were evicted too).
On a small-RAM test (cap via the setting, or a cgroup/systemd-run -p MemoryMax=), a big-region teleport completes without OOM.

Related: #262 (transient peak during login/teleport, closed), #159 (CPU texture copies), #405 (RAM/VRAM epic), #585, #389.

## What happened (2026-10-10, measured) Teleport into a large region not visited in a long time (Canterlot), on a 256 GB machine, build `b475586aaf` + fixes: | Time (UTC) | RSS | Retained CPU texture copies | Decode in flight | |---|---|---|---| | 10:32:37 | ~6 GB | ~1.4 GB | ~8 MB | | 10:33:52 | 15.9 GB | 10.8 GB | 4.1 GB | | 10:34:22 | **22.7 GB** | **18.3 GB** | 0 | | 10:34:27 | 9.9 GB | 6.3 GB | 0 | | 10:34:32 | 9.2 GB | 5.2 GB | 0 | - The breakdown at the peak was mostly **"never VK-driven"** copies (~10 GB): textures LL fetched and decoded for objects in view that no Vulkan material referenced yet, because ingest was still catching up on ~22k objects. We only release a CPU copy once a VK image covers it, so they stayed until ingest registered their objects, then dropped all at once. - The memory governor's `retained` **allowance was 4096 MB**, but retention reached **18 GB**: denials only stop *new* decodes (up to ~380 denied/s), and nothing evicts copies that are already decoded. - On a 16 GB machine this teleport would likely have run out of memory. - Decoding itself was healthy: OpenJPEG 2.5.4, zero failures, ~14k fetches decoded in ~90 s, and the decode queue drained. ## Do 1. **Size the limit to the machine.** Derive the retained-copy allowance (and the decode in-flight budget) from available physical memory and the memgov reserve, with sane floor and ceiling. For example, a fraction of `avail - reserve`, clamped to something like 512 MB to 4 GB, with the exact numbers justified in the PR. Recompute live as availability changes (other apps, PSI). A user setting can override it. This follows the soft-degrade rule: temporary caps, settings untouched. 2. **Enforce the limit on what is already decoded, not only on new decodes.** When retained copies exceed the allowance, drop the decoded CPU copies of textures no VK material uses yet ("never VK-driven" / undriven / off-screen first, LRU by last demand), keeping the compressed bytes (raw-bytes cache / disk cache). When ingest later needs one, re-decode it; decode is cheap per the numbers above. 3. **Optional, measure first:** don't let LL fetch and decode at full priority for objects ingest hasn't registered yet, so fetch can't run far ahead of what we can use. 4. **Stats:** the VKMem `RAM:` line gets allowance (and how it was derived), retained, evicted-undriven MB/count and re-decodes, plus the peak per teleport/login in MemTimeline. ## Done when - The same teleport keeps retained copies at or under the computed allowance, and peak RSS drops accordingly. Check the MemTimeline[teleport] lines. - No texture stays blurry or blank because its copy was evicted: re-decode works, with no fetch churn (re-fetch only if raw bytes were evicted too). - On a small-RAM test (cap via the setting, or a cgroup/`systemd-run -p MemoryMax=`), a big-region teleport completes without OOM. Related: #262 (transient peak during login/teleport, closed), #159 (CPU texture copies), #405 (RAM/VRAM epic), #585, #389.

sky_line added the

perf

bug

labels

2026-10-10 10:36:43 +00:00

sky_line added this to the Slipstream project

2026-10-10 10:36:43 +00:00

sky_line referenced this issue

2026-10-12 06:11:42 +00:00

Allocation failures: system RAM exhaustion retries, then drops -- never crash #698