Epic: RAM/VRAM management -- one budget, tiered texture storage, run on a stated minimum

sky_line commented

2026-10-05 03:08:10 +00:00

Owner

One plan for how Slipstream uses RAM and VRAM. Goals:

Stay under a budget we can state. We should publish a real minimum spec and actually run on it.
Share the machine. Running a second viewer, or a game, must not tip it into thrashing (#403).
Use memory when it's free. Cache aggressively when there's room, and fall back to disk when there isn't.

This epic ties together every open memory ticket. The plan below was reviewed by the owner on 2026-10-05. The decisions section overrides anything in the draft that disagrees with it.

What exists today (2026-10-05)

VRAM
- A texture residency budget derived from VK_EXT_memory_budget (per-process heap budget).
- A pressure mode at 90% usage that raises mip bias, trims pools and reduces RT.
- Alt-tab recovery when the OS budget shrinks.
- Epic #282 already reorders this so the texture pool absorbs pressure first and RT fits 8 GB cards.
RAM
- LLMemoryGovernor (#262) samples available system memory (MemAvailable / GlobalMemoryStatusEx, Linux PSI) and keeps a reserve for other programs.
- States are COMFORTABLE / TIGHT / CRITICAL. It paces heavy decode and upload work, and asks owners to trim retained decoded copies when not comfortable.
Measured tonight (quiet region, 1080p, RT on)
- RSS 6.1 GB, of which 1.7 GB is CPU texture copies.
- VRAM 6.1 GB: tracked 4.6 GB (tex 914, geom ~820, RT AS 667, RT mask 354, probes 277, cull 253, shadow 156, rt-aux 151, staging 100) plus ~1.5 GB untracked.
Baseline 2026-09-29 (busy two-sim view)
- RSS 28.3 GB, of which 16.7 GB is CPU texture copies.
- VRAM 9.2 GB.

Estimated minimum (to be confirmed by measurement)

VRAM, RT off (low probes, 1K shadows, coarse textures, 1080p busy scene): ~2.5-3 GB.
VRAM, RT on: ~4-4.5 GB.
Proposed spec: minimum 4 GB VRAM, recommended 8 GB. Final numbers come from part E.

Decisions (owner, 2026-10-05)

Order: A and B in parallel. They touch mostly separate code (A: renderer and budget; B: texture pipeline and cache), so they can go to separate agents.
Reserve: 10% of VRAM. The reserve is not off-limits. Usage inside it is allowed, but entering it means the soft ladder drops fast: multiple steps per check, shorter intervals, no waiting for hysteresis. Outside the reserve, normal pacing applies.
Second instance: rely on the new budget system (OS per-process budget, continuous reaction, soft ladder). No special half-budget start for now; revisit after A ships if #403-style cases still thrash.
Soft ladder order (temporary, automatic, settings untouched). Within each texture step, the largest textures go first, for the most memory freed per step:
1. Off-screen textures: mip down, largest first.
2. Reflection probes: lower resolution, then fewer probes (#295).
3. Far textures: mip down, largest first.
4. Mesh / terrain detail: aggressive LOD down (effective LOD factor, mesh and terrain detail), far first.
5. Far shadows off, both RT and shadow maps (distance-limited), before the geometry itself goes.
6. Far geometry drop: unload far objects and their textures, and lower the effective draw distance so they don't stream back in until pressure is gone.
7. Smaller RT budget: fewer/coarser BLAS (distant and rigged avatars first), lower RT mask resolution, shorter RT distance (#164).
8. Mirrors off: hero/planar mirror reflections stop rendering and release their targets.
9. Near textures: mip down, largest first.
10. Projector shadows off.
11. Sun shadows off.
12. Render scale through FSR (internal resolution), the last soft step.
Recovery runs in reverse order, automatically, with hysteresis. Turning a shadow type off here is temporary and doesn't touch the setting. The hard step (write + latch + notify) stays reserved for crash avoidance only.
Minimum spec: 4 GB VRAM minimum, 8 GB recommended, until the 2 GB stretch goal (#408) is met.
- Once #407 and the per-release benchmark (#163) give measured numbers, list actual card names.
- Possibly keep a wiki page of supported/tested GPUs with notes per card.

Draft plan

A. One VRAM budget that everything fits under (priority 1)

Rule: ordinary pressure never changes the user's settings. Settings are the user's ceiling. Under pressure the budget manager lowers effective values below that ceiling, temporarily, and puts them back when memory frees up. Rewriting a setting is reserved for a last-ditch step to keep the viewer from crashing.

Budget: min(user cap, OS per-process budget) - reserve.
- The user cap comes from a new Advanced Graphics "Maximum video memory" setting (Auto, or a size).
- The reserve leaves room for the compositor, browser and other apps.
Fixed costs first. Swapchain, render targets, rings and cull buffers scale with resolution and settings. Account for them, including the ~1.5 GB currently untracked (#244, #407).

Soft ladder (temporary, automatic, settings untouched). When over budget, step down, lightest visual impact first:

Texture detail: global mip level, with distant / small / off-screen textures first and your own avatar and HUDs last (exists today).
Ray tracing budget: cap RT geometry (fewer and coarser BLAS for distant or rigged avatars), lower RT mask resolution, shorter RT distance; fall back to shadow maps for some lights internally (#164).
Probes: effective resolution, then effective count (#295).
Shadow map resolution.
Render scale through FSR (internal resolution).

Rules for the soft ladder:

Each step is an effective cap owned by the budget manager. Preferences keep showing the user's chosen values.
Recovery is automatic, one step at a time, after usage has stayed below a lower threshold for a while. There's hysteresis and back-off, so it doesn't flap (#268 lesson).
No popups. The current step and what it is holding back show in a status readout (part D) and in the log. An optional, unobtrusive "reduced for memory" indicator is open for discussion.
Setting changes that would need more than the budget allows (e.g. raising probe resolution) are pre-checked (#226). Instead of crashing, the new setting applies up to what fits, with a note in the readout. The setting itself keeps the user's choice.

Hard step (last resort, crash avoidance only).

When: the soft ladder is fully exhausted and allocations are failing or OOM is imminent (device-loss risk).
What: turn off the single biggest optional consumer (RT first, then probes, then high shadow resolution). It writes the real setting, latches it (no automatic re-enable), and shows one notification saying what changed, why, and where to turn it back on.
This replaces today's silent RT auto-off (#227).

Sharing between processes

On Windows the OS budget already shrinks when another process takes VRAM. We react to it continuously, not only at 90% or on focus change.
On Linux/NVIDIA the budget is closer to "free + ours", so we also cap ourselves explicitly.
A second Slipstream instance detects the first (instance marker) and starts with half the budget. That's a budget split, not a settings change, and it directly fixes #403.

Floor: if even the hard step can't get under budget, we keep running at the minimum and warn once, instead of thrashing.

B. Tiered texture storage: RAM when it's cheap, disk when it isn't (#159, #274)

Tier	What	Cost to restore to GPU
0	GPU-resident (VRAM)	none
1	Decoded copy in RAM	upload only
2	GPU-ready disk cache: block-compressed (BC) textures stored on disk (#274)	disk read + upload, no J2C decode / re-encode
3	J2C asset cache	disk read + decode + BC encode + upload

Tier 2 is the key. Restoring from disk becomes cheap, so most tier-1 RAM copies can go.
Tier 1 is kept only where a feature needs CPU pixels. Audit and list them; candidates are sculpt maps, alpha/pick masks, avatar bake/composite inputs, local textures, media, terrain composition and the texture picker/preview. Everything else drops its CPU copy after upload.
The governor drives the policy. This is soft, with no setting changes.
- COMFORTABLE: keep tier-1 copies up to a RAM-scaled allowance, for the fastest re-upload.
- TIGHT / CRITICAL: drop tier-1 copies (except the required ones) and serve from tier 2. Copies come back as RAM frees.
Advanced Graphics setting "Texture memory: Auto / Prefer disk cache (lowest RAM)": forces tier-2 serving regardless of free RAM.
Tier 2 needs: a byte budget and LRU (no unbounded growth), a format/version key, and validation of partial or corrupt files. The sound cache (#176) set the pattern for temp-file-and-rename plus header checks.

C. Transient peaks and other RAM

Login/teleport peak (#262): pace by the governor, and with tier 2 bursts decode less.
Mesh and other decoded caches stay under governor-scaled budgets.

D. Make it visible

One status line in Preferences (or the stats window): VRAM budget / used / fixed / per consumer, RAM tiers, and the current ladder step.
Logs: a one-line summary on each ladder change.

E. Measure and guard (#163)

A "low memory" test preset (4 GB VRAM cap, RT off, low probes) plus a busy-scene benchmark, run per release, recording RSS and VRAM peaks.
That turns the minimum spec into a measured number and catches regressions.

Child tickets

VRAM budget and ladder:

#282 Epic: VRAM pressure redesign (texture pool first, RT fits 8 GB)
#164 VRAM: ray tracing must fit a VRAM-scaled budget
#295 VRAM pressure: reflection probes as a ladder step
#226 VRAM: pre-check setting-driven allocations, revert + notify
#227 RT: rethink the silent auto-off / degrade under low VRAM
#244 VRAM footprint: investigate leak / over-allocation (untracked VRAM)
#403 Two viewers on one machine thrash (8 GB card, 16 GB RAM)
#407 VRAM: find and track the untracked ~1.5 GB; feed measured overhead into the budget
#411 VRAM: suballocate textures from shared blocks (~0.9 GB page padding)

RAM and texture tiers:

#159 RAM: CPU texture copies kept after GPU upload
#274 GPU-ready texture cache: block-compressed textures on disk
#262 RAM: transient peak during login/teleport
#263 Explore: further texture memory reductions
#300 Texture cache I/O pattern and LRU (tier-2 read performance)
#302 Asset storage: duplicate cache probes, main-thread disk round trip

Measurement:

#163 Track RAM + VRAM per release

Stretch:

#408 Run in 2 GB of VRAM (iGPUs / very low spec): very-low tier and its feature cuts

#296 PTLAS / cluster BLAS, for RT memory and perf.

One plan for how Slipstream uses RAM and VRAM. Goals: - **Stay under a budget we can state.** We should publish a real minimum spec and actually run on it. - **Share the machine.** Running a second viewer, or a game, must not tip it into thrashing (#403). - **Use memory when it's free.** Cache aggressively when there's room, and fall back to disk when there isn't. This epic ties together every open memory ticket. The plan below was reviewed by the owner on 2026-10-05. **The decisions section overrides anything in the draft that disagrees with it.** ## What exists today (2026-10-05) - **VRAM** - A texture residency budget derived from `VK_EXT_memory_budget` (per-process heap budget). - A pressure mode at 90% usage that raises mip bias, trims pools and reduces RT. - Alt-tab recovery when the OS budget shrinks. - Epic #282 already reorders this so the texture pool absorbs pressure first and RT fits 8 GB cards. - **RAM** - `LLMemoryGovernor` (#262) samples available system memory (MemAvailable / GlobalMemoryStatusEx, Linux PSI) and keeps a reserve for other programs. - States are COMFORTABLE / TIGHT / CRITICAL. It paces heavy decode and upload work, and asks owners to trim retained decoded copies when not comfortable. - **Measured tonight (quiet region, 1080p, RT on)** - RSS 6.1 GB, of which 1.7 GB is CPU texture copies. - VRAM 6.1 GB: tracked 4.6 GB (tex 914, geom ~820, RT AS 667, RT mask 354, probes 277, cull 253, shadow 156, rt-aux 151, staging 100) plus ~1.5 GB untracked. - **Baseline 2026-09-29 (busy two-sim view)** - RSS 28.3 GB, of which **16.7 GB is CPU texture copies**. - VRAM 9.2 GB. ## Estimated minimum (to be confirmed by measurement) - **VRAM, RT off** (low probes, 1K shadows, coarse textures, 1080p busy scene): **~2.5-3 GB**. - **VRAM, RT on**: ~4-4.5 GB. - **Proposed spec:** minimum 4 GB VRAM, recommended 8 GB. Final numbers come from part E. ## Decisions (owner, 2026-10-05) 1. **Order: A and B in parallel.** They touch mostly separate code (A: renderer and budget; B: texture pipeline and cache), so they can go to separate agents. 2. **Reserve: 10% of VRAM.** The reserve is not off-limits. Usage *inside* it is allowed, but entering it means the soft ladder drops fast: multiple steps per check, shorter intervals, no waiting for hysteresis. Outside the reserve, normal pacing applies. 3. **Second instance: rely on the new budget system** (OS per-process budget, continuous reaction, soft ladder). No special half-budget start for now; revisit after A ships if #403-style cases still thrash. 4. **Soft ladder order** (temporary, automatic, settings untouched). Within each texture step, the **largest textures go first**, for the most memory freed per step: 1. Off-screen textures: mip down, largest first. 2. Reflection probes: lower resolution, then fewer probes (#295). 3. Far textures: mip down, largest first. 4. Mesh / terrain detail: aggressive LOD down (effective LOD factor, mesh and terrain detail), far first. 5. Far shadows off, both RT and shadow maps (distance-limited), before the geometry itself goes. 6. Far geometry drop: unload far objects **and their textures**, and lower the effective draw distance so they don't stream back in until pressure is gone. 7. Smaller RT budget: fewer/coarser BLAS (distant and rigged avatars first), lower RT mask resolution, shorter RT distance (#164). 8. Mirrors off: hero/planar mirror reflections stop rendering and release their targets. 9. Near textures: mip down, largest first. 10. Projector shadows off. 11. Sun shadows off. 12. Render scale through FSR (internal resolution), the last soft step. Recovery runs in reverse order, automatically, with hysteresis. Turning a shadow type off here is temporary and doesn't touch the setting. The hard step (write + latch + notify) stays reserved for crash avoidance only. 5. **Minimum spec: 4 GB VRAM minimum, 8 GB recommended**, until the 2 GB stretch goal (#408) is met. - Once #407 and the per-release benchmark (#163) give measured numbers, list actual card names. - Possibly keep a wiki page of supported/tested GPUs with notes per card. ## Draft plan ### A. One VRAM budget that everything fits under (priority 1) **Rule: ordinary pressure never changes the user's settings.** Settings are the user's ceiling. Under pressure the budget manager lowers *effective* values below that ceiling, temporarily, and puts them back when memory frees up. Rewriting a setting is reserved for a last-ditch step to keep the viewer from crashing. - **Budget:** `min(user cap, OS per-process budget) - reserve`. - The user cap comes from a new **Advanced Graphics "Maximum video memory"** setting (Auto, or a size). - The reserve leaves room for the compositor, browser and other apps. - **Fixed costs first.** Swapchain, render targets, rings and cull buffers scale with resolution and settings. Account for them, including the ~1.5 GB currently untracked (#244, #407). **Soft ladder (temporary, automatic, settings untouched).** When over budget, step down, lightest visual impact first: 1. Texture detail: global mip level, with distant / small / off-screen textures first and your own avatar and HUDs last (exists today). 2. Ray tracing budget: cap RT geometry (fewer and coarser BLAS for distant or rigged avatars), lower RT mask resolution, shorter RT distance; fall back to shadow maps for some lights *internally* (#164). 3. Probes: effective resolution, then effective count (#295). 4. Shadow map resolution. 5. Render scale through FSR (internal resolution). Rules for the soft ladder: - Each step is an effective cap owned by the budget manager. Preferences keep showing the user's chosen values. - **Recovery is automatic**, one step at a time, after usage has stayed below a lower threshold for a while. There's hysteresis and back-off, so it doesn't flap (#268 lesson). - **No popups.** The current step and what it is holding back show in a status readout (part D) and in the log. An optional, unobtrusive "reduced for memory" indicator is open for discussion. - Setting changes that would need more than the budget allows (e.g. raising probe resolution) are pre-checked (#226). Instead of crashing, the new setting applies up to what fits, with a note in the readout. The setting itself keeps the user's choice. **Hard step (last resort, crash avoidance only).** - **When:** the soft ladder is fully exhausted **and** allocations are failing or OOM is imminent (device-loss risk). - **What:** turn off the single biggest optional consumer (RT first, then probes, then high shadow resolution). It writes the real setting, latches it (no automatic re-enable), and shows **one** notification saying what changed, why, and where to turn it back on. - This replaces today's silent RT auto-off (#227). **Sharing between processes** - On Windows the OS budget already shrinks when another process takes VRAM. We react to it continuously, not only at 90% or on focus change. - On Linux/NVIDIA the budget is closer to "free + ours", so we also cap ourselves explicitly. - A second Slipstream instance detects the first (instance marker) and starts with half the budget. That's a budget split, not a settings change, and it directly fixes #403. **Floor:** if even the hard step can't get under budget, we keep running at the minimum and warn once, instead of thrashing. ### B. Tiered texture storage: RAM when it's cheap, disk when it isn't (#159, #274) | Tier | What | Cost to restore to GPU | |---|---|---| | 0 | GPU-resident (VRAM) | none | | 1 | Decoded copy in RAM | upload only | | 2 | **GPU-ready disk cache**: block-compressed (BC) textures stored on disk (#274) | disk read + upload, **no J2C decode / re-encode** | | 3 | J2C asset cache | disk read + decode + BC encode + upload | - **Tier 2 is the key.** Restoring from disk becomes cheap, so most tier-1 RAM copies can go. - **Tier 1 is kept only where a feature needs CPU pixels.** Audit and list them; candidates are sculpt maps, alpha/pick masks, avatar bake/composite inputs, local textures, media, terrain composition and the texture picker/preview. Everything else drops its CPU copy after upload. - **The governor drives the policy.** This is soft, with no setting changes. - COMFORTABLE: keep tier-1 copies up to a RAM-scaled allowance, for the fastest re-upload. - TIGHT / CRITICAL: drop tier-1 copies (except the required ones) and serve from tier 2. Copies come back as RAM frees. - **Advanced Graphics setting "Texture memory: Auto / Prefer disk cache (lowest RAM)"**: forces tier-2 serving regardless of free RAM. - **Tier 2 needs:** a byte budget and LRU (no unbounded growth), a format/version key, and validation of partial or corrupt files. The sound cache (#176) set the pattern for temp-file-and-rename plus header checks. ### C. Transient peaks and other RAM - Login/teleport peak (#262): pace by the governor, and with tier 2 bursts decode less. - Mesh and other decoded caches stay under governor-scaled budgets. ### D. Make it visible - **One status line** in Preferences (or the stats window): VRAM budget / used / fixed / per consumer, RAM tiers, and the current ladder step. - **Logs:** a one-line summary on each ladder change. ### E. Measure and guard (#163) - A "low memory" test preset (4 GB VRAM cap, RT off, low probes) plus a busy-scene benchmark, run per release, recording RSS and VRAM peaks. - That turns the minimum spec into a measured number and catches regressions. ## Child tickets VRAM budget and ladder: - [ ] #282 Epic: VRAM pressure redesign (texture pool first, RT fits 8 GB) - [x] #164 VRAM: ray tracing must fit a VRAM-scaled budget - [x] #295 VRAM pressure: reflection probes as a ladder step - [x] #226 VRAM: pre-check setting-driven allocations, revert + notify - [x] #227 RT: rethink the silent auto-off / degrade under low VRAM - [x] #244 VRAM footprint: investigate leak / over-allocation (untracked VRAM) - [x] #403 Two viewers on one machine thrash (8 GB card, 16 GB RAM) - [x] #407 VRAM: find and track the untracked ~1.5 GB; feed measured overhead into the budget - [x] #411 VRAM: suballocate textures from shared blocks (~0.9 GB page padding) RAM and texture tiers: - [x] #159 RAM: CPU texture copies kept after GPU upload - [x] #274 GPU-ready texture cache: block-compressed textures on disk - [x] #262 RAM: transient peak during login/teleport - [x] #263 Explore: further texture memory reductions - [x] #300 Texture cache I/O pattern and LRU (tier-2 read performance) - [ ] #302 Asset storage: duplicate cache probes, main-thread disk round trip Measurement: - [ ] #163 Track RAM + VRAM per release Stretch: - [ ] #408 Run in 2 GB of VRAM (iGPUs / very low spec): very-low tier and its feature cuts Related: - #296 PTLAS / cluster BLAS, for RT memory and perf.

sky_line added the

perf

epic

labels

Rows
Columns

Epic: RAM/VRAM management -- one budget, tiered texture storage, run on a stated minimum #405