Perf: frame rate degrades after moving between sims (leaks / leftover per-frame work) #169

New issue

Closed

opened 2026-09-29 15:55:11 +00:00 by sky_line · 1 comment

sky_line commented

2026-09-29 15:55:11 +00:00

Owner

The same scene runs at >150 FPS right after a fresh login. After the session had moved between sims earlier, it measured ~88 FPS (11.35 ms/frame), with the GPU busy for ~11.2 ms of that. Something from regions we left, or from the teleport/region-crossing path, keeps costing time or memory.

What to look for

Compare a fresh login with a session that has visited N sims, in the same spot and on the same camera:

Scene stores: instance, slot and mesh-asset counts and bytes. Check that objects from regions we left are actually removed from LLCleanInstanceStore / mesh slots, and are no longer submitted to the cull or ingest (vk_ingestTick, vk_syncFromStores, sweep_total, instance count in the cull push).
GPU per-frame work: c_cull / c_shadows / c_preskin / c_tlas / c_blas_refit against the live instance count; shadow casters and RT BLAS/TLAS entries for objects that are gone; avatars (skin palettes, GPU animation slots, joint-table slots) from earlier regions.
Lights and probes: local lights and reflection probes still registered for objects that are gone (bwf_lights, bwf_probes, probe capture cost).
Textures: residency and demand entries for textures only used in the old region, and CPU copies (cpu tex in the VKMem line).
Region objects: LLViewerRegion / LLSurface / water objects, terrain patches, sim-surround meshes for regions no longer connected.
Memory: RSS / VRAM growth per sim change (see #163 for the method).

Method

Log in fresh in a fixed spot. Capture 30 s (RenderVKSectionLog + RenderVKGpuTimers), plus the VKMem line and instance and light counts.
Teleport to 3–5 busy sims, then back to the same spot and camera. Capture again.
Diff every section, GPU pass and store count. Anything that scales with sims visited rather than with what's on screen is the leak.

Related: #163 (per-release RAM/VRAM tracking), #157 (CPU cost in busy sims).

The same scene runs at **>150 FPS right after a fresh login**. After the session had moved between sims earlier, it measured **~88 FPS (11.35 ms/frame)**, with the GPU busy for ~11.2 ms of that. Something from regions we left, or from the teleport/region-crossing path, keeps costing time or memory. ## What to look for Compare a fresh login with a session that has visited N sims, in the same spot and on the same camera: - **Scene stores:** instance, slot and mesh-asset counts and bytes. Check that objects from regions we left are actually removed from `LLCleanInstanceStore` / mesh slots, and are no longer submitted to the cull or ingest (`vk_ingestTick`, `vk_syncFromStores`, `sweep_total`, instance count in the cull push). - **GPU per-frame work:** `c_cull` / `c_shadows` / `c_preskin` / `c_tlas` / `c_blas_refit` against the live instance count; shadow casters and RT BLAS/TLAS entries for objects that are gone; avatars (skin palettes, GPU animation slots, joint-table slots) from earlier regions. - **Lights and probes:** local lights and reflection probes still registered for objects that are gone (`bwf_lights`, `bwf_probes`, probe capture cost). - **Textures:** residency and demand entries for textures only used in the old region, and CPU copies (`cpu tex` in the VKMem line). - **Region objects:** `LLViewerRegion` / `LLSurface` / water objects, terrain patches, sim-surround meshes for regions no longer connected. - **Memory:** RSS / VRAM growth per sim change (see #163 for the method). ## Method 1. Log in fresh in a fixed spot. Capture 30 s (`RenderVKSectionLog` + `RenderVKGpuTimers`), plus the VKMem line and instance and light counts. 2. Teleport to 3–5 busy sims, then back to the same spot and camera. Capture again. 3. Diff every section, GPU pass and store count. Anything that scales with sims visited rather than with what's on screen is the leak. Related: #163 (per-release RAM/VRAM tracking), #157 (CPU cost in busy sims).

sky_line commented

2026-09-29 15:58:11 +00:00

Author

Owner

First measurement: same spot and camera, same build family, occlusion off. Session A had visited other sims. Session B is a fresh login. 30 s each.

	A: after sim changes	B: fresh login
Frame	11.35 ms (88 fps)	6.66 ms (150 fps)
GPU total	~11.2 ms	~6.4 ms
GPU compute	~6.0 ms	~2.1 ms
`c_cull`	0.65 ms	0.095 ms (~7x: the cull is iterating far more instances)
`c_shadows`	3.5 ms	0.80 ms
`c_stream_inst`	0.074 ms	0.022 ms
`c_tlas` / `c_blas_refit`	0.62 / 0.34 ms	0.38 / 0.20 ms
`c_preskin`	0.64 ms	0.40 ms
probe capture	~0.75 ms	~0.13 ms
CPU `vk_syncFromStores`	1.73 ms	0.79 ms
CPU `vk_texUpload`	1.53 ms	0.75 ms
CPU `vk_ingestTick`	1.20 ms	0.83 ms
CPU `bwf_lights`	0.83 ms	0.14 ms (~6x: lights from departed regions?)
CPU `LL_updateUI`	2.0 ms	0.87 ms

pass1 is the same in both (~2.3 ms): what's on screen didn't change. Everything that scales with registered scene size grew several-fold: cull, shadow casters, instance streaming, RT, lights, probes. That points to instances, lights and probes from regions we left staying registered. Next step: log the instance, light and probe counts before and after a teleport.

**First measurement: same spot and camera, same build family, occlusion off.** Session A had visited other sims. Session B is a fresh login. 30 s each. | | A: after sim changes | B: fresh login | |---|---|---| | Frame | 11.35 ms (88 fps) | 6.66 ms (150 fps) | | GPU total | ~11.2 ms | ~6.4 ms | | GPU compute | ~6.0 ms | ~2.1 ms | | `c_cull` | 0.65 ms | 0.095 ms (**~7x**: the cull is iterating far more instances) | | `c_shadows` | 3.5 ms | 0.80 ms | | `c_stream_inst` | 0.074 ms | 0.022 ms | | `c_tlas` / `c_blas_refit` | 0.62 / 0.34 ms | 0.38 / 0.20 ms | | `c_preskin` | 0.64 ms | 0.40 ms | | probe capture | ~0.75 ms | ~0.13 ms | | CPU `vk_syncFromStores` | 1.73 ms | 0.79 ms | | CPU `vk_texUpload` | 1.53 ms | 0.75 ms | | CPU `vk_ingestTick` | 1.20 ms | 0.83 ms | | CPU `bwf_lights` | 0.83 ms | 0.14 ms (**~6x**: lights from departed regions?) | | CPU `LL_updateUI` | 2.0 ms | 0.87 ms | `pass1` is the same in both (~2.3 ms): what's on screen didn't change. Everything that scales with *registered* scene size grew several-fold: cull, shadow casters, instance streaming, RT, lights, probes. That points to instances, lights and probes from regions we left staying registered. Next step: log the instance, light and probe counts before and after a teleport.

sky_line added this to the Slipstream project

2026-10-06 15:04:15 +00:00

sky_line closed this issue

2026-10-09 08:52:45 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

sky_line/Slipstream#169

No description provided.

Rows
Columns