Hitch (2-3 s) ~30-60 s after a region unloads: deferred cleanup of the dropped region's assets in one burst #290

New issue

Closed

opened 2026-10-04 09:48:53 +00:00 by sky_line · 0 comments

sky_line commented

2026-10-04 09:48:53 +00:00

Owner

Report: teleporting Horse Heights → Cloudsdale works and the old region unloads correctly (the region-reachability fix: Dropping region Horse Heights ...: unreachable at 09:42:20). But about 30-60 s after its objects disappear, the viewer hitches for 2-3 seconds.

Section/hitch logging was off in that session, so there's no HITCH / SECTIONS breakdown yet. Repro with RenderVKSectionLog / GPU timers on, teleport between two busy regions far apart, and capture the HITCH line.

State around it (VKMem at 09:42:40): inst 70k, mat freed 24966, mesh garbage v2724 i189 (high), CPU texture copies large, memgov comfortable.

The delay points at timed/deferred cleanup, not the drop itself. Suspects to check

CPU texture copy scale-down / thumbnailing. Thousands of textures from the dropped region stop being demanded at once. After VK_STALE_THUMB_SEC (15 s) / idle-thumb (30 s) they all become eligible together. Scale-down jobs are capped in flight (32/128), but the main-thread part (applyVKDemand, vkScaleDownCpuCopy posting, LLImageRaw frees) may still burst. The 30-60 s timing matches these timers.
GPU texture slot eviction (residency sweep window 600 passes / every 120) evicting a whole region's slots in one lap, with descriptor rewrites and vkFreeMemory / suballocator frees.
Mesh geometry garbage compaction: garbage is high after the drop. If compaction moves everything in one go (copy + rebinding offsets) that's a multi-second stall.
Material store frees (24966 freed) or instance store compaction.
Big CPU frees: malloc_trim after hundreds of MB freed (the memory governor's trim cadence), or freeing LLViewerFetchedTexture / LLImageRaw objects in bulk on the main thread (gTextureList cleanup of zero-ref images).
LLVOCache write of the dropped region on teardown (saveObjectCache), or texture cache writes.
vkDeviceWaitIdle in any resize/free path triggered by the shrink.

Fix direction

Identify the section via HITCH logs.
Then make the offending cleanup incremental and time-sliced (a per-frame budget like RenderVKTexResidencyBudgetMs): spread evictions, frees, compaction and trims across frames; offload bulk frees to a worker; never do whole-region cleanup in one frame.

**Report:** teleporting Horse Heights → Cloudsdale works and the old region unloads correctly (the region-reachability fix: `Dropping region Horse Heights ...: unreachable` at 09:42:20). But about **30-60 s after its objects disappear, the viewer hitches for 2-3 seconds**. Section/hitch logging was off in that session, so there's no `HITCH` / `SECTIONS` breakdown yet. Repro with `RenderVKSectionLog` / GPU timers on, teleport between two busy regions far apart, and capture the `HITCH` line. **State around it** (VKMem at 09:42:40): inst 70k, mat freed 24966, mesh garbage v2724 i189 (high), CPU texture copies large, memgov comfortable. ### The delay points at timed/deferred cleanup, not the drop itself. Suspects to check 1. **CPU texture copy scale-down / thumbnailing.** Thousands of textures from the dropped region stop being demanded at once. After `VK_STALE_THUMB_SEC` (15 s) / idle-thumb (30 s) they all become eligible together. Scale-down jobs are capped in flight (32/128), but the *main-thread* part (`applyVKDemand`, `vkScaleDownCpuCopy` posting, `LLImageRaw` frees) may still burst. The 30-60 s timing matches these timers. 2. **GPU texture slot eviction** (residency sweep window 600 passes / every 120) evicting a whole region's slots in one lap, with descriptor rewrites and `vkFreeMemory` / suballocator frees. 3. **Mesh geometry garbage compaction:** garbage is high after the drop. If compaction moves everything in one go (copy + rebinding offsets) that's a multi-second stall. 4. **Material store frees** (24966 freed) or **instance store compaction**. 5. **Big CPU frees:** `malloc_trim` after hundreds of MB freed (the memory governor's trim cadence), or freeing LLViewerFetchedTexture / LLImageRaw objects in bulk on the main thread (`gTextureList` cleanup of zero-ref images). 6. **LLVOCache write** of the dropped region on teardown (`saveObjectCache`), or texture cache writes. 7. `vkDeviceWaitIdle` in any resize/free path triggered by the shrink. ### Fix direction - Identify the section via HITCH logs. - Then make the offending cleanup incremental and time-sliced (a per-frame budget like `RenderVKTexResidencyBudgetMs`): spread evictions, frees, compaction and trims across frames; offload bulk frees to a worker; never do whole-region cleanup in one frame.

sky_line added the

bug

perf

labels

2026-10-04 09:48:53 +00:00

sky_line referenced this issue

2026-10-05 11:52:46 +00:00

Post-teleport hitch when old regions drop: ~7.8k objects torn down in single frames -- time-slice / batch region teardown #433

sky_line referenced this issue

2026-10-05 22:16:28 +00:00

Test plan: fly across all of Zindra (mainland) at max flight speed -- the region-crossing / streaming / memory endurance test #469

sky_line added this to the Slipstream project