Smoother region crossings (sim-to-sim transfer)

sky_line commented

2026-10-08 21:02:59 +00:00

Owner

Region (sim) crossings should be seamless: no hitch, no vehicle/avatar snap or rubber-band, no objects popping out/in, no camera jump, no stall while the new region's objects stream.

Symptoms today

Frame hitch at the crossing (CPU work on the main thread: region handoff, object reparenting, cache loads, re-registration into the VK scene).
Avatar/vehicle position snaps or freezes briefly while the agent is handed to the neighbour (interpolation stops at the border).
Objects of the new region pop in late, or neighbour-region content flickers as it is torn down/re-ingested.

Goal

Measure first (section/GPU timers + per-phase logging across a crossing on foot, flying and in a vehicle), then reduce the main-thread work and smooth motion across the handoff. Options writeup to follow in comments.

Labels: perf, feature.

Region (sim) crossings should be seamless: no hitch, no vehicle/avatar snap or rubber-band, no objects popping out/in, no camera jump, no stall while the new region's objects stream. ## Symptoms today - Frame hitch at the crossing (CPU work on the main thread: region handoff, object reparenting, cache loads, re-registration into the VK scene). - Avatar/vehicle position snaps or freezes briefly while the agent is handed to the neighbour (interpolation stops at the border). - Objects of the new region pop in late, or neighbour-region content flickers as it is torn down/re-ingested. ## Goal Measure first (section/GPU timers + per-phase logging across a crossing on foot, flying and in a vehicle), then reduce the main-thread work and smooth motion across the handoff. Options writeup to follow in comments. Labels: perf, feature.

sky_line added the

feature

perf

triage

labels

2026-10-08 21:02:59 +00:00

sky_line added this to the Slipstream project

2026-10-08 21:02:59 +00:00

sky_line commented

2026-10-08 21:08:18 +00:00

Author

Owner

How a crossing works today (code map, 2026-10-08)

The viewer never blocks on the server. Hitches are main-thread bursts in the crossing frame plus server handoff gaps.
Crossing frame (AgentMovementComplete): shiftObjects walks all objects; LLCleanInstanceStore::shiftOrigin rewrites every instance matrix and dirties the whole store, so the next frame does a full memcpy, updatePickBoxes(0,n) (~15 ms in dense sims per the code comment) and a full instance-buffer upload. The same frame invalidates the shadow-cascade cache, resets TAA history and does a full TLAS rebuild. updateWaterObjects runs twice and recreates hole-water.
Around the crossing:
- Each new ring region does a synchronous object-cache file read on RegionHandshake.
- Old-region teardown is synchronous: killObjects over all objects, cleanDeadObjects, a saveObjectCache disk write, and surface delete.
Server-bound:
- AgentUpdates (control input) keep going to the old sim until AgentMovementComplete.
- If the old sim's KillObject lands before the new sim's update, the vehicle or its children are killed and recreated (a pop).
Small issues:
- InterpolationTime and InterpolationPhaseOut are passed to the wrong setters (llviewerobjectlist.cpp ~943), so prediction hard-stops instead of tapering.
- A duplicate seed-cap fetch runs on every crossing.

Options

A. Measure: per-phase timers and logs across a crossing (on foot, flying, vehicle).
B. Quick wins: fix the interpolation setters; skip the duplicate caps fetch; rebuild water only when the hole set changes; cache the isRegionListed check in shiftObjects.
C. Remove the origin-shift frame (biggest client hitch):
- Keep a world-origin offset applied on the GPU (builds on #555 camera-relative rendering) instead of rewriting every instance.
- Rebase TAA history and the cached shadow cascades by the shift instead of resetting them.
- Refit the TLAS instead of rebuilding it.
D. Async region lifecycle: read the object cache on a worker, prefetched when the ring anchor moves; slice old-region teardown over several frames and move the cache write to a worker.
E. Mask the server handoff:
- Smart vehicle/avatar extrapolation through the handoff (our own version of a region-cross extrapolation limiter).
- Briefly defer KillObject for objects near a region edge so the new sim's update can re-key them instead of popping.
- Smooth the camera through the handoff.

## How a crossing works today (code map, 2026-10-08) - The viewer never blocks on the server. Hitches are main-thread bursts in the crossing frame plus server handoff gaps. - **Crossing frame (AgentMovementComplete):** `shiftObjects` walks all objects; `LLCleanInstanceStore::shiftOrigin` rewrites every instance matrix and dirties the whole store, so the next frame does a full memcpy, `updatePickBoxes(0,n)` (~15 ms in dense sims per the code comment) and a full instance-buffer upload. The same frame invalidates the shadow-cascade cache, resets TAA history and does a full TLAS rebuild. `updateWaterObjects` runs twice and recreates hole-water. - **Around the crossing:** - Each new ring region does a synchronous object-cache file read on RegionHandshake. - Old-region teardown is synchronous: `killObjects` over all objects, `cleanDeadObjects`, a `saveObjectCache` disk write, and surface delete. - **Server-bound:** - AgentUpdates (control input) keep going to the old sim until AgentMovementComplete. - If the old sim's KillObject lands before the new sim's update, the vehicle or its children are killed and recreated (a pop). - **Small issues:** - `InterpolationTime` and `InterpolationPhaseOut` are passed to the wrong setters (llviewerobjectlist.cpp ~943), so prediction hard-stops instead of tapering. - A duplicate seed-cap fetch runs on every crossing. ## Options - **A. Measure:** per-phase timers and logs across a crossing (on foot, flying, vehicle). - **B. Quick wins:** fix the interpolation setters; skip the duplicate caps fetch; rebuild water only when the hole set changes; cache the `isRegionListed` check in `shiftObjects`. - **C. Remove the origin-shift frame (biggest client hitch):** - Keep a world-origin offset applied on the GPU (builds on #555 camera-relative rendering) instead of rewriting every instance. - Rebase TAA history and the cached shadow cascades by the shift instead of resetting them. - Refit the TLAS instead of rebuilding it. - **D. Async region lifecycle:** read the object cache on a worker, prefetched when the ring anchor moves; slice old-region teardown over several frames and move the cache write to a worker. - **E. Mask the server handoff:** - Smart vehicle/avatar extrapolation through the handoff (our own version of a region-cross extrapolation limiter). - Briefly defer KillObject for objects near a region edge so the new sim's update can re-key them instead of popping. - Smooth the camera through the handoff.

sky_line commented

2026-10-08 23:00:29 +00:00

Author

Owner

Owner finding (2026-10-08): server-side handoff time is not measured

Some crossings lose control for seconds on the server side. That loss doesn't show up in our client-side numbers.
Client-side, the crossing frame stays at about 35–40 ms, the round trip at about 100 ms, and the first avatar update at about 200–260 ms.

Measure the server-side handoff in Crossing568:
- per worn attachment, when its first update arrives from the new sim, and when the last one arrives;
- when control comes back: the first server avatar update whose velocity follows the held movement input.
Show users which of their own attachments make crossings slow, so they can decide what to wear.
Masking: the server owns position, so local prediction through a long gap only smooths visuals and snaps back. It's worth it for short gaps only.

## Owner finding (2026-10-08): server-side handoff time is not measured - Some crossings lose control for seconds on the server side. That loss doesn't show up in our client-side numbers. - Client-side, the crossing frame stays at about 35–40 ms, the round trip at about 100 ms, and the first avatar update at about 200–260 ms. ## Next 1. Measure the server-side handoff in `Crossing568`: - per worn attachment, when its first update arrives from the new sim, and when the last one arrives; - when control comes back: the first server avatar update whose velocity follows the held movement input. 2. Show users which of their own attachments make crossings slow, so they can decide what to wear. 3. Masking: the server owns position, so local prediction through a long gap only smooths visuals and snaps back. It's worth it for short gaps only.

sky_line referenced this issue from a commit

2026-10-09 04:37:34 +00:00

Refs #568: region-crossing quick wins + Crossing568 measurement

sky_line referenced this issue from a commit

2026-10-09 04:37:34 +00:00

#568: keep the duplicate-seed caps re-check (SH-3895 safety net), log at INFO

sky_line referenced this issue from a commit

2026-10-09 04:37:35 +00:00

#568: CROSSING_SCOPE records the section timer without a second Tracy zone (clashed with LL_PROFILE_ZONE_SCOPED)

sky_line referenced this issue from a commit

2026-10-09 04:37:36 +00:00

#568: CROSSING_SCOPE drops the function-local LLTrace block timer (late-registered handle overran accumulator buffers: memory corruption / crash at world init)

sky_line referenced this issue from a commit

2026-10-09 04:37:36 +00:00

Merge branch 'llperf-vocache-terrain' into vulkan-port-tb (#568 cache-load trace moves to the async apply)

sky_line referenced this issue from a commit

2026-10-09 04:37:36 +00:00

Refs #568: measure attachment handoff and control restore in the crossing trace; slow-to-transfer attachment notice

sky_line referenced this issue from a commit

2026-10-09 04:37:36 +00:00

#568: control-restored uses the control flags as last sent (gAgent's are reset before messages are processed)

sky_line referenced this issue from a commit

2026-10-09 04:37:36 +00:00

Refs #568: stable render-world frame for the instance store (no O(N) rebase on a crossing)

sky_line referenced this issue from a commit

2026-10-09 04:37:36 +00:00

Refs #568: split the LL shift chunks in the crossing trace (object loop, shiftPos loop, shiftRegions)

sky_line referenced this issue from a commit

2026-10-09 04:37:36 +00:00

Refs #568: object cache I/O completions on their own per-frame queue

sky_line referenced this issue from a commit

2026-10-09 04:37:36 +00:00

Refs #568: incremental old-region teardown (delist now, kill objects in slices, save, delete)

sky_line referenced this issue from a commit

2026-10-09 04:37:36 +00:00

Refs #568: drop the tree ingest's now-unused agent-space position (signature is region-local)

sky_line referenced this issue from a commit

2026-10-09 04:37:36 +00:00

Refs #568: addRegion flushes the region it replaces (same handle) right after retiring it

sky_line added

status

in-progress

and removed

triage

labels

2026-10-09 04:40:42 +00:00

sky_line referenced this issue

2026-10-09 05:23:59 +00:00

Our own asset fetcher: pipelined HTTP/1.1 client for texture/mesh GETs to the asset CDN #485

sky_line referenced this issue from a commit

2026-10-09 08:32:54 +00:00

Fix crash: event poll posted raw `this` to gMainloopWork; capture sender by value + weak impl, stop the poll at region delist

sky_line commented

2026-10-09 09:25:24 +00:00

Author

Owner

All four options are in: A (handoff measurement + slow-attachment notice) and B-D (stable render-world frame with re-anchoring, VOCache completion queue, incremental region teardown) shipped in Beta 6; Beta 7 adds network draw distance, the 25-region ring cap with paced connects, and the global-stall guard, which removed the disconnects on long fast flights. Moving to testing; further crossing smoothness work gets its own tickets.

sky_line added

and removed

labels

2026-10-09 09:25:24 +00:00

sky_line referenced this issue

2026-10-09 22:59:07 +00:00

Region crossings: keep avatar assets (ours always, others within network draw distance) instead of unloading and reloading them #607

sky_line commented