sky_line/Slipstream

Fork 1

[Epic] VR support (OpenXR) — stretch goal, 2.0.0+ #251

New issue

Open

opened 2026-10-02 14:34:23 +00:00 by sky_line · 2 comments

sky_line commented

2026-10-02 14:34:23 +00:00

Owner

Stretch goal: probably targeted at 2.0.0 or later. This is an epic to scope and break into child tickets; it's not near-term work.

Goal

First-class VR in Slipstream on our Vulkan renderer, via OpenXR (the standard, vendor-neutral runtime API: SteamVR, Monado on Linux, Meta / WMR / Pico runtimes). It should not be a bolted-on second renderer: the GPU-driven pipeline renders both eyes efficiently.

Prior art

Firestorm VR mod (P373R)
Kokua / Singularity experiments

These are all GL, side-by-side or OpenVR-era. Useful for UX lessons on SL-specific problems, not code.

Major workstreams (each likely a child ticket)

OpenXR integration
- instance / session / swapchain via XR_KHR_vulkan_enable2;
- frame loop (xrWaitFrame / xrBeginFrame / xrEndFrame) driving our frame pacing;
- runtime discovery;
- a desktop mirror window.
Stereo rendering in the VK pipeline
- Multiview (VK_KHR_multiview) or layered rendering for main-view passes: pre-pass, G-buffer, deferred lighting compute, pass 2, post.
- Per-eye view / projection in the param ring (#238); per-eye cull, or a combined frustum cull with per-eye visibility.
- Asymmetric off-axis projections; reversed-Z per eye.
- Shadows / RT / probes shared across eyes (rendered once).
- Screen-space effects need stereo-correct handling, or must be disabled in VR: SSR, SSAO, god rays, TAA / FSR (per-eye history), DoF (off), motion blur (off), lens flare (off).
Performance budget
- 90+ Hz per eye at headset resolution (~2× 2k×2k).
- Foveated rendering (fixed, or eye-tracked where available: VRS / fragment density map).
- Reprojection / spacewarp friendliness: depth submission, motion vectors via XR_FB_space_warp where supported.
- Strict frame pacing (async compute already helps).
- CPU frame time (the SL object / avatar update path) is the likely limiter.
Head tracking and the SL camera model
- Head pose drives the camera with SL's agent / camera rules.
- Mouselook vs third person in VR.
- Comfort options: snap turn, vignette, seated / standing, height calibration.
Input
- OpenXR action sets for motion controllers: locomotion (walk / fly / teleport-to-point), turning, interaction.
- Pointing and clicking on prims (touch, sit, pay): a laser-pointer pick through our existing raycast.
- Edit tools in VR (stretch).
UI in VR
- The 2D UI (floaters, chat, IM, inventory) rendered to a world-space or head-locked quad / cylinder layer (OpenXR composition layers), with laser-pointer input.
- HUD attachments (SL HUDs are screen-space) need a VR treatment.
- A text-input story: a virtual keyboard and system keyboard.
Avatar embodiment (stretch)
- Hand / head IK driving your own avatar (animation override layer; local-only vs server-visible considerations).
- Hide own head in first person (we already have the mouselook head-hide path).
Media / CEF and audio
- Media-on-a-prim works as-is (world texture).
- Spatial audio already positional; verify HRTF / listener follows head pose.
Platform
- Linux (Monado / SteamVR) and Windows (SteamVR / Oculus / WMR).
- Feature-tiered: VR requires a capable GPU, and the desktop stays unaffected when VR is off.
- Settings / toggles to enter and leave VR at runtime.

Risks / open questions

SL content is not built for VR scale and comfort: tiny UI text, HUD-heavy workflows, camera-constrained sims.
CPU-side frame time in busy sims vs 90 Hz.
Interaction with RLVa camera restrictions.
Testing hardware coverage.

Dependencies / enablers already in place

Vulkan-only GPU-driven pipeline: multi-view probe cull, param ring per view (#238), async compute.
GL removal (no second renderer to keep in sync).
Frame pipelining (phase 3b).

**Stretch goal:** probably targeted at **2.0.0** or later. This is an epic to scope and break into child tickets; it's not near-term work. ## Goal First-class VR in Slipstream on our Vulkan renderer, via **OpenXR** (the standard, vendor-neutral runtime API: SteamVR, Monado on Linux, Meta / WMR / Pico runtimes). It should not be a bolted-on second renderer: the GPU-driven pipeline renders both eyes efficiently. ## Prior art - Firestorm VR mod (P373R) - Kokua / Singularity experiments These are all GL, side-by-side or OpenVR-era. Useful for UX lessons on SL-specific problems, not code. ## Major workstreams (each likely a child ticket) 1. **OpenXR integration** - instance / session / swapchain via `XR_KHR_vulkan_enable2`; - frame loop (`xrWaitFrame` / `xrBeginFrame` / `xrEndFrame`) driving our frame pacing; - runtime discovery; - a desktop mirror window. 2. **Stereo rendering in the VK pipeline** - Multiview (`VK_KHR_multiview`) or layered rendering for main-view passes: pre-pass, G-buffer, deferred lighting compute, pass 2, post. - Per-eye view / projection in the param ring (#238); per-eye cull, or a combined frustum cull with per-eye visibility. - Asymmetric off-axis projections; reversed-Z per eye. - Shadows / RT / probes shared across eyes (rendered once). - Screen-space effects need stereo-correct handling, or must be disabled in VR: SSR, SSAO, god rays, TAA / FSR (per-eye history), DoF (off), motion blur (off), lens flare (off). 3. **Performance budget** - 90+ Hz per eye at headset resolution (~2× 2k×2k). - Foveated rendering (fixed, or eye-tracked where available: VRS / fragment density map). - Reprojection / spacewarp friendliness: depth submission, motion vectors via `XR_FB_space_warp` where supported. - Strict frame pacing (async compute already helps). - CPU frame time (the SL object / avatar update path) is the likely limiter. 4. **Head tracking and the SL camera model** - Head pose drives the camera with SL's agent / camera rules. - Mouselook vs third person in VR. - Comfort options: snap turn, vignette, seated / standing, height calibration. 5. **Input** - OpenXR action sets for motion controllers: locomotion (walk / fly / teleport-to-point), turning, interaction. - Pointing and clicking on prims (touch, sit, pay): a laser-pointer pick through our existing raycast. - Edit tools in VR (stretch). 6. **UI in VR** - The 2D UI (floaters, chat, IM, inventory) rendered to a world-space or head-locked quad / cylinder layer (OpenXR composition layers), with laser-pointer input. - HUD attachments (SL HUDs are screen-space) need a VR treatment. - A text-input story: a virtual keyboard and system keyboard. 7. **Avatar embodiment (stretch)** - Hand / head IK driving your own avatar (animation override layer; local-only vs server-visible considerations). - Hide own head in first person (we already have the mouselook head-hide path). 8. **Media / CEF and audio** - Media-on-a-prim works as-is (world texture). - Spatial audio already positional; verify HRTF / listener follows head pose. 9. **Platform** - Linux (Monado / SteamVR) and Windows (SteamVR / Oculus / WMR). - Feature-tiered: VR requires a capable GPU, and the desktop stays unaffected when VR is off. - Settings / toggles to enter and leave VR at runtime. ## Risks / open questions - SL content is not built for VR scale and comfort: tiny UI text, HUD-heavy workflows, camera-constrained sims. - CPU-side frame time in busy sims vs 90 Hz. - Interaction with RLVa camera restrictions. - Testing hardware coverage. ## Dependencies / enablers already in place - Vulkan-only GPU-driven pipeline: multi-view probe cull, param ring per view (#238), async compute. - GL removal (no second renderer to keep in sync). - Frame pipelining (phase 3b).

sky_line added the

feature

rendering

epic

labels

2026-10-02 14:34:23 +00:00

sky_line added this to the Slipstream project

2026-10-06 15:04:13 +00:00

sky_line referenced this issue

2026-10-06 15:11:39 +00:00

Investigate SL Puppetry (LEAP-driven real-time avatar animation): status, code, fit with VR (#251) #494

sky_line commented

2026-10-06 15:11:40 +00:00

Author

Owner

Related: #494, an investigation into SL Puppetry (LEAP joint streaming), which could carry headset head and hand tracking to other users.

sky_line commented

2026-10-08 02:16:37 +00:00

Author

Owner

Idea (owner): drive the avatar from VR controls through Pose Studio.

Pose Studio (#500) already applies per-bone poses locally and shares them with other Slipstream users through the #Slipstream Helper bridge; IK handles (#530) solve a chain from an end-effector target. VR tracking maps onto that directly:

Targets from tracking: HMD → head/neck look-at target; controllers → hand (front-hoof) IK handles; optional trackers → feet/hips. The IK solver (#530) turns those into bone rotations every frame, with the per-bone limits keeping it natural.
Local view: full frame rate on your own screen (no network involved).
Others see it via pose sync, rate-limited so it doesn't flood the region/sim:
- send only bones that changed beyond a small threshold (quantised quaternions, as the sync format already does);
- cap the update rate (e.g. ~4-10 updates/s, adaptive: lower when many people are syncing, or when the region is busy), and coalesce in between;
- receivers interpolate (slerp) between updates so motion stays smooth at a low rate;
- respect the LSL chat throttles the bridge is subject to (pose lines are region-say from the helper), and back off if lines are dropped;
- only Slipstream users who show synced poses see it; everyone else sees the normal animations.
Consent rules still apply (it's your own avatar, so none needed; posing others stays consent-gated).
Not via uploaded animations: baking poses to .anim and playing them is far too slow for live tracking; pose sync is the realistic path until/unless LL ships server-side puppetry (#494).

**Idea (owner): drive the avatar from VR controls through Pose Studio.** Pose Studio (#500) already applies per-bone poses locally and shares them with other Slipstream users through the #Slipstream Helper bridge; IK handles (#530) solve a chain from an end-effector target. VR tracking maps onto that directly: - **Targets from tracking:** HMD → head/neck look-at target; controllers → hand (front-hoof) IK handles; optional trackers → feet/hips. The IK solver (#530) turns those into bone rotations every frame, with the per-bone limits keeping it natural. - **Local view:** full frame rate on your own screen (no network involved). - **Others see it via pose sync, rate-limited** so it doesn't flood the region/sim: - send only bones that changed beyond a small threshold (quantised quaternions, as the sync format already does); - cap the update rate (e.g. ~4-10 updates/s, adaptive: lower when many people are syncing, or when the region is busy), and coalesce in between; - receivers interpolate (slerp) between updates so motion stays smooth at a low rate; - respect the LSL chat throttles the bridge is subject to (pose lines are region-say from the helper), and back off if lines are dropped; - only Slipstream users who show synced poses see it; everyone else sees the normal animations. - **Consent rules still apply** (it's your own avatar, so none needed; posing others stays consent-gated). - **Not via uploaded animations:** baking poses to .anim and playing them is far too slow for live tracking; pose sync is the realistic path until/unless LL ships server-side puppetry (#494).

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

sky_line/Slipstream#251

No description provided.

Rows
Columns