Investigate SL Puppetry (LEAP-driven real-time avatar animation): status, code, fit with VR (#251) #494

New issue

Open

opened 2026-10-06 15:11:39 +00:00 by sky_line · 0 comments

sky_line commented

2026-10-06 15:11:39 +00:00

Owner

Investigate Second Life Puppetry: https://wiki.secondlife.com/wiki/How_Puppetry_Works

Related to the VR epic #251: head and hand tracking from an OpenXR headset could drive the avatar through the same joint-streaming path.

What it is (from the wiki)

Real-time avatar animation without uploaded animation assets. An external program streams joint targets to the viewer, which solves IK, animates the avatar locally, and relays the joint data through the simulator to other viewers.
Data flow: plug-in → viewer over LEAP (LLSD Event API Plug-in) → IK → local animation → simulator → other viewers.
Viewer side: LLPuppetModule in LL's Puppetry project viewer.
Joint modes:
- inverse_kinematics: avatar-frame position/rotation targets.
- joint_state: parent-frame position, rotation and scale.
Commands:
- Plug-in → viewer: set, get.
- Viewer → plug-in: enable_parts (head/face/hands/fingers mask), set_camera, set_skeleton, stop.
Examples: https://github.com/secondlife/leap (arm_wave.py, webcam_puppetry.py).
Wiki status: experimental, Preview grid (Aditi) only, regions Bunraku, Marionette and Castelet, special Puppetry project viewer. Face and fingers need Bento.

Questions to answer

Is Puppetry still alive server-side? Does any region or grid still run the simulator relay, and did LL shelve the project? Check recent LL statements, the project viewer's last build, and whether those Aditi regions still exist. If the simulator no longer relays it, it's only useful locally (self-view and VR) until LL revives it.
Where is the code? Locate LL's Puppetry viewer branch: LLPuppetModule, the IK solver, the message names and caps used for the simulator relay. Work out how much of it is already in our LL-derived base and how far it has diverged. (LL's code is fine to build on.)
LEAP in our viewer: do we still ship LEAP (LLLeap, --leap)? Is it safe to expose? It runs external processes, so it needs a security review before it goes anywhere near users.
Fit with VR (#251): could OpenXR head and hand poses feed the same IK / joint_state path, so other users see your real head and hand movement? Compare that with a purely local VR rig.
Fit with our animation pipeline: IK and local joint overrides vs our GPU skinning and palette generation. Where would puppetry poses enter, and what's the cost per avatar for remote puppeted avatars?
Webcam tracking: worth it (MediaPipe-style, run locally), or leave it to external LEAP plug-ins?

Deliverable

A short write-up covering server status, the code location and its divergence from our base, an integration sketch (local-only first, then networked if the server supports it), the security notes on LEAP, and a go/no-go with an estimate. No implementation in this ticket.

Investigate Second Life **Puppetry**: https://wiki.secondlife.com/wiki/How_Puppetry_Works Related to the VR epic #251: head and hand tracking from an OpenXR headset could drive the avatar through the same joint-streaming path. ## What it is (from the wiki) - Real-time avatar animation **without uploaded animation assets**. An external program streams joint targets to the viewer, which solves IK, animates the avatar locally, and relays the joint data through the simulator to other viewers. - **Data flow:** plug-in → viewer over **LEAP** (LLSD Event API Plug-in) → IK → local animation → simulator → other viewers. - **Viewer side:** `LLPuppetModule` in LL's Puppetry project viewer. - **Joint modes:** - `inverse_kinematics`: avatar-frame position/rotation targets. - `joint_state`: parent-frame position, rotation and scale. - **Commands:** - Plug-in → viewer: `set`, `get`. - Viewer → plug-in: `enable_parts` (head/face/hands/fingers mask), `set_camera`, `set_skeleton`, `stop`. - **Examples:** https://github.com/secondlife/leap (`arm_wave.py`, `webcam_puppetry.py`). - **Wiki status:** experimental, **Preview grid (Aditi) only**, regions Bunraku, Marionette and Castelet, special Puppetry project viewer. Face and fingers need Bento. ## Questions to answer 1. **Is Puppetry still alive server-side?** Does any region or grid still run the simulator relay, and did LL shelve the project? Check recent LL statements, the project viewer's last build, and whether those Aditi regions still exist. If the simulator no longer relays it, it's only useful locally (self-view and VR) until LL revives it. 2. **Where is the code?** Locate LL's Puppetry viewer branch: `LLPuppetModule`, the IK solver, the message names and caps used for the simulator relay. Work out how much of it is already in our LL-derived base and how far it has diverged. (LL's code is fine to build on.) 3. **LEAP in our viewer:** do we still ship LEAP (`LLLeap`, `--leap`)? Is it safe to expose? It runs external processes, so it needs a security review before it goes anywhere near users. 4. **Fit with VR (#251):** could OpenXR head and hand poses feed the same IK / `joint_state` path, so other users see your real head and hand movement? Compare that with a purely local VR rig. 5. **Fit with our animation pipeline:** IK and local joint overrides vs our GPU skinning and palette generation. Where would puppetry poses enter, and what's the cost per avatar for remote puppeted avatars? 6. **Webcam tracking:** worth it (MediaPipe-style, run locally), or leave it to external LEAP plug-ins? ## Deliverable A short write-up covering server status, the code location and its divergence from our base, an integration sketch (local-only first, then networked if the server supports it), the security notes on LEAP, and a go/no-go with an estimate. No implementation in this ticket.

sky_line added the

labels

2026-10-06 15:11:39 +00:00

sky_line added this to the Slipstream project

2026-10-06 15:11:39 +00:00

sky_line referenced this issue

2026-10-06 15:11:40 +00:00

[Epic] VR support (OpenXR) — stretch goal, 2.0.0+ #251