Picking: can't click sculpts that use back faces for a cel-shade outline #172

New issue

Closed

opened 2026-09-29 18:40:52 +00:00 by sky_line · 1 comment

sky_line commented

2026-09-29 18:40:52 +00:00

Owner

Sculpties built as a cel-shade / toon outline can't be clicked (touch, select, edit). These use a slightly larger, inside-out (inverted-normal / mirrored) shell around the object, so from outside you only ever see its back faces, which draw as the dark outline.

Likely cause (to confirm)

The pick ray test is one-sided. LLOctreeTriangleRayIntersect::visit (llmath/llvolumeoctree.cpp:134) and the volume path (llmath/llvolume.cpp:4421) both call LLTriangleRayIntersect, which rejects triangles facing away from the ray (llvolume.cpp:226-245).
An inside-out shell faces away from the camera everywhere you can see it, so the ray passes straight through its near side. LLTriangleRayIntersectTwoSided exists (llvolume.h:1160) but isn't used for picking.
Compare with the official viewer / Firestorm on the same object to confirm they can click it, and find out how:
- the far wall of the shell faces the camera, so it may be what they hit;
- they may use two-sided hits for sculpts with the INVERT/MIRROR sculpt flags;
- or GL-era render-to-pick behaviour.
Check that the Vulkan path doesn't add its own rejection. Picking under VK runs on the CPU volume octree; the sculpt geometry used for picking must be the displaced sculpt, not the placeholder (see the VKSculpt "no map data yet" deferrals).

Fix direction

Use a two-sided test for picking, or at least for faces whose front side is never visible (inverted sculpts, double-sided glTF). Keep the closest hit. For parity, don't make genuinely invisible back faces of ordinary prims clickable from behind unless LL does too.

Repro

Rez a sculpt with the invert (or mirror) sculpt flag, or a known cel-shade outline item.
Try to click or touch it from outside. Nothing is picked.

Related: #120 (only part of a mesh is clickable).

Sculpties built as a cel-shade / toon outline can't be clicked (touch, select, edit). These use a slightly larger, **inside-out** (inverted-normal / mirrored) shell around the object, so from outside you only ever see its back faces, which draw as the dark outline. ## Likely cause (to confirm) - The pick ray test is **one-sided**. `LLOctreeTriangleRayIntersect::visit` (`llmath/llvolumeoctree.cpp:134`) and the volume path (`llmath/llvolume.cpp:4421`) both call `LLTriangleRayIntersect`, which rejects triangles facing away from the ray (`llvolume.cpp:226-245`). - An inside-out shell faces away from the camera everywhere you can see it, so the ray passes straight through its near side. `LLTriangleRayIntersectTwoSided` exists (`llvolume.h:1160`) but isn't used for picking. - Compare with the official viewer / Firestorm on the same object to confirm they can click it, and find out how: - the far wall of the shell faces the camera, so it may be what they hit; - they may use two-sided hits for sculpts with the INVERT/MIRROR sculpt flags; - or GL-era render-to-pick behaviour. - Check that the Vulkan path doesn't add its own rejection. Picking under VK runs on the CPU volume octree; the sculpt geometry used for picking must be the displaced sculpt, not the placeholder (see the VKSculpt "no map data yet" deferrals). ## Fix direction Use a two-sided test for picking, or at least for faces whose front side is never visible (inverted sculpts, double-sided glTF). Keep the closest hit. For parity, don't make genuinely invisible back faces of ordinary prims clickable from behind unless LL does too. ## Repro 1. Rez a sculpt with the invert (or mirror) sculpt flag, or a known cel-shade outline item. 2. Try to click or touch it from outside. Nothing is picked. Related: #120 (only part of a mesh is clickable).

sky_line commented

2026-09-29 19:01:11 +00:00

Author

Owner

Fixed in ceff65953c (verified in-world): the VK pick broadphase used only the first-captured LOD bounds; pick boxes now union every resident LOD (as the cull does) and refresh when mesh LODs land. The cel-shade outline mesh (sculpt type 5 = mesh) is clickable from outside. Diagnostic RenderVKPickDiag (9183932eff) stays for future pick bugs.

Fixed in ceff65953c (verified in-world): the VK pick broadphase used only the first-captured LOD bounds; pick boxes now union every resident LOD (as the cull does) and refresh when mesh LODs land. The cel-shade outline mesh (sculpt type 5 = mesh) is clickable from outside. Diagnostic RenderVKPickDiag (9183932eff) stays for future pick bugs.

sky_line referenced this issue from a commit

2026-09-29 19:01:11 +00:00

VK diag: RenderVKPickDiag -- why a click misses an object

sky_line referenced this issue from a commit

2026-09-29 19:01:11 +00:00

VK picking: pick boxes cover every resident LOD, not just the first captured

sky_line closed this issue

2026-09-29 19:01:11 +00:00