sky_line/Slipstream

Fork 1

[Epic] Cluster LOD (meshlets + DAG, Nanite-style) for avatars, extensible to dense static mesh #642

New issue

Open

opened 2026-10-10 12:52:36 +00:00 by sky_line · 3 comments

sky_line commented

2026-10-10 12:52:36 +00:00

Owner

Exploration / epic. Cluster-based continuous LOD (meshlets + a cluster DAG, Nanite-style) for avatars first, designed so it can extend to dense static mesh later.

Why

Rigged avatars dominate crowd cost:

GPU work that scales with vertices: pre-skin, weld, BLAS refit, the G-buffer/shadow/probe draws.
Today's controls are coarse: a rigged LOD controller with whole-mesh decimated steps (#561, #612 un-subdivide), choosing one LOD per mesh per avatar.
The problems with that:
- A mesh's LOD step pops, and is wrong in parts: the face of a distant avatar wants less detail than the hands of a near one.
- Body and clothing at different whole-mesh LODs cause poke-through (#611).
- Creators ship LODs of wildly varying quality, or none at all.

Cluster LOD gives per-cluster, screen-error-driven detail with crack-free transitions. Vertex work then follows what's visible on screen, not what the creator uploaded.

Concept

Clustering at ingest (offline per mesh asset, cached on disk like #274's GPU-ready textures):
- Split each mesh LOD0 into meshlets (~64-128 triangles).
- Build a DAG by repeatedly grouping neighbouring clusters, simplifying each group with its outer boundary locked, and re-splitting. This gives a hierarchy where any "cut" through the DAG is watertight.
- Store per-cluster bounds and a simplification error.
Runtime cut selection (GPU compute): per cluster, project its error to screen pixels and choose the cut where error < ~1 px. Cull clusters (frustum, Hi-Z occlusion, backface cone). Emit the visible clusters into indirect draws, or mesh-shader / task-shader dispatch where supported.
Avatar-specific parts (the hard part):
- Skinning per cluster: skin only selected clusters' vertices, which ties into pre-skin (#632) and the avatar-local space work.
- Deforming bounds: cluster bounds and errors must hold under animation. Conservative per-cluster bounds expanded by joint-motion bounds, or bounds recomputed from skinned cluster corners.
- Skin weights must survive simplification (weight-aware simplify; #618 has related upload-side work).
- Seams between attachments: the rest-pose weld pairs (#641) must map onto whatever cluster cut is chosen. Lock weld-boundary vertices in the DAG, or remap pairs per cut.
- Body/clothing poke-through: choose cuts so layered clothing isn't simplified below the body under it (#611). Bias error per layer or lock layer-contact regions.
- Alpha/mask materials, bake textures, and per-face materials stay intact: clusters never cross material boundaries.
Ray tracing: BLAS per selected cut, or NVIDIA cluster acceleration structures (CLAS/partitioned TLAS, see #296 and #581) where available, so RT shadows match the raster LOD. Refit cost then follows visible detail.
Static dense mesh (phase 2): the same pipeline without skinning: dense builds and distant multi-sim views. Pairs naturally with the visibility buffer (#256), which removes the 2x2-quad cost of the tiny triangles that cluster LOD still leaves.

Open questions / risks

Ingest cost and cache size per mesh (clustering + DAG simplification is CPU-heavy). Needs worker threads and an on-disk cache keyed by mesh asset id + LOD.
Interaction with the existing per-mesh rigged LOD controller and VRAM ladder (#561, #613): replace them, or keep them as a fallback for non-capable GPUs?
Hardware tiers: mesh shaders are not universal. A compute-cull + indirect-draw path must exist (#98).
Animation-correct bounds vs over-conservative culling.
Validation corpus: popular mesh bodies/heads (incl. Bento), layered clothing, rigged hair, animesh.

Measure first

In a crowded venue, record rigged vertex counts per frame, plus c_preskin / c_rigweld / c_blas_refit / g_gbuf_rigged, against the estimated vertex count a 1-px-error cut would select. Proceed if the cut is well below today's submitted vertices.

Related: #561, #612, #613, #611, #618, #632, #641, #256, #296, #581, #98, #274.

**Exploration / epic.** Cluster-based continuous LOD (meshlets + a cluster DAG, Nanite-style) for **avatars first**, designed so it can extend to **dense static mesh** later. ## Why Rigged avatars dominate crowd cost: - **GPU work** that scales with vertices: pre-skin, weld, BLAS refit, the G-buffer/shadow/probe draws. - **Today's controls are coarse:** a rigged LOD controller with whole-mesh decimated steps (#561, #612 un-subdivide), choosing one LOD per mesh per avatar. - **The problems with that:** - A mesh's LOD step pops, and is wrong in parts: the face of a distant avatar wants less detail than the hands of a near one. - Body and clothing at different whole-mesh LODs cause poke-through (#611). - Creators ship LODs of wildly varying quality, or none at all. Cluster LOD gives **per-cluster, screen-error-driven detail** with **crack-free transitions**. Vertex work then follows what's visible on screen, not what the creator uploaded. ## Concept 1. **Clustering at ingest** (offline per mesh asset, cached on disk like #274's GPU-ready textures): - Split each mesh LOD0 into meshlets (~64-128 triangles). - Build a DAG by repeatedly grouping neighbouring clusters, simplifying each group with its outer boundary locked, and re-splitting. This gives a hierarchy where any "cut" through the DAG is watertight. - Store per-cluster bounds and a simplification error. 2. **Runtime cut selection (GPU compute):** per cluster, project its error to screen pixels and choose the cut where error < ~1 px. Cull clusters (frustum, Hi-Z occlusion, backface cone). Emit the visible clusters into indirect draws, or mesh-shader / task-shader dispatch where supported. 3. **Avatar-specific parts (the hard part):** - **Skinning per cluster:** skin only selected clusters' vertices, which ties into pre-skin (#632) and the avatar-local space work. - **Deforming bounds:** cluster bounds and errors must hold under animation. Conservative per-cluster bounds expanded by joint-motion bounds, or bounds recomputed from skinned cluster corners. - **Skin weights** must survive simplification (weight-aware simplify; #618 has related upload-side work). - **Seams between attachments:** the rest-pose weld pairs (#641) must map onto whatever cluster cut is chosen. Lock weld-boundary vertices in the DAG, or remap pairs per cut. - **Body/clothing poke-through:** choose cuts so layered clothing isn't simplified below the body under it (#611). Bias error per layer or lock layer-contact regions. - **Alpha/mask materials, bake textures, and per-face materials** stay intact: clusters never cross material boundaries. 4. **Ray tracing:** BLAS per selected cut, or NVIDIA cluster acceleration structures (CLAS/partitioned TLAS, see #296 and #581) where available, so RT shadows match the raster LOD. Refit cost then follows visible detail. 5. **Static dense mesh (phase 2):** the same pipeline without skinning: dense builds and distant multi-sim views. Pairs naturally with the visibility buffer (#256), which removes the 2x2-quad cost of the tiny triangles that cluster LOD still leaves. ## Open questions / risks - Ingest cost and cache size per mesh (clustering + DAG simplification is CPU-heavy). Needs worker threads and an on-disk cache keyed by mesh asset id + LOD. - Interaction with the existing per-mesh rigged LOD controller and VRAM ladder (#561, #613): replace them, or keep them as a fallback for non-capable GPUs? - Hardware tiers: mesh shaders are not universal. A compute-cull + indirect-draw path must exist (#98). - Animation-correct bounds vs over-conservative culling. - Validation corpus: popular mesh bodies/heads (incl. Bento), layered clothing, rigged hair, animesh. ## Measure first In a crowded venue, record rigged vertex counts per frame, plus `c_preskin` / `c_rigweld` / `c_blas_refit` / `g_gbuf_rigged`, against the estimated vertex count a 1-px-error cut would select. Proceed if the cut is well below today's submitted vertices. Related: #561, #612, #613, #611, #618, #632, #641, #256, #296, #581, #98, #274.

sky_line added the

labels

2026-10-10 12:52:36 +00:00

sky_line added this to the Slipstream project

2026-10-10 12:52:36 +00:00

sky_line commented

2026-10-13 01:48:28 +00:00

Author

Owner

Design decision: always on; pressure only moves the tolerance

Always active. The DAG cut is how the mesh is drawn, not a pressure response. At ~1 px of error a cut looks the same as full detail at that distance, and drawing more is pure waste (tiny-triangle quad overdraw, plus skinning, BLAS and shadow work that changes nothing on screen). Most of the crowd win comes in the comfortable case: distant avatars drop to a few percent of their vertices with no visible change.

Pressure is one continuous knob: the pixel tolerance. Normal ~1 px; under GPU-time or VRAM pressure 2 -> 4 -> 8 px. It degrades smoothly and everywhere at once.

This replaces the current ladder (#561 / #613): decimated steps, pixel-tolerance levels and creator-LOD pressure. Creator LODs are never used. The old controller stays only as the fallback for GPUs that can't run the cluster path.
Important avatars get a lower tolerance (self and the nearest at ~0.5 px) instead of an exemption list (today: 10 avatars / 3 m / 100 px). This removes cliff-edge boundaries like the ~20 m drop reported in #741.

"1 px" is not zero error. Things to handle:

Shading error: the metric is positional, but normals, UVs and skin weights also change. The simplifier must weigh attributes, and close-up surfaces may need a tighter tolerance than the positional one.
Animation: the error is measured in rest pose; joint stretch can enlarge it. Scale each cluster's error by a bound on its joints' stretch.
Temporal stability: no blending between levels; the sub-pixel threshold hides switches, but under TAA/FSR small cut changes can still shimmer. Add hysteresis on cut changes.
Ray tracing: a cut change means a BLAS rebuild, not a refit. Give the RT cut its own coarser tolerance with hysteresis, or use cluster acceleration structures where available (#296 / #581).
Per-view error: main view, shadow cascades and reflection probe faces each project the error in their own pixels. A 128 px probe face needs far less detail than the main view.
Snapshots / Photo Studio: compute the error at the snapshot resolution, so high-resolution photos get full detail.

## Design decision: always on; pressure only moves the tolerance **Always active.** The DAG cut is how the mesh is drawn, not a pressure response. At ~1 px of error a cut looks the same as full detail at that distance, and drawing more is pure waste (tiny-triangle quad overdraw, plus skinning, BLAS and shadow work that changes nothing on screen). Most of the crowd win comes in the comfortable case: distant avatars drop to a few percent of their vertices with no visible change. **Pressure is one continuous knob: the pixel tolerance.** Normal ~1 px; under GPU-time or VRAM pressure 2 -> 4 -> 8 px. It degrades smoothly and everywhere at once. - This **replaces** the current ladder (#561 / #613): decimated steps, pixel-tolerance levels and creator-LOD pressure. **Creator LODs are never used.** The old controller stays only as the fallback for GPUs that can't run the cluster path. - Important avatars get a **lower tolerance** (self and the nearest at ~0.5 px) instead of an exemption list (today: 10 avatars / 3 m / 100 px). This removes cliff-edge boundaries like the ~20 m drop reported in #741. **"1 px" is not zero error. Things to handle:** - **Shading error:** the metric is positional, but normals, UVs and skin weights also change. The simplifier must weigh attributes, and close-up surfaces may need a tighter tolerance than the positional one. - **Animation:** the error is measured in rest pose; joint stretch can enlarge it. Scale each cluster's error by a bound on its joints' stretch. - **Temporal stability:** no blending between levels; the sub-pixel threshold hides switches, but under TAA/FSR small cut changes can still shimmer. Add hysteresis on cut changes. - **Ray tracing:** a cut change means a BLAS rebuild, not a refit. Give the RT cut its own coarser tolerance with hysteresis, or use cluster acceleration structures where available (#296 / #581). - **Per-view error:** main view, shadow cascades and reflection probe faces each project the error in their own pixels. A 128 px probe face needs far less detail than the main view. - **Snapshots / Photo Studio:** compute the error at the snapshot resolution, so high-resolution photos get full detail.

sky_line commented

2026-10-13 01:57:39 +00:00

Author

Owner

Streaming, older hardware, and the rollout rules

Getting clusters onto the GPU without costing FPS

The frame never waits for geometry: if a finer page isn't resident yet, the cut uses its coarser parent for a frame or two.

Build once, off the frame: clustering + DAG build on worker threads at first ingest, cached on disk (like #274). Roughly tens of ms per heavy mesh, once per asset.
Pages with a resident root: each mesh's DAG is stored as fixed-size pages, coarse to fine. The coarsest pages are always resident (tiny, watertight). Finer pages live in a fixed-size VRAM page pool sized from the memory budget, with LRU eviction.
The GPU asks for what it needs: the selection shader writes missing finer pages to a request buffer, read back a frame or two later without waiting (like the GPU pick readback).
Budgeted uploads off the graphics queue: async transfer queue, per-frame byte budget (e.g. 16-32 MB/frame), fence-gated residency. Page-in/out only updates a page table; no per-cluster CPU work.
Less RAM: no full CPU mirror of geometry; pages come from the disk cache, RAM holds only in-flight uploads.
Skinning follows the cut: pre-skin walks the selected-cluster list, so skinning cost scales with what's on screen.

Expected fixed cost: ~0.2-0.5 ms/frame for selection + culling over tens of thousands of clusters. A small loss in an empty scene, a large win in a crowd.

Older cards (RTX 20 / RX 6000)

On balance a help, most of all on older cards (vertex-, tiny-triangle- and VRAM-limited).

The floor path is compute selection + multi-draw indirect, which runs everywhere we run; no mesh shaders required.
The page pool is sized to the card, which is safer on 6-8 GB cards than full-detail residency or creator-LOD fallback.
Risk: first-generation RT (RTX 20, RX 6000). BLAS rebuilds on cut changes are expensive. Needs a coarser RT tolerance + hysteresis, or cluster acceleration structures where they exist.
Risk: fixed selection cost in light scenes; first-visit build cost at a crowded arrival on a slow CPU.

Measurements to take

Selection-pass cost in an empty scene vs a crowd, per tier.
Page upload bandwidth and pop-in during a teleport into a busy event.
RTX 20 / RX 6000 class with RT on: BLAS rebuild time per frame as cuts change, vs the old controller.
VRAM: page pool + resident roots vs today's rigged geometry, same scene.
On the 5090: every forced fallback tier vs the native tier, same scene.

Rollout rules (owner decision)

Toggle, not replacement (for now): RenderVKClusterLOD, default off. The current rigged LOD controller (#561 / #613) stays fully functional with the toggle off. If RT on first-generation cards can't be made decent with clusters, the old controller remains the path that gives those users decent RT in crowds.
Newest features first: each feature uses the best path the running GPU supports (target: RTX 5090 -- mesh shaders, NV cluster + partitioned acceleration structures, device-generated commands, memory decompression, indirect copies, 64-bit atomics), falling back to slower paths only when an extension is missing. Tiers are chosen by extension presence at runtime, never by GPU name.
Every fallback is testable on the newest card: a debug setting forces each lower tier.

Work groups

Sub-tickets below; wave 0 lands the shared contract, waves 1 and 2 build on it.

## Streaming, older hardware, and the rollout rules ### Getting clusters onto the GPU without costing FPS The frame never waits for geometry: if a finer page isn't resident yet, the cut uses its coarser parent for a frame or two. 1. **Build once, off the frame:** clustering + DAG build on worker threads at first ingest, cached on disk (like #274). Roughly tens of ms per heavy mesh, once per asset. 2. **Pages with a resident root:** each mesh's DAG is stored as fixed-size pages, coarse to fine. The coarsest pages are always resident (tiny, watertight). Finer pages live in a fixed-size VRAM **page pool** sized from the memory budget, with LRU eviction. 3. **The GPU asks for what it needs:** the selection shader writes missing finer pages to a request buffer, read back a frame or two later without waiting (like the GPU pick readback). 4. **Budgeted uploads off the graphics queue:** async transfer queue, per-frame byte budget (e.g. 16-32 MB/frame), fence-gated residency. Page-in/out only updates a page table; no per-cluster CPU work. 5. **Less RAM:** no full CPU mirror of geometry; pages come from the disk cache, RAM holds only in-flight uploads. 6. **Skinning follows the cut:** pre-skin walks the selected-cluster list, so skinning cost scales with what's on screen. Expected fixed cost: ~0.2-0.5 ms/frame for selection + culling over tens of thousands of clusters. A small loss in an empty scene, a large win in a crowd. ### Older cards (RTX 20 / RX 6000) On balance a help, most of all on older cards (vertex-, tiny-triangle- and VRAM-limited). - The floor path is compute selection + multi-draw indirect, which runs everywhere we run; no mesh shaders required. - The page pool is sized to the card, which is safer on 6-8 GB cards than full-detail residency or creator-LOD fallback. - **Risk: first-generation RT** (RTX 20, RX 6000). BLAS rebuilds on cut changes are expensive. Needs a coarser RT tolerance + hysteresis, or cluster acceleration structures where they exist. - **Risk:** fixed selection cost in light scenes; first-visit build cost at a crowded arrival on a slow CPU. ### Measurements to take - Selection-pass cost in an empty scene vs a crowd, per tier. - Page upload bandwidth and pop-in during a teleport into a busy event. - RTX 20 / RX 6000 class with RT on: BLAS rebuild time per frame as cuts change, vs the old controller. - VRAM: page pool + resident roots vs today's rigged geometry, same scene. - On the 5090: every forced fallback tier vs the native tier, same scene. ### Rollout rules (owner decision) - **Toggle, not replacement (for now):** `RenderVKClusterLOD`, default off. The current rigged LOD controller (#561 / #613) stays fully functional with the toggle off. If RT on first-generation cards can't be made decent with clusters, the old controller remains the path that gives those users decent RT in crowds. - **Newest features first:** each feature uses the best path the running GPU supports (target: RTX 5090 -- mesh shaders, NV cluster + partitioned acceleration structures, device-generated commands, memory decompression, indirect copies, 64-bit atomics), falling back to slower paths only when an extension is missing. Tiers are chosen by extension presence at runtime, never by GPU name. - **Every fallback is testable on the newest card:** a debug setting forces each lower tier. ### Work groups Sub-tickets below; wave 0 lands the shared contract, waves 1 and 2 build on it.

sky_line referenced this issue

2026-10-13 01:57:39 +00:00

Cluster LOD 0: shared contract -- data format, tiers, settings, module skeleton #743

sky_line referenced this issue

2026-10-13 01:57:39 +00:00

Cluster LOD 1a: ingest -- clusterize + DAG build, attribute/weight-aware, disk cache #744

sky_line referenced this issue

2026-10-13 01:57:40 +00:00

Cluster LOD 1b: GPU cut selection, culling and draw emission (MDI floor, mesh-shader tier) #745

sky_line referenced this issue

2026-10-13 01:57:40 +00:00

Cluster LOD 1c: streaming and residency -- page pool, page table, budgeted async uploads #746

sky_line referenced this issue

2026-10-13 01:57:40 +00:00

Cluster LOD 2a: skinning and avatar deformation over selected clusters #747

sky_line referenced this issue

2026-10-13 01:57:40 +00:00

Cluster LOD 2b: ray tracing for cluster cuts -- CLAS / partitioned TLAS, per-cut BLAS fallback #748

sky_line referenced this issue

2026-10-13 01:57:40 +00:00

Cluster LOD 2c: tolerance controller, per-view error, protection and diagnostics #749

sky_line commented

2026-10-13 01:57:40 +00:00

Author

Owner

Sub-tickets

#743 Cluster LOD 0: shared contract -- data format, tiers, settings, module skeleton
#744 Cluster LOD 1a: ingest -- clusterize + DAG build, attribute/weight-aware, disk cache
#745 Cluster LOD 1b: GPU cut selection, culling and draw emission (MDI floor, mesh-shader tier)
#746 Cluster LOD 1c: streaming and residency -- page pool, page table, budgeted async uploads
#747 Cluster LOD 2a: skinning and avatar deformation over selected clusters
#748 Cluster LOD 2b: ray tracing for cluster cuts -- CLAS / partitioned TLAS, per-cut BLAS fallback
#749 Cluster LOD 2c: tolerance controller, per-view error, protection and diagnostics

### Sub-tickets - [ ] #743 Cluster LOD 0: shared contract -- data format, tiers, settings, module skeleton - [ ] #744 Cluster LOD 1a: ingest -- clusterize + DAG build, attribute/weight-aware, disk cache - [ ] #745 Cluster LOD 1b: GPU cut selection, culling and draw emission (MDI floor, mesh-shader tier) - [ ] #746 Cluster LOD 1c: streaming and residency -- page pool, page table, budgeted async uploads - [ ] #747 Cluster LOD 2a: skinning and avatar deformation over selected clusters - [ ] #748 Cluster LOD 2b: ray tracing for cluster cuts -- CLAS / partitioned TLAS, per-cut BLAS fallback - [ ] #749 Cluster LOD 2c: tolerance controller, per-view error, protection and diagnostics

sky_line referenced this issue from a commit

2026-10-14 05:50:49 +00:00

#743 cluster LOD wave 0: shared contract, tiers, settings, module skeleton

sky_line added the

status

up-next

label

2026-10-18 22:32:44 +00:00

sky_line referenced this issue

2026-10-24 01:31:07 +00:00

[rendering] Cluster LOD perf: close the gap to legacy in avatar-heavy scenes (RT cut, shadow rig cull, pre-cluster pass, raster) #1008