Reflection probes: compressed formats for higher resolution -- Part 1 R11G11B10 (2x), Part 2 BC6H with GPU encode (8x)

sky_line commented

2026-10-08 19:23:54 +00:00

Owner

Reflection probes are RGBA16F cube arrays (kProbeHdr, llcleanrenderervk.cpp ~8966): 8 B/px with full mip chains.

At 128 px × 256 probes the radiance array is ~270 MB (the "probes ~284 MB" VKMem line).
At 256 px it would be ~1.1 GB, and at 512 px ~4.3 GB.

That memory cost is what holds probe resolution back. Do this in two parts and measure each before moving on.

Shared prerequisite: the probe alpha channel carries distance statistics for the Chebyshev visibility test (#88/#89, the P1 dome-leak fix). Neither target format has alpha, so move it to a separate small R16G16 moments texture, which may be lower resolution than radiance. Verify the leak fix still holds.

Part 1: R11G11B10 float radiance (2× smaller, no encoder)

Switch the radiance (and irradiance, if it fits) cube arrays to VK_FORMAT_B10G11R11_UFLOAT_PACK32. Check the format supports storage/attachment and filtering on the target GPUs, and the capture/filter compute paths.
HDR range: no sign bit, ~5–6 bit mantissa. Check for banding on smooth sky gradients and on very bright sources (sun disk clamp).
Measure: VRAM before and after, visual A/B (mirrors, glossy floors, sky), and capture cost.
Expected: probe memory halves, which allows the next resolution step at today's cost.

Part 2: BC6H-compressed probes (8× smaller)

Add a GPU compute BC6H encoder (unsigned float, fast mode) that runs after a probe face is captured and filtered, for every mip.
Encode cost to measure: roughly 0.05–0.1 ms per 128² face expected. Reuse or extend the existing GPU texture-compression path (texcomp) where possible.
Keep real-time updated faces (frequently updated / near-camera probe 0) uncompressed or in the Part 1 format. Compress the probes that rarely update, using a per-probe or per-face policy based on update frequency.
Check for block artifacts on mirror-like (roughness ~0) surfaces and smooth gradients. Probe 0 can stay uncompressed for sharp mirrors.
Measure:
- VRAM, which should be about 135 MB at 256 px × 256 probes;
- encode GPU time per frame under real-time probe updates;
- visual A/B.
Goal: allow 256 or 512 px probes at or below today's memory.

Done when: both parts have been measured, the result is recorded here, and the chosen defaults are applied (possibly a resolution bump).

Reflection probes are RGBA16F cube arrays (`kProbeHdr`, `llcleanrenderervk.cpp` ~8966): 8 B/px with full mip chains. - At 128 px × 256 probes the radiance array is ~270 MB (the "probes ~284 MB" VKMem line). - At 256 px it would be ~1.1 GB, and at 512 px ~4.3 GB. That memory cost is what holds probe resolution back. Do this in two parts and measure each before moving on. **Shared prerequisite:** the probe **alpha channel carries distance statistics** for the Chebyshev visibility test (#88/#89, the P1 dome-leak fix). Neither target format has alpha, so move it to a separate small R16G16 moments texture, which may be lower resolution than radiance. Verify the leak fix still holds. ## Part 1: R11G11B10 float radiance (2× smaller, no encoder) - Switch the radiance (and irradiance, if it fits) cube arrays to `VK_FORMAT_B10G11R11_UFLOAT_PACK32`. Check the format supports storage/attachment and filtering on the target GPUs, and the capture/filter compute paths. - HDR range: no sign bit, ~5–6 bit mantissa. Check for banding on smooth sky gradients and on very bright sources (sun disk clamp). - **Measure:** VRAM before and after, visual A/B (mirrors, glossy floors, sky), and capture cost. - **Expected:** probe memory halves, which allows the next resolution step at today's cost. ## Part 2: BC6H-compressed probes (8× smaller) - Add a GPU compute BC6H encoder (unsigned float, fast mode) that runs after a probe face is captured and filtered, for every mip. - Encode cost to measure: roughly 0.05–0.1 ms per 128² face expected. Reuse or extend the existing GPU texture-compression path (texcomp) where possible. - Keep **real-time updated faces** (frequently updated / near-camera probe 0) uncompressed or in the Part 1 format. Compress the probes that rarely update, using a per-probe or per-face policy based on update frequency. - Check for block artifacts on mirror-like (roughness ~0) surfaces and smooth gradients. Probe 0 can stay uncompressed for sharp mirrors. - **Measure:** - VRAM, which should be about 135 MB at 256 px × 256 probes; - encode GPU time per frame under real-time probe updates; - visual A/B. - **Goal:** allow 256 or 512 px probes at or below today's memory. **Done when:** both parts have been measured, the result is recorded here, and the chosen defaults are applied (possibly a resolution bump).

sky_line added the

labels

2026-10-08 19:23:54 +00:00

sky_line added this to the Slipstream project

2026-10-08 19:23:54 +00:00

sky_line commented

2026-10-10 11:37:32 +00:00

Author

Owner

Plan update 2026-10-10: one selectable format setting, all options built together for a side-by-side test

Probes are still RGBA16F today (kProbeHdr, llcleanrenderervk.cpp:11361): nothing has been quantized yet.

New setting RenderVKProbeFormat (live, or applied on probe re-allocation):

0 = RGBA16F, today's format (8 B/px).
1 = RGB9E5 (E5B9G9R9_UFLOAT_PACK32, 4 B/px). 9-bit mantissas with a shared exponent, closest to RGBA16F at half the size.
- Not renderable or storable on most GPUs, so the compute filter writes the packed bits through an R32_UINT alias view and samples as RGB9E5.
- Weak spot: strongly saturated colours, because the exponent is shared across channels.
2 = R11G11B10 (B10G11R11_UFLOAT_PACK32, 4 B/px). 6/6/5-bit mantissas, no cross-channel coupling.
- Weak spot: banding on smooth gradients.
3 = BC6H (unsigned float, 1 B/px) via a GPU compute encoder after capture and filter, for every mip.
- Probes that update in real time / probe 0 stay in format 1 (or 0); rarely-updated probes get compressed.
- Weak spots: 4x4 block artifacts on mirror-like surfaces and in gradients; encode cost.

Prerequisite for 1-3: the probe alpha (Chebyshev distance moments for the #88/#89 dome-leak fix) moves to its own R16G16 texture. The leak fix must still hold.

Memory at 256 probes:

	128 px	256 px	512 px
RGBA16F	~270 MB	~1.1 GB	~4.3 GB
RGB9E5 / R11G11B10	~135 MB	~540 MB	~2.2 GB
BC6H	~34 MB	~135 MB	~540 MB

All four go in at once for an in-world A/B/C/D, with a failure-mode test plan. The default stays 0 until the owner picks from the comparison.

## Plan update 2026-10-10: one selectable format setting, all options built together for a side-by-side test Probes are still **RGBA16F** today (`kProbeHdr`, llcleanrenderervk.cpp:11361): nothing has been quantized yet. **New setting `RenderVKProbeFormat`** (live, or applied on probe re-allocation): - **0 = RGBA16F**, today's format (8 B/px). - **1 = RGB9E5** (`E5B9G9R9_UFLOAT_PACK32`, 4 B/px). 9-bit mantissas with a shared exponent, closest to RGBA16F at half the size. - Not renderable or storable on most GPUs, so the compute filter writes the packed bits through an R32_UINT alias view and samples as RGB9E5. - Weak spot: strongly saturated colours, because the exponent is shared across channels. - **2 = R11G11B10** (`B10G11R11_UFLOAT_PACK32`, 4 B/px). 6/6/5-bit mantissas, no cross-channel coupling. - Weak spot: banding on smooth gradients. - **3 = BC6H** (unsigned float, 1 B/px) via a GPU compute encoder after capture and filter, for every mip. - Probes that update in real time / probe 0 stay in format 1 (or 0); rarely-updated probes get compressed. - Weak spots: 4x4 block artifacts on mirror-like surfaces and in gradients; encode cost. **Prerequisite for 1-3:** the probe alpha (Chebyshev distance moments for the #88/#89 dome-leak fix) moves to its own R16G16 texture. The leak fix must still hold. **Memory at 256 probes:** | | 128 px | 256 px | 512 px | |---|---|---|---| | RGBA16F | ~270 MB | ~1.1 GB | ~4.3 GB | | RGB9E5 / R11G11B10 | ~135 MB | ~540 MB | ~2.2 GB | | BC6H | ~34 MB | ~135 MB | ~540 MB | All four go in at once for an in-world A/B/C/D, with a failure-mode test plan. The default stays 0 until the owner picks from the comparison.

sky_line commented

2026-10-10 11:37:33 +00:00

Author

Owner

Started 2026-10-10: all four probe formats are being built as one selectable setting for a side-by-side test (plan in the previous comment).

sky_line added

and removed

labels

2026-10-10 11:37:33 +00:00

sky_line commented

2026-10-10 12:18:41 +00:00

Author

Owner

Correction 2026-10-10: there is no distance-moments alpha in the Vulkan probes

The plan comment's "prerequisite" was wrong. In the VK path:

clean_probe_filter.comp stores vec4(col, 1.0).
Every uProbeRadiance / uProbeIrradiance read is .rgb.
There is no Chebyshev visibility code in indra/vkclean/shaders.

The P1 dome-leak fix (alpha distance statistics, #88/#89) was GL-only and went away with the GL removal (0b3a52696a). So no moments texture was needed, and formats 1-3 lose nothing.

Side effect: the Chebyshev dome-leak protection does not exist under Vulkan. The test plan's dome/house check is now a check of whether interior reflections leak outside sky at all, in every format. If they do, that is a separate (pre-existing) bug to file.

Implementation status: all four formats are merged (RenderVKProbeFormat 0-3, with debug view 20 showing which probes read the uncompressed live cube). Measured memory and the test plan are in the agent's report.

## Correction 2026-10-10: there is no distance-moments alpha in the Vulkan probes The plan comment's "prerequisite" was wrong. In the VK path: - `clean_probe_filter.comp` stores `vec4(col, 1.0)`. - Every `uProbeRadiance` / `uProbeIrradiance` read is `.rgb`. - There is no Chebyshev visibility code in `indra/vkclean/shaders`. The P1 dome-leak fix (alpha distance statistics, #88/#89) was GL-only and went away with the GL removal (0b3a52696a). So no moments texture was needed, and formats 1-3 lose nothing. **Side effect:** the Chebyshev dome-leak protection does not exist under Vulkan. The test plan's dome/house check is now a check of whether interior reflections leak outside sky at all, in every format. If they do, that is a separate (pre-existing) bug to file. Implementation status: all four formats are merged (`RenderVKProbeFormat` 0-3, with debug view 20 showing which probes read the uncompressed live cube). Measured memory and the test plan are in the agent's report.