Explore: Steam Audio (ray-traced occlusion, reverb, pathing, HRTF) for in-world sounds

sky_line commented

2026-10-11 12:24:33 +00:00

Owner

Explore: physically based / ray-traced audio for in-world sounds using Steam Audio (Valve). It is free and open source under Apache 2.0 (since 2024), which is compatible with our LGPL viewer. Prompted by Rye's note that modern games use RT audio.

What it would give us

Occlusion / obstruction / transmission: a wall muffles a sound (frequency-dependent), it doesn't just lower the volume; a half-open door lets more through.
Reflections + reverb ("room resonance"): reverb estimated from the real room shape and materials. A tiled bathroom rings, a soft-furnished club sounds dead, caves echo. Early reflections help localisation.
Pathing / diffraction: sound bends around corners and through doorways.
HRTF spatialisation: proper 3D placement (height, front/back) on headphones.

Questions to answer

Integration point: our audio stack is OpenAL (LLAudioEngine_OpenAL). Do we use Steam Audio's effects (binaural, direct, reflection, path) on OpenAL sources via an effect/mixing stage, or move the mix to our own mixer with Steam Audio applied per source? What is the latency and CPU cost per source?
Geometry: Steam Audio wants a scene (its own BVH, or Embree / Radeon Rays). Options:
- feed it simplified region geometry (terrain + prims, physics shapes or low LOD)
- use its custom ray-tracer callback to trace against our existing Vulkan RT TLAS (GPU) instead of a second copy of the world
- per-region static scenes plus dynamic instanced meshes for moving objects
SL geometry changes constantly. Measure scene build/update cost.
Materials: SL surfaces have no acoustic properties. Use one default material, a heuristic from texture/material names or PBR properties (metallic/rough → reflective), or a per-parcel / per-object override (a viewer-side setting or script-readable hint).
Scale: busy venues have many simultaneous sounds plus streaming music. Which sources get simulated (nearest N, priority by volume)? Parcel audio streams and voice stay dry by default; consider applying reverb to voice optionally.
Fallbacks: CPU path without RT hardware, a quality tiers setting, and off by default until proven.
Platforms: Linux / Windows / macOS builds of the Steam Audio SDK (prebuilt binaries vs source build), and their licence notices.

Deliverable

A prototype behind a setting: occlusion + reverb for in-world sounds in one test region, with measured CPU/GPU cost, then a go/no-go and an implementation plan.

**Explore:** physically based / ray-traced audio for in-world sounds using **Steam Audio** (Valve). It is free and open source under **Apache 2.0** (since 2024), which is compatible with our LGPL viewer. Prompted by Rye's note that modern games use RT audio. ## What it would give us - **Occlusion / obstruction / transmission:** a wall muffles a sound (frequency-dependent), it doesn't just lower the volume; a half-open door lets more through. - **Reflections + reverb ("room resonance"):** reverb estimated from the real room shape and materials. A tiled bathroom rings, a soft-furnished club sounds dead, caves echo. Early reflections help localisation. - **Pathing / diffraction:** sound bends around corners and through doorways. - **HRTF spatialisation:** proper 3D placement (height, front/back) on headphones. ## Questions to answer 1. **Integration point:** our audio stack is OpenAL (`LLAudioEngine_OpenAL`). Do we use Steam Audio's effects (binaural, direct, reflection, path) on OpenAL sources via an effect/mixing stage, or move the mix to our own mixer with Steam Audio applied per source? What is the latency and CPU cost per source? 2. **Geometry:** Steam Audio wants a scene (its own BVH, or Embree / Radeon Rays). Options: - feed it simplified region geometry (terrain + prims, physics shapes or low LOD) - use its **custom ray-tracer callback** to trace against our existing Vulkan RT TLAS (GPU) instead of a second copy of the world - per-region static scenes plus dynamic instanced meshes for moving objects SL geometry changes constantly. Measure scene build/update cost. 3. **Materials:** SL surfaces have no acoustic properties. Use one default material, a heuristic from texture/material names or PBR properties (metallic/rough → reflective), or a per-parcel / per-object override (a viewer-side setting or script-readable hint). 4. **Scale:** busy venues have many simultaneous sounds plus streaming music. Which sources get simulated (nearest N, priority by volume)? Parcel audio streams and voice stay dry by default; consider applying reverb to voice optionally. 5. **Fallbacks:** CPU path without RT hardware, a quality tiers setting, and off by default until proven. 6. **Platforms:** Linux / Windows / macOS builds of the Steam Audio SDK (prebuilt binaries vs source build), and their licence notices. ## Deliverable A prototype behind a setting: occlusion + reverb for in-world sounds in one test region, with measured CPU/GPU cost, then a go/no-go and an implementation plan.

sky_line added the

feature

label

2026-10-11 12:24:33 +00:00

sky_line added this to the Slipstream project

2026-10-11 12:24:33 +00:00

sky_line added the

status

up-next

label

2026-10-18 22:32:44 +00:00

sky_line referenced this issue

2026-10-19 00:09:40 +00:00

Explore: small on-device image model that infers surface acoustic material (for room acoustics) #852

sky_line referenced this issue from a commit

2026-10-19 01:29:02 +00:00

#683: EFX acoustics plumbing in the OpenAL engine (inert until driven)

sky_line referenced this issue from a commit

2026-10-19 01:29:02 +00:00

#683 phase 0: acoustics prototype -- GPU room / occlusion rays drive EFX

sky_line referenced this issue from a commit

2026-10-19 01:29:02 +00:00

#683: keep HUD-attachment sounds dry; no acoustics module on offscreen renderers

sky_line referenced this issue from a commit

2026-10-19 01:29:02 +00:00

Merge #683 phase 0: listener-driven reverb + occlusion muffling (OpenAL EFX, GPU ray queries)

sky_line referenced this issue from a commit

2026-10-19 01:29:02 +00:00

#683: material + clutter aware room absorption, acoustics retune

sky_line referenced this issue from a commit

2026-10-19 01:29:02 +00:00

Merge #683: acoustics room absorption from prim material + geometric clutter, retuned decay mapping

sky_line referenced this issue from a commit

2026-10-19 01:29:02 +00:00

Acoustics (#683): room rays see alpha-mask foliage (alpha-tested), flagged per hit

sky_line referenced this issue from a commit

2026-10-19 01:29:02 +00:00

Acoustics (#683): open area counts as absorption; size / clutter only when enclosed; foliage soft

sky_line referenced this issue from a commit

2026-10-19 01:29:02 +00:00

Acoustics (#683): foliage counts 0.25 of a wall (hedge arch beside a building near-dry)

sky_line referenced this issue from a commit

2026-10-19 01:29:02 +00:00

Merge #683: acoustics open area as absorption (Sabine), enclosure-gated room size, foliage seen as soft by room rays

sky_line referenced this issue from a commit

2026-10-19 01:29:02 +00:00

Regenerate world SPIR-V after the #683 acoustics merges

sky_line referenced this issue from a commit

2026-10-19 01:29:02 +00:00

Merge #683: occluded sources' reverb sends follow the occlusion filter (AudioAcousticsSendOcclusion)

sky_line referenced this issue

2026-10-19 01:41:37 +00:00

Explore: non-RT fallback for room acoustics via a GPU voxel grid around the listener #856

sky_line referenced this issue

2026-10-19 01:56:41 +00:00

Acoustics: per-source room estimate (sources in another room get their own reverb and send level) #857

sky_line referenced this issue

2026-10-19 01:56:41 +00:00

Acoustics: approximate diffraction via first-bounce visibility (sound around corners and through doorways) #858

sky_line commented

2026-10-19 01:56:46 +00:00

Author

Owner

Follow-ups filed:

#857: per-source room estimate (sources in another room get their own reverb and send level).
#858: approximate diffraction via first-bounce visibility (sound around corners and through doorways).

Also related:

#856: non-RT fallback via a GPU voxel grid (exploration).
#852: image model for surface acoustic material (exploration).

The full multi-bounce per-source echograms (Steam-Audio-class) remain this ticket's Phase 1 question.

**Follow-ups filed:** - #857: per-source room estimate (sources in another room get their own reverb and send level). - #858: approximate diffraction via first-bounce visibility (sound around corners and through doorways). **Also related:** - #856: non-RT fallback via a GPU voxel grid (exploration). - #852: image model for surface acoustic material (exploration). The full multi-bounce per-source echograms (Steam-Audio-class) remain this ticket's Phase 1 question.

sky_line referenced this issue from a commit

2026-10-19 11:20:27 +00:00

Preferences > Sound & Media: Room acoustics checkbox + reverb level slider (#683)

sky_line referenced this issue from a commit

2026-10-19 11:20:27 +00:00

Acoustics: stand down (soft, auto-resume) under VRAM ladder / RT budget pressure (#683)

sky_line referenced this issue from a commit

2026-10-19 11:20:27 +00:00

Merge #683: room acoustics preference (Sound & Media) + stands down under VRAM pressure / RT release

sky_line referenced this issue from a commit

2026-10-19 11:20:27 +00:00

Merge #683: acoustics cost reductions (room rays every N frames, 30 m room range, far foliage without alpha test)

sky_line commented

2026-10-20 15:03:21 +00:00

Author

Owner

Shipped in 1.4.0 Beta 1. Closing; reopen if it misbehaves.

sky_line

2026-10-20 15:03:21 +00:00

closed this issue
removed the
status
up-next
label

sky_line referenced this issue

2026-10-20 16:30:16 +00:00

Parcel audio from in-world speaker objects (positional, through room acoustics) #892

sky_line reopened this issue

2026-10-20 16:38:43 +00:00

sky_line commented

2026-10-20 16:38:43 +00:00

Author

Owner

Reopened: closed by mistake in a board sweep. The room acoustics shipped in Beta 1, but this ticket still has its Phase 1 question open: full multi-bounce reverb calculated per sound source. Follow-ups: #857, #858, #856, #852, #892.