Explore: small on-device image model that infers surface acoustic material (for room acoustics) #852

New issue

Open

opened 2026-10-19 00:09:40 +00:00 by sky_line · 0 comments

sky_line commented

2026-10-19 00:09:40 +00:00

Owner

Exploration only. Deferred; we'll look into it later.

Context

The room acoustics (#683) estimate how absorbent a room is from:

the object's physical material code (stone, metal, glass, wood and so on), which builders rarely set;
room clutter from the ray geometry;
metallic / gloss hints.

SL surfaces carry no acoustic data. Texture and material names aren't available for other people's content, and colour alone is ambiguous (brick reads as wood, grey fabric as concrete). Visual roughness is not acoustic absorption: concrete is visually rough but acoustically hard.

Idea

A small image-recognition network that looks at a surface's textures (base colour, plus normal / ORM maps where present) and outputs either:

a material class (wood, concrete, brick, stone, tile, plaster, metal, glass, fabric, carpet, foliage, ...) that we map through a database of acoustic properties (absorption per frequency band, scattering); or
the acoustic properties directly (absorption at low / mid / high frequency, scattering).

Questions to answer

Model: size, architecture and accuracy needed. Something MobileNet-sized or smaller, run once per texture, not per frame.
Where it runs: CPU (ONNX Runtime / a tiny hand-rolled inference) or a Vulkan compute shader. Licence and packaging of any runtime.
Cost: classify each texture once at decode/upload time, cache the result by texture UUID (it could live in the cache store, #715), budget the work so it never competes with frame time, and skip textures never hit by acoustic rays.
Training data: public material datasets (licences must allow redistribution of the trained weights), plus SL textures labelled by hand or by building names. Measure accuracy on real SL builds.
Inputs: base colour tinted by the face colour, normal-map detail (brick and stone have strong relief), texture scale relative to the face.
Combining: with the existing signals. The physical material code still wins when it's set; clutter stays as the geometric signal.
Payoff: does it audibly beat the current estimate in blind A/B listening across typical builds (wood cabin, concrete or brick building, tiled bathroom, furnished club, cave)?

Deliverable

A go/no-go with a prototype classifier evaluated offline on a labelled SL texture set: accuracy, size, cost per texture, and a listening comparison.

**Exploration only.** Deferred; we'll look into it later. ## Context The room acoustics (#683) estimate how absorbent a room is from: - the object's physical material code (stone, metal, glass, wood and so on), which builders rarely set; - room clutter from the ray geometry; - metallic / gloss hints. SL surfaces carry no acoustic data. Texture and material names aren't available for other people's content, and colour alone is ambiguous (brick reads as wood, grey fabric as concrete). Visual roughness is not acoustic absorption: concrete is visually rough but acoustically hard. ## Idea A small image-recognition network that looks at a surface's textures (base colour, plus normal / ORM maps where present) and outputs either: - **a material class** (wood, concrete, brick, stone, tile, plaster, metal, glass, fabric, carpet, foliage, ...) that we map through a database of acoustic properties (absorption per frequency band, scattering); or - **the acoustic properties directly** (absorption at low / mid / high frequency, scattering). ## Questions to answer - **Model:** size, architecture and accuracy needed. Something MobileNet-sized or smaller, run once per texture, not per frame. - **Where it runs:** CPU (ONNX Runtime / a tiny hand-rolled inference) or a Vulkan compute shader. Licence and packaging of any runtime. - **Cost:** classify each texture once at decode/upload time, cache the result by texture UUID (it could live in the cache store, #715), budget the work so it never competes with frame time, and skip textures never hit by acoustic rays. - **Training data:** public material datasets (licences must allow redistribution of the trained weights), plus SL textures labelled by hand or by building names. Measure accuracy on real SL builds. - **Inputs:** base colour tinted by the face colour, normal-map detail (brick and stone have strong relief), texture scale relative to the face. - **Combining:** with the existing signals. The physical material code still wins when it's set; clutter stays as the geometric signal. - **Payoff:** does it audibly beat the current estimate in blind A/B listening across typical builds (wood cabin, concrete or brick building, tiled bathroom, furnished club, cave)? ## Deliverable A go/no-go with a prototype classifier evaluated offline on a labelled SL texture set: accuracy, size, cost per texture, and a listening comparison.

sky_line added the

feature

priority

low

labels

2026-10-19 00:09:40 +00:00

sky_line added this to the Slipstream project

2026-10-19 00:09:40 +00:00

sky_line referenced this issue

2026-10-19 01:56:46 +00:00

Explore: Steam Audio (ray-traced occlusion, reverb, pathing, HRTF) for in-world sounds #683

sky_line referenced this issue