VR: local speech-to-text for chat and text fields (on-screen keyboard only as fallback) #1118

New issue

Open

opened 2026-10-26 20:55:23 +00:00 by sky_line · 1 comment

sky_line commented

2026-10-26 20:55:23 +00:00

Owner

Goal: dictate into any focused text field on the UI wall or wrist panel (nearby chat, IMs, search, notecards...): hold a button, speak, release, and the text appears in the field.

Model and runtime (to evaluate, measured, not guessed):

Released, permissively licensed, credited (released-code-only and licence-credit rules). Candidates: whisper.cpp (MIT) with small/base or distil English models; Moonshine (MIT, built for short low-latency utterances). Prefer our own integration code around a library.
Runs fully on the user's machine; no audio leaves it. Download the model on first use (not in the base package), with size shown and a checksum.
Budget: the GPU is busy rendering the headset, so measure CPU inference first (latency for a 5 s utterance on the owner's machine, the 3060 box and a low-end CPU), GPU (Vulkan backend) only if it fits the VR frame budget. The VR safety governor must not trip because of STT.
Multilingual model option for non-English users.

Audio: microphone capture independent of voice chat (SLVoice / WebRTC may already hold the mic -- check sharing); push-to-talk only, never always-listening.

UX: a visible "listening" indicator on the wall / wrist while held; the result lands in the field for editing (not auto-sent) when the UI is up. Keyboard fallback: a basic on-screen keyboard on the wall for corrections and for users without a mic.

Desktop: the same dictation could be offered on the desktop later; VR first. VR hard rule: nothing loads or runs when VR and dictation are off.

Test: dictate 3 sentences into nearby chat on the wall; latency from release to text; CPU/GPU cost while in VR; governor level unchanged.

Part of the VR epic #1072 (M3/M5 usability). Owner, 2026-10-26: on-screen keyboards are bad in VR; text input should come from a small, good, LOCAL speech-to-text model, with an on-screen keyboard as the fallback. **Goal:** dictate into any focused text field on the UI wall or wrist panel (nearby chat, IMs, search, notecards...): hold a button, speak, release, and the text appears in the field. **Model and runtime (to evaluate, measured, not guessed):** - Released, permissively licensed, credited (released-code-only and licence-credit rules). Candidates: whisper.cpp (MIT) with small/base or distil English models; Moonshine (MIT, built for short low-latency utterances). Prefer our own integration code around a library. - Runs fully on the user's machine; no audio leaves it. Download the model on first use (not in the base package), with size shown and a checksum. - Budget: the GPU is busy rendering the headset, so measure CPU inference first (latency for a 5 s utterance on the owner's machine, the 3060 box and a low-end CPU), GPU (Vulkan backend) only if it fits the VR frame budget. The VR safety governor must not trip because of STT. - Multilingual model option for non-English users. **Audio:** microphone capture independent of voice chat (SLVoice / WebRTC may already hold the mic -- check sharing); push-to-talk only, never always-listening. **UX:** a visible "listening" indicator on the wall / wrist while held; the result lands in the field for editing (not auto-sent) when the UI is up. Keyboard fallback: a basic on-screen keyboard on the wall for corrections and for users without a mic. **Desktop:** the same dictation could be offered on the desktop later; VR first. VR hard rule: nothing loads or runs when VR and dictation are off. **Test:** dictate 3 sentences into nearby chat on the wall; latency from release to text; CPU/GPU cost while in VR; governor level unchanged.

sky_line added the

feature

priority

medium

labels

2026-10-26 20:55:23 +00:00

sky_line self-assigned this

2026-10-26 20:55:23 +00:00

sky_line added this to the Slipstream project

2026-10-26 20:55:23 +00:00

sky_line referenced this issue

2026-10-26 20:57:11 +00:00