Investigate: local (private) chat translation + grammar help -- CPU translation models, LanguageTool, bring-your-own endpoint #440

New issue

Open

opened 2026-10-05 16:50:14 +00:00 by sky_line · 0 comments

sky_line commented

2026-10-05 16:50:14 +00:00

Owner

User request: a "lightweight translation and grammar bot" powered by AI, running locally for privacy (no chat text sent to a cloud service).

Investigation, low priority. This ticket covers what's feasible and what we'd build. No commitment to ship.

Constraints

No AI model on the GPU. Many users run 4–6 GB cards, and the viewer already lives under a video memory budget (#405); the pressure ladder exists because people run out. Even a 1–3B model, heavily quantized, needs about 1–2.5 GB of video memory. It would compete with rendering (stutter, texture starvation), and models that small translate poorly anyway.
Privacy is the point. The requester wants text to stay on their machine, so cloud services don't satisfy the request (they remain an option for others).

Options, local first

Dedicated translation models on the CPU (best local fit).
- The models: OPUS-MT (MarianMT, roughly 300 MB per language pair) or NLLB-200 distilled (about 600M parameters, many languages in one model).
- The runtime: CTranslate2, a C++ library with int8 CPU inference and an MIT licence. Short chat lines should take tens to hundreds of ms on a modern CPU (to measure).
- They're far better at translation than a tiny general-purpose LLM, and they use no video memory.
- Costs: a large optional per-language download (not in the installer), a model manager UI, and integration work. It must run off the main thread at low priority, with a cache and a rate cap for busy nearby chat.
Local grammar checking: LanguageTool.
- Open source, rule-based, multilingual. It can run as a local server (Java), and the viewer talks to localhost.
- Possible UI: underline problems in the chat bar, or a "fix" button before sending.
- Costs: the user installs or runs the server (we could document it or offer a helper), and it's a Java dependency outside the viewer.
"Bring your own endpoint": an OpenAI-compatible provider (URL, optional key, model) in the existing translation system (lltranslate.cpp).
- Covers local Ollama or LM Studio on the same PC (if it has the hardware) or on another machine on the LAN, which keeps text private. Also covers cloud services for users who don't mind.
- The viewer doesn't run the model, so no video memory is used.
- Handles translation, and grammar via a "polish my message" action.
- The cheapest option to build.
Existing cloud translation (already in the code base). LL's chat translation supports Azure, Google and DeepL with the user's own API key. First step regardless of the rest: verify it still works in Slipstream, then document it on the wiki. Not private, but useful to many.

Suggested order if this is picked up

Verify and document the existing translation feature (#4). Small.
Add the OpenAI-compatible provider (#3). Small to medium, and it serves the privacy case for anyone with a local model server.
Optional LanguageTool integration (#2). Medium.
Only if there's real demand: built-in CPU translation with CTranslate2 and downloadable models (#1). Large.

Rules for any option

Opt-in only, never on by default for incoming chat.
A visible "translated" marker, with the original one click away.
The settings say where text goes (local vs. which service).
Translate on demand or only for marked languages. Cache results, cap the rate, and never block a frame.

**User request:** a "lightweight translation and grammar bot" powered by AI, running **locally** for privacy (no chat text sent to a cloud service). **Investigation, low priority.** This ticket covers what's feasible and what we'd build. No commitment to ship. ## Constraints - **No AI model on the GPU.** Many users run 4–6 GB cards, and the viewer already lives under a video memory budget (#405); the pressure ladder exists because people run out. Even a 1–3B model, heavily quantized, needs about 1–2.5 GB of video memory. It would compete with rendering (stutter, texture starvation), and models that small translate poorly anyway. - **Privacy is the point.** The requester wants text to stay on their machine, so cloud services don't satisfy the request (they remain an option for others). ## Options, local first 1. **Dedicated translation models on the CPU (best local fit).** - The models: OPUS-MT (MarianMT, roughly 300 MB per language pair) or NLLB-200 distilled (about 600M parameters, many languages in one model). - The runtime: CTranslate2, a C++ library with int8 CPU inference and an MIT licence. Short chat lines should take tens to hundreds of ms on a modern CPU (to measure). - They're far better at translation than a tiny general-purpose LLM, and they use no video memory. - Costs: a large optional per-language download (not in the installer), a model manager UI, and integration work. It must run off the main thread at low priority, with a cache and a rate cap for busy nearby chat. 2. **Local grammar checking: LanguageTool.** - Open source, rule-based, multilingual. It can run as a local server (Java), and the viewer talks to `localhost`. - Possible UI: underline problems in the chat bar, or a "fix" button before sending. - Costs: the user installs or runs the server (we could document it or offer a helper), and it's a Java dependency outside the viewer. 3. **"Bring your own endpoint": an OpenAI-compatible provider** (URL, optional key, model) in the existing translation system (`lltranslate.cpp`). - Covers local Ollama or LM Studio on the same PC (if it has the hardware) or on another machine on the LAN, which keeps text private. Also covers cloud services for users who don't mind. - The viewer doesn't run the model, so no video memory is used. - Handles translation, and grammar via a "polish my message" action. - The cheapest option to build. 4. **Existing cloud translation (already in the code base).** LL's chat translation supports Azure, Google and DeepL with the user's own API key. First step regardless of the rest: verify it still works in Slipstream, then document it on the wiki. Not private, but useful to many. ## Suggested order if this is picked up 1. Verify and document the existing translation feature (#4). Small. 2. Add the OpenAI-compatible provider (#3). Small to medium, and it serves the privacy case for anyone with a local model server. 3. Optional LanguageTool integration (#2). Medium. 4. Only if there's real demand: built-in CPU translation with CTranslate2 and downloadable models (#1). Large. ## Rules for any option - Opt-in only, never on by default for incoming chat. - A visible "translated" marker, with the original one click away. - The settings say where text goes (local vs. which service). - Translate on demand or only for marked languages. Cache results, cap the rate, and never block a frame.

sky_line added the

priority

low

feature

labels

2026-10-05 16:50:14 +00:00

sky_line added this to the Slipstream project

2026-10-06 15:04:14 +00:00