feat(desktop): make local speech transcription work end to end

Local dictation had never produced a transcript on an installed build. The
engine itself was healthy; every connection to it was broken.

Installed builds shipped no speech engine at all: the packaging config had no
entry for the faster-whisper sidecar and no pipeline step built one, so the app
always fell back to a system Python without the runtime. Development was broken
too, because the sidecar and SoX paths were resolved against the Vite output
directory instead of the app root, which also meant recording failed with a SoX
ENOENT. On hosts where localhost resolves only to IPv6, every local request was
refused outright, which silently disabled both local transcription and the local
LLM.

The sidecar is now built and bundled (including the Silero VAD data it needs),
gated by a packaging check that fails when the engine or its data is missing.
Paths are discovered from the app root and fail loudly when the engine is
absent. Local engine URLs are normalized to the IPv4 loopback, decoding is tuned
so repeated hallucinations cannot compound (the same transcript now takes about
a fifth of the time), the engine is warmed up at startup, and holding the hotkey
now shows the text forming live in the recording tip.
This commit is contained in:
Yun Chan 2026-09-18 00:48:47 +09:00
parent 359b244dc9
commit 2d585bfc29
52 changed files with 1450 additions and 3861 deletions

View file

@ -13,6 +13,55 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Cloud-optional backup (encrypted, opt-in)
- Plugin system for custom pipelines
## [1.3.0] - 2026-09-18
> Published from an annotated tag through CI. Installer and update metadata are
> served by the canonical Forgejo feed; no binaries are committed to this repository.
### Added
- **Live partial transcript while dictating**: while the hotkey is held, the app now
transcribes the recent window of audio every 1.5 seconds and shows it in the recording
tip, so the text can be seen forming before the key is released. Partials never reach
the clipboard or the result popup; only the final transcription is inserted.
- **Instant first dictation**: the local speech engine (sidecar process + Whisper model)
is warmed up in the background at app start, so the first press does not wait for the
model to load.
- **Packaged local speech engine**: desktop installers now ship the faster-whisper
sidecar (`sidecar.exe` plus runtime data, including the Silero VAD model) and ffmpeg, so
local transcription works on a fresh install without Python on the machine.
- Sidecar build tooling: `npm --prefix apps/desktop run sidecar:setup` and
`sidecar:build`, plus a packaging-time bundle verifier that fails the build when the
engine or its VAD data is missing.
### Changed
- Local engine connections now target the IPv4 loopback (`127.0.0.1`) instead of
`localhost`. On machines where `localhost` resolves only to IPv6, every local request
(Ollama and the speech sidecar) was refused and local AI silently did nothing.
- Whisper decoding is tuned for dictation: previous-text conditioning is disabled so
repeated hallucinations cannot compound, silence is trimmed more aggressively, and
low-confidence fallbacks are bounded. Same transcript quality, roughly five times
faster on the same machine.
- The sidecar reuses an already-loaded model instead of reloading it, and reports the
load time it measured.
- Local engine logs stream to the app log as complete UTF-8 lines instead of mangled
fragments, so failures are diagnosable.
- Failed local-engine startup now fails immediately with an actionable message (missing
bundled engine, damaged virtualenv, or missing SoX) instead of waiting for a 30 second
health check and reporting a generic error.
### Fixed
- **Local transcription never worked in packaged builds**: the sidecar was not part of
the packaged resources and no pipeline step built it, so the app always fell back to a
system Python that had no faster-whisper installed.
- **Local transcription and local LLM never worked in development**: the sidecar and SoX
paths were resolved against the Vite output directory (`out/main`) instead of the app
root, so recording failed with a SoX `ENOENT` and the sidecar fell back to a Python
without the runtime dependencies.
- **Silero VAD data was missing from the bundled engine**, which would have made
silence-trimmed transcription fail at runtime even with the engine bundled.
- Audio capture and the speech sidecar no longer flash a console window on Windows, and
SoX/spawn failures name the fix (`npm --prefix apps/desktop run setup:sox`).
## [1.2.0] - 2026-09-16
> Published from an annotated tag through CI. Installer and update metadata are