yunchan/d3ro-voice
1
0
Fork 0
Code Issues Pull requests Projects Releases 2 Packages Wiki Activity Actions

feat(desktop): make local speech transcription work end to end

Browse source 
Local dictation had never produced a transcript on an installed build. The
engine itself was healthy; every connection to it was broken.

Installed builds shipped no speech engine at all: the packaging config had no
entry for the faster-whisper sidecar and no pipeline step built one, so the app
always fell back to a system Python without the runtime. Development was broken
too, because the sidecar and SoX paths were resolved against the Vite output
directory instead of the app root, which also meant recording failed with a SoX
ENOENT. On hosts where localhost resolves only to IPv6, every local request was
refused outright, which silently disabled both local transcription and the local
LLM.

The sidecar is now built and bundled (including the Silero VAD data it needs),
gated by a packaging check that fails when the engine or its data is missing.
Paths are discovered from the app root and fail loudly when the engine is
absent. Local engine URLs are normalized to the IPv4 loopback, decoding is tuned
so repeated hallucinations cannot compound (the same transcript now takes about
a fifth of the time), the engine is warmed up at startup, and holding the hotkey
now shows the text forming live in the recording tip.
This commit is contained in:
Yun Chan 2026-09-18 00:48:47 +09:00
parent 359b244dc9
commit 2d585bfc29
52 changed files with 1450 additions and 3861 deletions
Download patch file Download diff file Expand all files Collapse all files

7
docs/map/10-feature-catalog.md
View file

@ -16,11 +16,11 @@ Status quick-reference: `[x]` done+verified · `[~]` partial/unverified · `[ ]`
|---|---|---|---|---|---|---|
| CAP-01 | Push-to-talk dictation (hold/release) | [x] | [-] | [x] | [-] | Desktop `VoiceModeService`; mobile RecordScreen via app CTA/notification action (no global hotkey) |
| CAP-02 | Hands-free toggle dictation | [x] | [-] | [x] | [-] | Desktop double-press; mobile toggle |
| CAP-03 | Live partial transcript while recording | [x] | [ ] | [ ] | [-] | Desktop `voice:partialTranscript` + recording-tip |
| CAP-03 | Live partial transcript while recording | [x] | [ ] | [ ] | [-] | Desktop `voice:partialTranscript` + recording-tip; producer added in `1.3.0` (`VoiceModeService._runPartial` → `LocalSTTService.transcribePartial`, 1.5 s cadence / 7.5 s window, never inserted). The row was `[x]` before any producer existed. |
| CAP-04 | Recording waveform + level meter | [x] | [x] | [x] | [-] | Desktop 9-bar cos distribution; mobile audio level |
| CAP-05 | Device/mic selection | [x] | [ ] | [~] | [-] | Desktop config; mobile uses system default |
| CAP-06 | System/loopback audio capture | [x] | [-] | [ ] | [-] | Desktop only (caption source); mobile policy-limited |
| CAP-07 | Local Whisper STT | [x] | [-] | [x] | [-] | Desktop sidecar; mobile on-device Whisper (supported devices) |
| CAP-07 | Local Whisper STT | [x] | [-] | [x] | [-] | Desktop ships the faster-whisper sidecar (`resources/sidecar`, built by `sidecar:build`, verified by `scripts/ci/verify-sidecar-bundle.mjs`), warms it up at app start, and connects over IPv4 loopback; mobile on-device Whisper (supported devices) |
| CAP-08 | Cloud STT (multi-provider) | [x] | [x] | [x] | [x] | Desktop 6 providers + D3RO Cloud; web/mobile via `stt-proxy`; .NET internal gateway |
| CAP-09 | STT auto-fallback + fail-closed | [x] | [x] | [x] | [x] | `STTManager`; SSOT R-021/R-022 GREEN |
| CAP-10 | STT model download/management UI | [x] | [-] | [~] | [-] | Desktop model manager + onboarding; mobile bundled model |
@ -191,7 +191,8 @@ Status quick-reference: `[x]` done+verified · `[~]` partial/unverified · `[ ]`
| INFRA-11 | Docker + NAS deploy | [x] | `docker-compose.nas.yml`, `scripts/deploy-nas.*` |
| INFRA-12 | Cloudflare edge + tunnel | [x] | `server/cloudflare-worker`, Cloudflare Tunnel `kd-nas` |
| INFRA-13 | Site deploy (Cloudflare Pages + GitHub Pages) | [x] | `.forgejo/workflows/deploy-site.yml`, `.github/workflows/deploy-site.yml` |
| INFRA-15 | Update & release system | [x] | Canonical Forgejo feed + channels/policy (`release/update-policy.json`, `src/main/update-policy.ts`), canonical publisher `scripts/ci/publish-forgejo-release.mjs`, legacy GitLab mirror; `npm run release:metadata:test`. v1.1.0 was published to Forgejo on 2026-09-15; product version moved to `1.2.0` as a forward-fix with CI-only publication, a same-version re-release guard, and download centers that link the feed instead of repository paths. |
| INFRA-15 | Update & release system | [x] | Canonical Forgejo feed + channels/policy (`release/update-policy.json`, `src/main/update-policy.ts`), canonical publisher `scripts/ci/publish-forgejo-release.mjs`, legacy GitLab mirror; `npm run release:metadata:test`. v1.1.0 was published to Forgejo on 2026-09-15; product version moved to `1.2.0` as a forward-fix with CI-only publication, a same-version re-release guard, and download centers that link the feed instead of repository paths. `1.3.0` (2026-09-18) carries the local-STT fixes; Windows publication still needs the CI signing secrets (`11` GAP-REL-02). |
| INFRA-16 | Desktop STT engine packaging | [x] | `apps/desktop/scripts/setup-sidecar.mjs` + `build-sidecar.mjs`, `electron-builder.yml` `extraResources` (`sidecar-dist/sidecar` → `resources/sidecar`, `resources/ffmpeg`), and `scripts/ci/verify-sidecar-bundle.mjs` run in `package-windows`/`package-macos` before electron-builder. Verified on the real bundle: `sidecar.exe` + `_internal` including `faster_whisper/assets/silero_vad_v6.onnx`, plus a packaged-engine transcription round-trip on GPU. |
---