feat(desktop): make local speech transcription work end to end

Local dictation had never produced a transcript on an installed build. The
engine itself was healthy; every connection to it was broken.

Installed builds shipped no speech engine at all: the packaging config had no
entry for the faster-whisper sidecar and no pipeline step built one, so the app
always fell back to a system Python without the runtime. Development was broken
too, because the sidecar and SoX paths were resolved against the Vite output
directory instead of the app root, which also meant recording failed with a SoX
ENOENT. On hosts where localhost resolves only to IPv6, every local request was
refused outright, which silently disabled both local transcription and the local
LLM.

The sidecar is now built and bundled (including the Silero VAD data it needs),
gated by a packaging check that fails when the engine or its data is missing.
Paths are discovered from the app root and fail loudly when the engine is
absent. Local engine URLs are normalized to the IPv4 loopback, decoding is tuned
so repeated hallucinations cannot compound (the same transcript now takes about
a fifth of the time), the engine is warmed up at startup, and holding the hotkey
now shows the text forming live in the recording tip.
This commit is contained in:
Yun Chan 2026-09-18 00:48:47 +09:00
parent 359b244dc9
commit 2d585bfc29
52 changed files with 1450 additions and 3861 deletions

View file

@ -97,7 +97,7 @@ Per-app commands that matter:
| App | Commands |
|---|---|
| desktop | `npm run dev --workspace=@d3ro/desktop`, `build`, `typecheck`, `test` (vitest, 1266 tests), playwright e2e |
| desktop | `npm run dev --workspace=@d3ro/desktop`, `build`, `typecheck`, `test` (vitest), playwright e2e; local STT engine: `npm --prefix apps/desktop run sidecar:setup` then `sidecar:build` (PyInstaller → `sidecar-dist/sidecar`), full local Windows package: `npm --prefix apps/desktop run dist:win:full`; `setup:sox` re-downloads the bundled SoX |
| mobile-rn | `npm run typecheck:mobile` / `lint:mobile` / `test:mobile` (root), or `npm --prefix apps/mobile-rn run lint/typecheck/test`; android gradle builds, Maestro E2E |
| api-server | `dotnet build`, `dotnet test` (also `apps/api-server.Tests`) |
| web | `next build`, playwright e2e in `apps/web/e2e` |
@ -136,7 +136,7 @@ See [`03-shared-packages.md`](./03-shared-packages.md). Summary:
### GitLab CI (`.gitlab-ci.yml`)
Stages `validate → test → build → e2e → package → publish → deploy`. Primary pipeline for desktop Windows/macOS releases (Forgejo Generic Registry is the canonical updater feed; GitLab project 1172 is a legacy mirror) and production mobile releases (`mobile-production-release`, manual/protected). Admin NAS deploy job is intentionally **disabled**.
Stages `validate → test → build → e2e → package → publish → deploy`. Primary pipeline for desktop Windows/macOS releases (Forgejo Generic Registry is the canonical updater feed; GitLab project 1172 is a legacy mirror) and production mobile releases (`mobile-production-release`, manual/protected). Admin NAS deploy job is intentionally **disabled**. `package-windows`/`package-macos` build the faster-whisper sidecar (`sidecar:setup``sidecar:build`) and run `scripts/ci/verify-sidecar-bundle.mjs` before electron-builder, so a release can never ship without the local STT engine.
### Forgejo Actions (`.forgejo/workflows/`)
@ -190,11 +190,11 @@ Full detail: [`09-supabase-backend.md`](./09-supabase-backend.md).
| File | Purpose |
|---|---|
| `release/product-version.json` | version `1.2.0`, `androidVersionCode`/`iosBuildNumber` `1020001`, releaseDate, desktop license keyId |
| `release/product-version.json` | version `1.3.0`, `androidVersionCode`/`iosBuildNumber` `1030001`, releaseDate, desktop license keyId |
| `release/android-release-identity.json` | package `com.d3ro.voice`, Play app ID, app-signing SHA-256, upload cert SHA-256, evidence keyId, AdMob unit IDs |
| `release/desktop-license-public.pem` | Ed25519 public key for desktop offline licenses |
| `release/mobile-release-evidence-public.pem` | Ed25519 public key for mobile release evidence |
| `apps/desktop/electron-builder.yml` | appId `com.d3ro.voice`, NSIS x64 (forced code signing), macOS DMG/ZIP arm64, generic Forgejo publish feed, asarUnpack native modules, extraResources (icons, sounds, sox, ollama) |
| `apps/desktop/electron-builder.yml` | appId `com.d3ro.voice`, NSIS x64 (forced code signing), macOS DMG/ZIP arm64, generic Forgejo publish feed, asarUnpack native modules + `@ffmpeg-installer`, extraResources (icons, sounds, sox, **sidecar**, ffmpeg, ollama) |
| `apps/desktop/src/main/update-feed.ts` | Auto-update feed SSOT (canonical Forgejo + legacy GitLab mirror, channels) |
| `release/update-policy.json` | Update policy SSOT (channels, minimum supported version, forced update, delta/full, staged rollout, kill switch) |
| `apps/desktop/src/main/update-policy.ts` | Policy parsing/decision logic |
@ -207,6 +207,8 @@ Version sync is enforced by `scripts/ci/sync-version.mjs` and `verify-release-me
## 10. Resources & tests
- `resources/sox/` — bundled Windows SoX (`sox.exe` + DLLs) for audio capture.
- `resources/ffmpeg/` — optional bundled ffmpeg (CI or manual); `getFfmpegPath()` also resolves the `@ffmpeg-installer/ffmpeg` binary from `app.asar.unpacked`.
- `apps/desktop/sidecar-dist/` — PyInstaller sidecar bundle consumed by `extraResources` (gitignored; built by `npm --prefix apps/desktop run sidecar:build`).
- `resources/icons/` — empty; electron-builder falls back to `build/icon.ico|png`.
- Desktop tests: `apps/desktop/tests/` (vitest unit + playwright e2e), `apps/desktop/test-results/`.
- Mobile tests: `apps/mobile-rn/__tests__/` (57 suites / 353 tests per mobile SSOT), `.maestro/` + `.maestro-output/` E2E evidence. Note: a full parallel Jest run can hit the 5s render timeout on slow machines; re-run the failing spec in isolation before treating it as a regression.