feat(desktop): make local speech transcription work end to end
Local dictation had never produced a transcript on an installed build. The engine itself was healthy; every connection to it was broken. Installed builds shipped no speech engine at all: the packaging config had no entry for the faster-whisper sidecar and no pipeline step built one, so the app always fell back to a system Python without the runtime. Development was broken too, because the sidecar and SoX paths were resolved against the Vite output directory instead of the app root, which also meant recording failed with a SoX ENOENT. On hosts where localhost resolves only to IPv6, every local request was refused outright, which silently disabled both local transcription and the local LLM. The sidecar is now built and bundled (including the Silero VAD data it needs), gated by a packaging check that fails when the engine or its data is missing. Paths are discovered from the app root and fail loudly when the engine is absent. Local engine URLs are normalized to the IPv4 loopback, decoding is tuned so repeated hallucinations cannot compound (the same transcript now takes about a fifth of the time), the engine is warmed up at startup, and holding the hotkey now shows the text forming live in the recording tip.
This commit is contained in:
parent
359b244dc9
commit
2d585bfc29
52 changed files with 1450 additions and 3861 deletions
|
|
@ -28,8 +28,8 @@ Singleton + `EventEmitter` pattern (`getXService()` accessors).
|
|||
| Service | Purpose |
|
||||
|---|---|
|
||||
| `VoiceModeService` | Orchestrator: 9-state `RecognitionState` + 4-state `AudioState`, dual-condition flush, action queue. Events: session-started/completed/cancelled, transcription-update, audio-level, recognition/audio-state-changed, premium-llm-fallback, error |
|
||||
| `AudioCaptureService` | Mic PCM16 16kHz mono (SoX on Windows, node-record-lpcm16 elsewhere). Events: audio-data, audio-level, device-changed, started, stopped, error |
|
||||
| `LocalSTTService` | faster-whisper Python sidecar manager (state machine, dual-flush, model download/cancel) |
|
||||
| `AudioCaptureService` | Mic PCM16 16kHz mono (bundled SoX on Windows, node-record-lpcm16 elsewhere). Spawns hidden (`windowsHide`); a missing SoX fails with the exact fix command |
|
||||
| `LocalSTTService` | faster-whisper Python sidecar manager (state machine, dual-flush, model download/cancel, background warm-up, live partial transcription). Connects over IPv4 loopback (`getSidecarBaseUrl`) and fails fast with an actionable message when the bundled engine or virtualenv is missing |
|
||||
| `HotkeyService` | uiohook-napi global hooking (dictation/hands-free/command/caption). Events: hotkey-pressed/released, double-press, error |
|
||||
| `TextInsertService` | Clipboard save→set→Ctrl+V→restore via nut-js |
|
||||
| `SoundEffectService` | Preloaded WAV feedback (start/stop/error/cancel/chime) |
|
||||
|
|
@ -37,7 +37,7 @@ Singleton + `EventEmitter` pattern (`getXService()` accessors).
|
|||
### STT engine layer (`services/stt/`)
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `STTManager` | Dispatcher across local + 6 cloud providers, auto-fallback (events provider-changed, config-changed, fallback-to-local) |
|
||||
| `STTManager` | Dispatcher across local + 6 cloud providers, auto-fallback (events provider-changed, config-changed, fallback-to-local). `transcribePartial`/`warmUpLocal` route to the local engine only |
|
||||
| `types.ts` | `ISTTDriver` contract |
|
||||
| `audio-utils.ts` | `pcmToWav`, `createProbeWav` |
|
||||
| `drivers/OpenAI|Groq|Deepgram|AssemblyAI|Google|Custom|D3ROCloud` | Provider drivers; `D3ROCloudDriver` uses Supabase access token |
|
||||
|
|
@ -184,6 +184,8 @@ DB schema (`src/main/db/schema.ts`, drizzle SQLite): `history`, `dictionary`, `s
|
|||
- Core dictation/LLM/history pipeline: **implemented + tested** (~590 desktop tests; vitest + playwright).
|
||||
- Cross-platform packaging: Windows NSIS (signed, `forceCodeSigning`), macOS DMG/ZIP arm64 (ad-hoc signing); auto-update via canonical Forgejo feed with update policy (`release/update-policy.json`).
|
||||
- Local-first AI (SoX + faster-whisper sidecar + bundled Ollama) and cloud paths both present.
|
||||
- **Local STT is packaged** (`1.3.0`): `electron-builder.yml` `extraResources` copies `sidecar-dist/sidecar` → `resources/sidecar` and `resources/ffmpeg` → `resources/ffmpeg`; `scripts/ci/verify-sidecar-bundle.mjs` gates packaging. Build locally with `npm --prefix apps/desktop run sidecar:setup && npm --prefix apps/desktop run sidecar:build`. The sidecar stays in console mode so `stdout`/`stderr` reach the app log (UTF-8, line-buffered); a packaged sidecar **must** exist or startup fails loudly instead of silently falling back to a system Python.
|
||||
- All local engine URLs (`LocalSTTService`, `LocalLLMService`, `RAGService`, `OnlineLLMService`, `STTManager`) pass through `src/main/utils/loopback.ts`, which rewrites `localhost` to `127.0.0.1`, because some Windows hosts resolve `localhost` to IPv6 only and local engines bind IPv4.
|
||||
- Meeting intelligence, RAG, voice conversation (local + Realtime), captions, file transcription: implemented.
|
||||
- **Ad mediation**: `DirectHouseSponsorAdapter` performs real configurable REST bids; the other 9 adapters remain fail-closed stubs pending official SDKs (see `11-gap-backlog.md` GAP-ADS-01/02).
|
||||
- Tier resolution now routes through `@d3ro/core/entitlement` (`resolveEntitlement`, `normalizeEntitlementTier`); `useLicenseState.isPro` includes `pro_plus`.
|
||||
|
|
@ -206,3 +208,5 @@ DB schema (`src/main/db/schema.ts`, drizzle SQLite): `history`, `dictionary`, `s
|
|||
| Renderer shell / routes | `src/renderer/components/AppLayout.tsx` |
|
||||
| Update feed SSOT | `src/main/update-feed.ts` |
|
||||
| Update policy SSOT | `release/update-policy.json` + `src/main/update-policy.ts` |
|
||||
| Path/loopback resolution | `src/main/utils/paths.ts`, `src/main/utils/loopback.ts` |
|
||||
| Sidecar source / packaging | `sidecar/main.py`, `scripts/setup-sidecar.mjs`, `scripts/build-sidecar.mjs`, `scripts/ci/verify-sidecar-bundle.mjs` |
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue