feat(desktop): make local speech transcription work end to end
Local dictation had never produced a transcript on an installed build. The engine itself was healthy; every connection to it was broken. Installed builds shipped no speech engine at all: the packaging config had no entry for the faster-whisper sidecar and no pipeline step built one, so the app always fell back to a system Python without the runtime. Development was broken too, because the sidecar and SoX paths were resolved against the Vite output directory instead of the app root, which also meant recording failed with a SoX ENOENT. On hosts where localhost resolves only to IPv6, every local request was refused outright, which silently disabled both local transcription and the local LLM. The sidecar is now built and bundled (including the Silero VAD data it needs), gated by a packaging check that fails when the engine or its data is missing. Paths are discovered from the app root and fail loudly when the engine is absent. Local engine URLs are normalized to the IPv4 loopback, decoding is tuned so repeated hallucinations cannot compound (the same transcript now takes about a fifth of the time), the engine is warmed up at startup, and holding the hotkey now shows the text forming live in the recording tip.
This commit is contained in:
parent
359b244dc9
commit
2d585bfc29
52 changed files with 1450 additions and 3861 deletions
|
|
@ -22,7 +22,7 @@ Legend: `[ ]` open · `[~]` in progress · `[!]` blocked externally · `[x]` res
|
|||
|---|---|---|---|---|
|
||||
| GAP-QA-01 | Quality | Extreme Red Team: headful end-to-end bug hunting across real desktop Electron, Web Next.js, and CI pipelines. | `red_team_log.md`, `tests/e2e/red_team_cycle*.spec.ts`, `apps/web/e2e/red_team_cycle4_web.spec.ts` | `[x]` 2026-09-15: 18 scenarios executed, 14 defects caught and 100% resolved (infinite chunking loop DEF-008, IPC signature mismatch DEF-004, markdown editor typing rollback DEF-006, Web RSC Link serialization DEF-012, secret scanner lookahead DEF-013, etc.). All 18 scenarios GREEN with zero regressions. |
|
||||
| GAP-REL-01 | Release | Official release publication to Forgejo and active public download center deployment. | `scripts/ci/publish-forgejo-release.mjs`, `apps/web/src/app/download/page.tsx`, `site/src/sections/Download.tsx`, `apps/web/e2e/red_team_cycle4_web.spec.ts` | `[~]` 2026-09-15: v1.1.0 release assets (`D3RO-Voice-Setup-1.1.0-x64.exe`, `.blockmap`, `latest.yml`, `update-policy.json`) published to canonical Forgejo registry and release hub. 2026-09-16: the published 1.1.0 installer carries no Authenticode signature, so it does not satisfy the release policy; product version moved to `1.2.0` and publication must come from CI with the signing gate GREEN. Download centers in `apps/web` (`/download`) and `site` (`#download`) link the canonical Forgejo feed. |
|
||||
| GAP-REL-02 | Release | Windows stable publication needs an external public-trust Authenticode PFX, its password, the exact signer subject, and a Forgejo token, none of which live in the repository. | `.forgejo/workflows/release.yml`, `.gitlab-ci.yml`, `scripts/ci/verify-windows-release-artifact.ps1` | `[!]` 2026-09-16: every publisher fails closed without `WIN_CSC_*` and `FORGEJO_TOKEN`; provide them as protected CI secrets, then re-run the `1.2.0` tag pipeline. |
|
||||
| GAP-REL-02 | Release | Windows stable publication needs an external public-trust Authenticode PFX, its password, the exact signer subject, and a Forgejo token, none of which live in the repository. | `.forgejo/workflows/release.yml`, `.gitlab-ci.yml`, `scripts/ci/verify-windows-release-artifact.ps1` | `[!]` 2026-09-16: every publisher fails closed without `WIN_CSC_*` and `FORGEJO_TOKEN`; provide them as protected CI secrets, then re-run the tag pipeline. Still open as of `1.3.0` (2026-09-18): the `v1.3.0` tag must be built by CI with the signing gate GREEN. Local packaging cannot produce a signed installer (`forceCodeSigning: true`). |
|
||||
| GAP-ADS-01 | Ads | 9 of 10 desktop ad adapters still extend `UnavailableAdAdapter` (`provider_not_integrated`). | `apps/desktop/src/main/services/ads/*` | `[~]` 2026-09-13: `DirectHouseSponsorAdapter` is now a real configurable REST adapter (bid/impression/click/reward via `endpointUrl`; fail-closed when unconfigured; 22 unit tests GREEN). Remaining 9 need official SDKs/authenticated endpoints. |
|
||||
| GAP-ADS-02 | Ads | Desktop mediation reward accounting is not wired to license quota (`claimReward` still returns no tokens). | `AdMediationEngine.ts`, `AppLayout.tsx` | Wire verified `reportRewardCompletion` to `LicenseService` quota after the direct sponsor endpoint exists. |
|
||||
| GAP-ID-01 | Identity | Supabase, .NET JWT/SQLite, and the desktop offline license each had their own tier/role shape. | `@d3ro/core/entitlement`, `LicenseService`, `entitlement-context` | `[~]` 2026-09-13: canonical `EntitlementSnapshot` + `resolveEntitlement` added with tests; desktop tier normalization + `isPro` fixed. Full adoption tracked as GAP-ID-02. |
|
||||
|
|
@ -42,6 +42,12 @@ Legend: `[ ]` open · `[~]` in progress · `[!]` blocked externally · `[x]` res
|
|||
| GAP-PUSH-02 | Push | Nothing triggered `send-push?mode=drain`; enqueued notifications never left the outbox. | `server/cloudflare-worker/src/push-drain.ts`, `wrangler.toml` | `[x]` 2026-09-13: Cloudflare Cron Trigger (`* * * * *`) drains the outbox; tests in CI. Requires `SUPABASE_URL` + `SUPABASE_SERVICE_ROLE_KEY` secret on the worker. |
|
||||
| GAP-PUSH-03 | Push | Mobile/web clients register only `fcm`; no service worker subscription or APNs device token. | `apps/mobile-rn/src/features/notifications/*`, `apps/web` | Add web service worker + `pushManager.subscribe` (store JSON subscription) and iOS APNs token registration. |
|
||||
| GAP-PUSH-04 | Push | Android still depends on FCM (`google-services.json`). | `apps/mobile-rn/android`, `send-push` | Decide: minimal Firebase project, or UnifiedPush/ntfy. See `docs/deployment/push-transport-without-firebase.md`. |
|
||||
| GAP-STT-01 | Local STT | Packaged desktop builds shipped **no** faster-whisper sidecar: `electron-builder.yml` had no `extraResources` entry for it and no pipeline job built it. Local transcription was impossible on any installed build; the app fell back to a system Python without the runtime. | `apps/desktop/electron-builder.yml`, `.gitlab-ci.yml`, `apps/desktop/scripts/build-sidecar.mjs` | `[x]` 2026-09-18: sidecar bundled via `extraResources` (`sidecar-dist/sidecar` → `resources/sidecar`), `sidecar:setup`/`sidecar:build` scripts, and `scripts/ci/verify-sidecar-bundle.mjs` fails packaging when the engine or its Silero VAD data is missing. Verified on the real 242 MB bundle. |
|
||||
| GAP-STT-02 | Local STT | Dev resolved the sidecar/SoX paths against the Vite output dir (`out/main`), so the sidecar ran a Python without faster-whisper and SoX recording died with `spawn sox ENOENT`. | `apps/desktop/src/main/utils/paths.ts`, `apps/desktop/tests/main/utils/paths.test.ts` | `[x]` 2026-09-18: app root is discovered by walking up for app markers; packaged mode fails loudly instead of silently falling back. Unit tests pin both branches. |
|
||||
| GAP-STT-03 | Local engines | On hosts where `localhost` resolves only to IPv6, every local engine call (STT sidecar and Ollama) was refused. Audio capture and local LLM appeared dead. | `apps/desktop/src/main/utils/loopback.ts`, `LocalSTTService`, `LocalLLMService`, `RAGService`, `OnlineLLMService`, `STTManager` | `[x]` 2026-09-18: loopback normalization to `127.0.0.1` for all local engine URLs; defaults updated; 9 unit tests. Verified against the live sidecar and Ollama on a host with an IPv6-only `localhost`. |
|
||||
| GAP-STT-04 | Local STT | Live partial transcript (`CAP-03`, `voice:partialTranscript`) was marked done but had **no producer**: the channel, popup UI, and preload existed, nothing ever emitted. | `apps/desktop/src/main/services/VoiceModeService.ts`, `LocalSTTService.transcribePartial`, `STTManager.transcribePartial` | `[x]` 2026-09-18: 1.5 s cadence over a 7.5 s trailing window, greedy decode, drained before the final transcription; never inserted. |
|
||||
| GAP-STT-05 | Local STT | The bundled sidecar lacked faster-whisper's Silero VAD data, so `vad_filter=true` transcription would have failed at runtime even with the engine bundled. | `apps/desktop/scripts/build-sidecar.mjs`, `scripts/ci/verify-sidecar-bundle.mjs` | `[x]` 2026-09-18: `--collect-all faster_whisper` plus a packaging-time presence check for `assets/silero_vad_v6.onnx`. |
|
||||
| GAP-STT-06 | Local STT | Decode settings were untuned: previous-text conditioning let repeated hallucinations compound, and no VAD parameters meant slow, uneven segments. | `apps/desktop/sidecar/main.py` | `[x]` 2026-09-18: `condition_on_previous_text=false`, bounded low-temperature fallback, `no_speech`/`compression_ratio`/`log_prob` thresholds, 300 ms silence trimming. Same transcript, ~5x faster on the reference machine (7.7 s audio: 1609 ms → 303 ms). |
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -97,7 +103,9 @@ These are the mobile SSOT rows still `[ ]` / `[~]`. Do not duplicate the full te
|
|||
|
||||
## 5. Quick "is X done?" lookup
|
||||
|
||||
- **Desktop local dictation / LLM / history / meetings / RAG / conversation:** yes, tested. Ads: one real adapter, rest stubs.
|
||||
- **Desktop local dictation / LLM / history / meetings / RAG / conversation:** local
|
||||
dictation works in dev **and** in packaged builds as of `1.3.0` (engine bundled, paths
|
||||
fixed, IPv4 loopback). Ads: one real adapter, rest stubs.
|
||||
- **Web console:** yes, feature-complete for server-shared data; knowledge upload/search and team feed implemented.
|
||||
- **Mobile:** code complete for most flows and tested locally; blocked mainly by external store/console gates, plus a11y and some E2E depth.
|
||||
- **Backend:** fail-closed AI proxies, RLS, billing, ads SSV implemented; push transports (FCM + webpush + APNs) and cron drain implemented.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue