yunchan/d3ro-voice
1
0
Fork 0
Code Issues Pull requests Projects Releases 3 Packages Wiki Activity Actions

feat(caption): let live captions use their own speech model

Browse source 
The speech engine now keeps an auxiliary model next to the dictation model
and transcribes with whichever the request names, reloading it once if the
engine restarted. Settings > STT gains a live-caption model so captions can
run on large-v3-turbo while dictation keeps its own model. The runtime
minimum rises to 1.7.0 because older engines would silently ignore the
model choice.

Suggestion paging moves to Up/Down: the page follows the selection and the
last item waits while more candidates are being generated. The Left/Right
page shortcuts are removed; they did nothing until a page had filled and
clash with Intel's display-rotation hotkeys.
This commit is contained in:
Yun Chan 2026-09-24 21:46:54 +09:00
parent 39b8e7448e
commit 4b0f685941
29 changed files with 222 additions and 234 deletions
Download patch file Download diff file Expand all files Collapse all files

4
docs/map/10-feature-catalog.md
View file

@ -26,7 +26,7 @@ Status quick-reference: `[x]` done+verified · `[~]` partial/unverified · `[ ]`
| CAP-10 | STT model download/management UI | [x] | [-] | [~] | [-] | Desktop model manager + onboarding; mobile bundled model |
| CAP-11 | File transcription (audio/video) | [x] | [ ] | [x] | [~] | Desktop ffmpeg chunking; mobile import picker; web deferred |
| CAP-12 | Audio import from other apps (share intent) | [-] | [-] | [x] | [-] | Mobile Android `ACTION_SEND`/`ACTION_VIEW` (SSOT R-016 GREEN) |
| CAP-13 | Live captions overlay | [x] | [-] | [-] | [-] | Desktop `CaptionService` + caption-overlay popup; popup assets verified in the packaged build (`scripts/ci/verify-desktop-renderer-bundles.mjs`); GAP-INFRA-05 **2026-09-24 (unreleased, 1.7.0 candidate):** fixed 6 s batches replaced by streaming (`StreamingCaptionTrack` + core `caption-streaming.ts`): the uncommitted buffer is re-recognised every 1 s (greedy `partial`) and sent as `caption:delta` {text, stable} where `stable` is the LocalAgreement-2 prefix; 0.7 s of silence finalises the line with a full pass; unbroken audio over 12 s commits segments ending before the last 1.5 s using Whisper segment times; idle audio is trimmed to 1.5 s. Captions always use the local engine; `auto` language is pinned after the first final. Finished lines are refined by the local LLM (`buildCaptionRefinePrompt`, `acceptCaptionRefinement` rejects >35 % change) and replaced via `caption:segmentUpdated`; toggle `captionRefineEnabled` (Settings). Overlay: draggable handle with remembered position, waiting notice until the first caption. |
| CAP-13 | Live captions overlay | [x] | [-] | [-] | [-] | Desktop `CaptionService` + caption-overlay popup; popup assets verified in the packaged build (`scripts/ci/verify-desktop-renderer-bundles.mjs`); GAP-INFRA-05 **2026-09-24 (unreleased, 1.7.0 candidate):** fixed 6 s batches replaced by streaming (`StreamingCaptionTrack` + core `caption-streaming.ts`): the uncommitted buffer is re-recognised every 1 s (greedy `partial`) and sent as `caption:delta` {text, stable} where `stable` is the LocalAgreement-2 prefix; 0.7 s of silence finalises the line with a full pass; unbroken audio over 12 s commits segments ending before the last 1.5 s using Whisper segment times; idle audio is trimmed to 1.5 s. Captions always use the local engine; `auto` language is pinned after the first final. Finished lines are refined by the local LLM (`buildCaptionRefinePrompt`, `acceptCaptionRefinement` rejects >35 % change) and replaced via `caption:segmentUpdated`; toggle `captionRefineEnabled` (Settings). Overlay: draggable handle with remembered position, waiting notice until the first caption. Separate caption model `captionSttModelId` (null = dictation model): the sidecar keeps one aux model next to the primary (`/load` `slot:"aux"`, `/transcribe` `model_id`, 409 `model_not_loaded` → reload + one retry); runtime minimum raised to 1.7.0. |
| CAP-14 | Recording persistence / crash recovery | [x] | [ ] | [x] | [-] | Desktop WAV persist; mobile durable queue + process-kill WAV recovery |
| CAP-15 | Android foreground recording service | [-] | [-] | [x] | [-] | Mobile API 34 FGS + persistent notification (SSOT R-005 GREEN) |
| CAP-16 | Rebindable global key bindings (keyboard + mouse) | [x] | [-] | [-] | [-] | Contract SSOT `packages/core/src/keybinding.ts`: `KEY_CATALOG` (10 groups, `:615`), `KEYBINDING_ACTIONS` (6 actions, `:719`), `validateBinding` (`:953`), `detectBindingConflicts` (`:1016`). Multiple bindings per action persist as one `AppConfig.keyBindings` map (`packages/core/src/types.ts:459`), replacing the four singular `*Shortcut` fields; `ConfigService` migrates legacy values once (`ConfigService.ts:142`). `KeyBindingService` hooks keyboard **and** mouse via uiohook (`KeyBindingService.ts:387`) — MB1 is not bindable, MB2/MB3 need a modifier, MB4/MB5 are free, and no mouse button can be suppressed, so the original click still fires (warning surfaced in the UI). Selection is either key-recording or a searchable grouped dropdown (`KeyBindingPicker.tsx:536`). `history-popup`/`command-popup` were hardcoded in `bootstrap.ts` and are now rebindable actions (`bootstrap.ts:159`). **Verified 2026-09-21 on Windows by a manual run** (`%APPDATA%/d3ro-voice/logs/main.log`, 12:53–13:06): `ConfigService` migrated the four legacy shortcuts with the user's non-default values preserved exactly, `KeyBindingService` loaded 6 bindings for 6 actions and started the uiohook keyboard **and** mouse hook with zero boot errors, and keyboard plus mouse (MB4/MB5) bindings were exercised through the UI. A `Loaded 7 key binding(s) … for 6 action(s)` line later in the same session shows multi-binding working end to end. The migrated map was read back from `d3ro-voice-config.json`: legacy `*Shortcut` fields gone, no `displayLabel` left. Contract evidence: `packages/core` 117 tests GREEN, no renderer type errors in the key-binding files. **Still open:** `KeyBindingService` has no unit test of its own, macOS/Linux mouse behavior is unconfirmed (`11` GAP-KEY-02), and `command` still falls back to the dictation pipeline (GAP-KEY-03). W/M `[-]`: no OS-level global binding surface exists there (browser sandbox; mobile has no global hotkey, see CAP-01). B `[-]`: device-local setting, nothing server-side. See `11` GAP-KEY-02/03 (open), GAP-KEY-01 (`[x]`), and `11` §7 CONSTRAINT-I18N-01. |
@ -170,7 +170,7 @@ end-to-end behaviour has **not been verified by typing in a real app** (`11` GAP
| INPUT-04 | Typed-text learning (UIA, password-excluded) | [~] | [-] | [-] | [-] | Text is read from the focused field via `GET /uia/focus` (sidecar UIA bridge) and diffed longest-common-prefix/suffix, so **IME-committed Hangul/kana is counted correctly** — keycodes cannot reconstruct CJK text. UIA sends `hasSelection` only, derived by TextPattern range Start/End comparison without calling `GetText` on the selection range or adding a selected-text payload; the existing focused-field text can still include a selection. A non-collapsed selection immediately clears suggestions as `selection-active`. `IsPassword` is checked before any read (fail-closed); IME composition suppresses both stats and suggestions. **2026-09-24:** text equal to the control name (empty-field placeholder, e.g. "메시지 입력") is treated as empty (`withoutPlaceholderText`); a suggestion requires `editedSinceFocus` + typing within `recentTypingWindowMs` (8 s), otherwise `not-typing` — clicking into a pre-filled field no longer triggers suggestions. |
| INPUT-05 | Personal phrase corpus (typed + voice) | [~] | [-] | [-] | [-] | Sentence-level phrases from typed text and from voice history (`HistoryService.create` feeds `recordExternalText`), ranked by frequency/recency as prompt hints; users can delete individual phrases. **2026-09-24:** typed text from `LEARNING_EXCLUDED_APPS` (terminals, code editors, Agent Switchboard) is never learned, and every phrase must pass `isLearnablePhrase` (no box/block/geometric glyphs, ≥60 % letters). `_pruneUnlearnableCorpus` re-applies both rules to existing phrases/samples/edges on start and every retention cycle (first run on the author machine removed 69 phrases / 91 samples of terminal status lines and agent chats). |
| INPUT-06 | Next-sentence suggestion (ghost text) | [~] | [-] | [-] | [-] | `SuggestionService` + `buildSuggestionPrompt` (`llm-prompts.ts` SSOT, instruction stays in the system prompt). The 2026-09-22 guard contract is 600 ms debounce, ≥5 s between requests, 6 requests/min by default (hard-config maximum 12), 3 candidates, 64 output tokens, 12-character growth before regeneration, 8 s request timeout and `keep_alive: 2m`; boot warmup is removed. Each request has its own cancellation signal. Presentation-active includes candidates, `generating`, `warmingUp` and `partialText`; clear/dismiss aborts, invalidates the generation token, clears TTL state and emits `cleared`/hide, and a final success resets `generating=false`/`partialText=null`. Focused evidence for the lifecycle and Windows-child-process change: five test files / 80 tests passed; desktop typecheck/lint, Python `py_compile`, and `git diff --check` exited 0 (core 131-test pass was independently verified earlier). Status remains `[~]`: this is not app-restart, GUI overlay, or real automatic-typing evidence. **2026-09-24 (supersedes the numbers above):** `keep_alive: 10m`, warm-up on boot, on Ollama reconnect and whenever the model is presumed cold (cold reload measured 11.4 s > 8 s timeout); a session asks for 1 candidate, then a sequential fill loop appends one unique candidate at a time up to `maxCandidatesTotal` 12 (avoid-list in the prompt, stops after 2 empty fills); only the first request consumes rate/daily budget; stale aborts refund the budget; any further typing ends the session (`matchesSessionPrefix`, IME last-syllable tolerant); prompt forbids assistant-style questions. Terminals (`TERMINAL_APPS`) never get suggestions. |
| INPUT-07 | Caret-anchored suggestion overlay | [~] | [-] | [-] | [-] | `suggestion-overlay` popup placed by `anchorFloatingPanel` (caret → element → cursor fallback, flip above when the caret is near the bottom, clamped to the work area). Non-focusable; click-through unless `suggestionOverlayInteractive`. While actually visible it continues periodic UIA validation after 5 s and revalidates 120 ms after mouse-up; unavailable/non-editable focus or a non-collapsed selection aborts and hides it. X first hides the renderer panel, then main IPC directly hides `BrowserWindow` and dismisses the service, so late tokened results cannot revive it. Accept/next/prev/dismiss are four global key bindings (`suggestion-accept`/`next`/`prev`/`dismiss`, default `Ctrl+Alt+→/↓/↑/←`), and the overlay has a mouse close button. Up to three candidates are shown in a scrollable list with a warm-up/generating spinner. **2026-09-24:** one page of 3 fixed-height (2-line) numbered items, footer = range/progress + source + key guide derived from the live bindings (`buildSuggestionKeyHints`, shared modifier shown once). Keys: Ctrl+Alt+↑/↓ move, Ctrl+Alt+←/→ page (`suggestion-page-next/prev`), Ctrl+Alt+Enter accept, plain Esc closes while visible (overlay-scoped, not a binding), Ctrl+Alt+Backspace secondary dismiss; old default bindings migrate at tuning revision 5. Streaming updates never reposition/re-present the window (`decideSuggestionOverlayAction`); placement uses `anchorKind` — caret: below the line, element: outside the element (below → above → right → left → inner bottom-right). Navigation re-arms the visible TTL. Window 460×208. |
| INPUT-07 | Caret-anchored suggestion overlay | [~] | [-] | [-] | [-] | `suggestion-overlay` popup placed by `anchorFloatingPanel` (caret → element → cursor fallback, flip above when the caret is near the bottom, clamped to the work area). Non-focusable; click-through unless `suggestionOverlayInteractive`. While actually visible it continues periodic UIA validation after 5 s and revalidates 120 ms after mouse-up; unavailable/non-editable focus or a non-collapsed selection aborts and hides it. X first hides the renderer panel, then main IPC directly hides `BrowserWindow` and dismisses the service, so late tokened results cannot revive it. Accept/next/prev/dismiss are four global key bindings (`suggestion-accept`/`next`/`prev`/`dismiss`, default `Ctrl+Alt+→/↓/↑/←`), and the overlay has a mouse close button. Up to three candidates are shown in a scrollable list with a warm-up/generating spinner. **2026-09-24:** one page of 3 fixed-height (2-line) numbered items, footer = range/progress + source + key guide derived from the live bindings (`buildSuggestionKeyHints`, shared modifier shown once). Keys: Ctrl+Alt+↑/↓ move (the page follows the active item; at the last item `next` waits while filling, otherwise wraps; the 1.6.0 Left/Right page actions were removed as unreachable before a page filled and colliding with Intel display rotation), Ctrl+Alt+Enter accept, plain Esc closes while visible (overlay-scoped, not a binding), Ctrl+Alt+Backspace secondary dismiss; old default bindings migrate at tuning revision 5. Streaming updates never reposition/re-present the window (`decideSuggestionOverlayAction`); placement uses `anchorKind` — caret: below the line, element: outside the element (below → above → right → left → inner bottom-right). Navigation re-arms the visible TTL. Window 460×208. |
| INPUT-08 | Per-app exclusions & consent controls | [~] | [-] | [-] | [-] | `inputExcludedApps` (executable names, case-insensitive) blocks both collection context and suggestions; telemetry master switch, pause, text-learning toggle and "delete collected data" all live in Settings → Input. 30-day retention prune runs on start. |
| INPUT-09 | Flow Radar | [~] | [-] | [-] | [-] | `rankFlowWindows` ranks hourly aggregate activity density, character volume and edit stability into potential-focus time windows. It is not a real-session detector or session record. |
| INPUT-10 | Edit Friction | [~] | [-] | [-] | [-] | `calculateFrictionInsight` derives friction from char/backspace quantities and reports edits per 100 chars; it does not infer sentiment or productivity. |