feat(keybinding): several shortcuts per action, mouse buttons, searchable picker

Shortcuts were defined in four places that drifted apart: per-action IPC channel
pairs, a hand-written VK table in the service, a second one in the renderer, and
three copies of the keycap styling. Adding an action meant editing all of them,
so two shortcuts stayed hardcoded in bootstrap and one had no settings entry at
all.

packages/core/src/keybinding.ts is now the single source for the binding type,
the selectable key catalog, the action catalog, normalization, validation,
conflict detection, display labels, search and deserialization. Main, preload
and renderer all read from it; nothing redefines keys or rules locally.

- Each action holds a list of bindings instead of one. AppConfig's four
  *Shortcut fields collapse into a single keyBindings map, migrated on launch.
- Mouse buttons can be bound. Left click is refused, right/middle need a
  modifier, side buttons are free. uiohook cannot swallow events, so the
  original click still fires and the UI says so.
- Keys can be picked from a grouped dropdown with a search box, not only by
  recording a keypress.
- HOTKEY's 14 channels become KEYBINDING's 9, taking the action as a parameter,
  so actions no longer multiply channels. The history and command popups moved
  out of bootstrap into ordinary actions.
- displayLabel is gone; labels derive from the binding and follow the app
  language and platform.

Fixes found on the way:
- Double-press hands-free was unreachable: lookup returned only the first
  matching action, and dictation shares its default binding.
- Reserved-combination checks compared joined key names, so a different modifier
  order let Ctrl+C through.
- Disabling shortcuts released every global registration in the process,
  including the popup ones, and never restored them.
- Enabling shortcuts after starting disabled left nothing registered.
- The dashboard stored the caption event payload instead of the state in it.
This commit is contained in:
Yun Chan 2026-09-21 13:41:47 +09:00
parent 0ca9e242fa
commit 4ad1ae6ed4
49 changed files with 5901 additions and 1792 deletions

View file

@ -2,7 +2,7 @@
> Status: ACTIVE
> Last full audit: 2026-09-13
> Last update: 2026-09-19 — GAP-INFRA-05 (desktop renderer popup bundle verification wired into CI); 1.3.7 published to the updater feed
> Last update: 2026-09-21 — CAP-16 (desktop key bindings rebuilt on one `@d3ro/core/keybinding` SSOT: multiple bindings per action, mouse buttons, `HOTKEY` → `KEYBINDING` IPC group); verified on Windows by a manual run, so CAP-16 and CAP-02 are `[x]` and GAP-KEY-01 is closed; GAP-KEY-02/03, GAP-QA-02, GAP-I18N-01/02, GAP-INFRA-06 remain open; `11` gained §7 for accepted design constraints (things deliberately kept, not gaps)
> Scope: entire monorepo `D:/workspace/D3ROVoice` at product version `1.3.7`
> Purpose: let any agent (or human) answer two questions in under a minute:
> 1. **What infrastructure exists?** (build, CI, services, APIs, data, packages, deploy)

View file

@ -15,7 +15,7 @@ A multi-platform AI voice assistant: press/hold or tap to speak, get a transcrip
| Surface | Path | Stack | Runtime model | Role |
|---|---|---|---|---|
| Desktop | `apps/desktop` | Electron 33 + React 19 + MUI 7 + Vite | **Local-first** (SoX, faster-whisper sidecar, Ollama, SQLite) with optional cloud sync | The flagship: global hotkey dictation, text insertion into other apps, meetings, captions, RAG, voice conversation, OS actions |
| Desktop | `apps/desktop` | Electron 33 + React 19 + MUI 7 + Vite | **Local-first** (SoX, faster-whisper sidecar, Ollama, SQLite) with optional cloud sync | The flagship: global key-binding dictation (keyboard or mouse, rebindable — CAP-16), text insertion into other apps, meetings, captions, RAG, voice conversation, OS actions |
| Web | `apps/web` | Next.js 15 App Router + Supabase | **Cloud** | Browser console: record/STT, history, commands, meetings, knowledge, teams, chat, billing |
| Mobile | `apps/mobile-rn` | React Native 0.85 + React 19 (CLI, not Expo) | **Cloud-first** (Supabase + Edge Functions), on-device Whisper fallback | Product mobile app: recording/import, history, meetings, memos, templates, teams, Talk, admin, data portability, IAP + ads |
| API server | `apps/api-server` | ASP.NET Core 10 + EF Core + SQLite | Cloud (self-hosted/NAS) | LLM/STT proxy and admin back-office backend for the .NET identity side |

View file

@ -114,7 +114,7 @@ See [`03-shared-packages.md`](./03-shared-packages.md). Summary:
| Package | Provides |
|---|---|
| `@d3ro/core` | Domain types, `D3ROError`/`ErrorCode`, IPC channel SSOT, constants, `crypto-license`, `pii-redactor`, `secure-memory`, `supabase-config`, `meeting-markdown`, `markdown-to-docx` |
| `@d3ro/core` | Domain types, `D3ROError`/`ErrorCode`, IPC channel SSOT, key-binding SSOT (`./keybinding`), constants, `crypto-license`, `pii-redactor`, `secure-memory`, `supabase-config`, `meeting-markdown`, `markdown-to-docx`; has vitest tests (`packages/core/vitest.config.ts`, `npm run test --workspace=@d3ro/core`) |
| `@d3ro/ui` | Theme tokens, CSS vars, MUI DS components (web/desktop) |
| `@d3ro/ui-native` | RN design system (MetalCard, PhosphorText, Led, PhysicalButton, WaveBars, …) |
| `@d3ro/i18n` | 12 locales, `I18nProvider`, `t()`, date/number/relative formatters |

View file

@ -15,7 +15,8 @@ The canonical place for types and cross-surface logic. Both desktop and web/mobi
|---|---|---|
| Types | `./types` | Domain types shared across surfaces |
| Errors | `./errors` | `D3ROError`, `ErrorCode` |
| IPC channels | `./ipc-channels` | `IPC_CHANNELS` object + `IPCChannel` union. **SSOT** for every desktop IPC channel (VOICE, AUDIO, STT, TTS, LLM, HOTKEY, CONFIG, HISTORY, DICTIONARY, WINDOW, SYSTEM, STATS, MEMO, VOICE_COMMAND, CONTEXT, CHAIN, CAPTION, FILE_TRANSCRIPTION, MEETING_SUMMARY, DICTATION_TEMPLATE, VOICE_CONVERSATION, RAG, VOICE_ACTION, MEETING_MODE, MEETING_DOC_TEMPLATE, MEETING_CHAT, LICENSE, CLOUD_SYNC, INSTRUCTION, SYSTEM_AUDIO, POPUP_RESULT, POPUP_HISTORY, POPUP_COMMAND, POPUP_CAPTION, VOICE_PARTIAL, CLIPBOARD, APP, ONLINE_AUTH, ADS, SUPPORT, PAYMENT) |
| IPC channels | `./ipc-channels` | `IPC_CHANNELS` object + `IPCChannel` union. **SSOT** for every desktop IPC channel (VOICE, AUDIO, STT, TTS, LLM, KEYBINDING, CONFIG, HISTORY, DICTIONARY, WINDOW, SYSTEM, STATS, MEMO, VOICE_COMMAND, CONTEXT, CHAIN, CAPTION, FILE_TRANSCRIPTION, MEETING_SUMMARY, DICTATION_TEMPLATE, VOICE_CONVERSATION, RAG, VOICE_ACTION, MEETING_MODE, MEETING_DOC_TEMPLATE, MEETING_CHAT, LICENSE, CLOUD_SYNC, INSTRUCTION, SYSTEM_AUDIO, POPUP_RESULT, POPUP_HISTORY, POPUP_COMMAND, POPUP_CAPTION, VOICE_PARTIAL, CLIPBOARD, APP, ONLINE_AUTH, ADS, SUPPORT, PAYMENT) |
| Key bindings | `./keybinding` | **SSOT** for every global shortcut in the app: `KeyBinding` (`device`/`code`/`ctrl`/`alt`/`shift`/`meta`), `KEY_CATALOG` (10 selectable groups incl. mouse), `KEYBINDING_ACTIONS` (6 rebindable actions), `bindingKey`/`normalizeBinding`/`validateBinding`/`detectBindingConflicts`/`formatBindingSegments`/`searchKeyCatalog`/`parseBindingMap`. Persisted as `AppConfig.keyBindings`. i18n keys are exposed as plain `string` so core stays independent of `@d3ro/i18n`; consumers narrow at the boundary (`asTranslationKey`) and a contract test guards the keys — accepted constraint, `11` §7 CONSTRAINT-I18N-01. Tests: `__tests__/keybinding*.test.ts` via `vitest.config.ts` (`npm run test --workspace=@d3ro/core`), 117 cases as of 2026-09-21 |
| Constants | `./constants` | Shared constants |
| Crypto license | `./utils/crypto-license` | Ed25519 license sign/verify (used by admin issuer + desktop verifier) |
| PII | `pii-redactor`, `secure-memory` | Redaction + secure memory helpers |

View file

@ -16,7 +16,7 @@
**Main entry** `src/main/index.ts`: sets app name/AppUserModelId, disables GPU acceleration, EPIPE/uncaught handlers, registers `d3ro-voice://` deep-link protocol (Supabase OAuth implicit + PKCE), single-instance lock, then `bootstrap()` + `setupLifecycle()`.
**Bootstrap** `src/main/bootstrap.ts`: ordered `BootstrapStep[]` — logger, config, **database (critical)**, license, create-windows (critical), tray, **ipc-handlers (critical)**, custom-instructions, voice-commands, sound-effects, auto-launch, popup-preload, hotkey, voice-mode, llm-polling, meeting-summary-wiring, meeting-mode, cloud-sync, auto-update. Wires VoiceMode events to sound + history persistence.
**Bootstrap** `src/main/bootstrap.ts`: ordered `BootstrapStep[]` — logger, config, **database (critical)**, license, create-windows (critical), tray, **ipc-handlers (critical)**, custom-instructions, voice-commands, sound-effects, auto-launch, popup-preload, key-bindings, voice-mode, stt-warmup, llm-polling, meeting-summary-wiring, meeting-mode, cloud-sync, auto-update. Wires VoiceMode events to sound + history persistence, and subscribes to `KeyBindingService` `triggered` for the `history-popup` / `command-popup` actions (`bootstrap.ts:159`) — those two were hardcoded accelerators before and are now rebindable like everything else.
---
@ -30,7 +30,7 @@ Singleton + `EventEmitter` pattern (`getXService()` accessors).
| `VoiceModeService` | Orchestrator: 9-state `RecognitionState` + 4-state `AudioState`, dual-condition flush, action queue. Events: session-started/completed/cancelled, transcription-update, audio-level, recognition/audio-state-changed, premium-llm-fallback, error |
| `AudioCaptureService` | Mic PCM16 16kHz mono (bundled SoX on Windows, node-record-lpcm16 elsewhere). Spawns hidden (`windowsHide`); a missing SoX fails with the exact fix command |
| `LocalSTTService` | faster-whisper Python sidecar manager (state machine, dual-flush, model download/cancel, background warm-up, live partial transcription). Connects over IPv4 loopback (`getSidecarBaseUrl`) and fails fast with an actionable message when the bundled engine or virtualenv is missing |
| `HotkeyService` | uiohook-napi global hooking (dictation/hands-free/command/caption). Events: hotkey-pressed/released, double-press, error |
| `KeyBindingService` | uiohook-napi global hooking for **keyboard and mouse**, driven by the `@d3ro/core/keybinding` contract: 6 rebindable actions (dictation, hands-free, command, caption, history-popup, command-popup), several bindings per action, structural reserved-combo checks. Events: `triggered` (in-process payload carries `actionId`, `type` (`pressed`/`released`), `isDoublePress`, `holdMode`, `timestamp`; the renderer-facing `keybinding:triggered` event is the narrower `KeyBindingTriggeredEvent`, `keybinding.ts:1092`), `changed`, `error`. `globalShortcut` is used only to mute the macOS system beep, and only for accelerators it registered itself. Mouse events cannot be suppressed by uiohook, so a bound button also performs its native action |
| `TextInsertService` | Clipboard save→set→Ctrl+V→restore via nut-js |
| `SoundEffectService` | Preloaded WAV feedback (start/stop/error/cancel/chime) |
@ -118,8 +118,8 @@ Registry: `src/main/ipc/index.ts` calls 29 `registerXHandlers()` in fixed order.
| `dictionary-handlers` | DICTIONARY |
| `file-transcription-handlers` | FILE_TRANSCRIPTION |
| `history-handlers` | HISTORY + `stats:getSummary` |
| `hotkey-handlers` | HOTKEY |
| `instruction-handlers` | INSTRUCTION |
| `keybinding-handlers` | KEYBINDING |
| `license-handlers` | LICENSE |
| `llm-handlers` | LLM + `llm:premium:*` + ONLINE_AUTH |
| `meeting-doc-template-handlers` | MEETING_DOC_TEMPLATE |
@ -138,7 +138,9 @@ Registry: `src/main/ipc/index.ts` calls 29 `registerXHandlers()` in fixed order.
| `voice-handlers` | VOICE |
| `window-handlers` | WINDOW + `SYSTEM.OPEN_EXTERNAL` |
Preload exposes **`window.electronAPI`** with 33 namespaces: `platform, audio, config, voice, stt, hotkey, llm (incl. premium), history, dictionary, stats, window, system, instruction, app, memo, voiceCommand, context, chain, caption, license, fileTranscription, meetingSummary, dictationTemplate, rag, voiceAction, voiceConversation, meetingMode, meetingChat, meetingDocTemplate, cloudSync, onlineAuth, ads, support, payment`. Envelope: `IPCResult<T>` (success/error); `app.onDataChanged` is the global refresh channel.
The **`KEYBINDING`** group replaced the old per-action `HOTKEY` group. `HOTKEY` had 14 channels — a get/set pair per action plus three that were never implemented — so every new action meant new channels. `KEYBINDING` is 9 channels that take the action **as a parameter**: `getMap`, `setBindings`, `resetAction`, `resetAll`, `validate`, `isEnabled`, `setEnabled`, plus the `triggered` / `changed` events (`packages/core/src/ipc-channels.ts:104`). Adding an action now costs zero channels.
Preload exposes **`window.electronAPI`** with 33 namespaces: `platform, audio, config, voice, stt, keybinding, llm (incl. premium), history, dictionary, stats, window, system, instruction, app, memo, voiceCommand, context, chain, caption, license, fileTranscription, meetingSummary, dictationTemplate, rag, voiceAction, voiceConversation, meetingMode, meetingChat, meetingDocTemplate, cloudSync, onlineAuth, ads, support, payment`. The `keybinding` bridge is 9 methods mirroring the channels above (`src/preload/index.ts:323`), replacing the 11-method `hotkey` bridge. Envelope: `IPCResult<T>` (success/error); `app.onDataChanged` is the global refresh channel.
---
@ -157,8 +159,8 @@ Vanilla popups (`src/renderer/popups/`):
|---|---|
| `recording-tip` | 9-bar waveform indicator, partial transcript |
| `result-popup` | Transcription result + copy, auto-close with hover pause |
| `history-popup` | Recent transcriptions; ↑↓/Enter/1-9/ESC |
| `command-popup` | Command selection (Ctrl+Shift+C) |
| `history-popup` | Recent transcriptions; ↑↓/Enter/1-9/ESC. Opened by the `history-popup` action (default `Ctrl+Shift+V`, rebindable) |
| `command-popup` | Command selection. Opened by the `command-popup` action (default `Ctrl+Shift+C`, rebindable) |
| `caption-overlay` | Live caption overlay (font/opacity/maxLines) |
---
@ -177,9 +179,11 @@ Routing is state-based in `AppLayout.tsx` (`Route` union + `NAV_ITEMS`), no reac
| `KnowledgeBasePage` | knowledge | Local RAG: add/index docs, semantic query, reindex/remove |
| `MeetingModePage` | meeting | Meeting studio: live transcript, memos, doc generation/export, diarization |
Modals/components: `SettingsModal` (tabs General/Audio/STT/LLM/License/Cloud/About), `LicenseModal`, `LicenseTab`, `CloudSyncSection`, `OnboardingModal`, `UpgradePromptModal`, `ProBadge`, `TemplateSection`, `FileDropZone`, `HotkeyRecordModal`, `OllamaGuideModal`, `CodexOAuthGuideModal`, `TitleBar`, `StatusBar`, meeting components (9), voice-conversation, payment (`CheckoutModal`, `checkout-flow.ts`), support (`SupportModal`), ads (`AdBanner`, `RewardedQuotaModal`), shared cards.
Modals/components: `SettingsModal` (tabs General/Audio/STT/LLM/License/Cloud/About), `LicenseModal`, `LicenseTab`, `CloudSyncSection`, `OnboardingModal`, `UpgradePromptModal`, `ProBadge`, `TemplateSection`, `FileDropZone`, `OllamaGuideModal`, `CodexOAuthGuideModal`, `TitleBar`, `StatusBar`, meeting components (9), voice-conversation, payment (`CheckoutModal`, `checkout-flow.ts`), support (`SupportModal`), ads (`AdBanner`, `RewardedQuotaModal`), shared cards.
Hooks: `useRealtimeConversation` (OpenAI Realtime WebRTC), `useLicenseState`, `useProFeature`.
Key-binding UI lives in `components/keybinding/` (`Keycap`, `KeyBindingPicker`, `KeyBindingField`, `translation-key`), embedded in the Settings **General** tab (`SettingsModal.tsx:239`) — one field per action plus a global on/off switch. The picker offers both key recording and a searchable grouped dropdown (MUI `Autocomplete` over `KEY_CATALOG`, `KeyBindingPicker.tsx:536`). It replaced `HotkeyRecordModal`. `renderer/utils/format-hotkey.ts` is now a 17-line platform adapter only; key names, modifier glyphs, and join rules come from `@d3ro/core/keybinding`.
Hooks: `useRealtimeConversation` (OpenAI Realtime WebRTC), `useLicenseState`, `useProFeature`, `useKeyBindingMap` (subscribes to `keybinding:changed`; the dashboard renders the live `dictation` binding through `BindingKeycaps`).
DB schema (`src/main/db/schema.ts`, drizzle SQLite): `history`, `dictionary`, `stats`, `memo_tags`, `daily_usage`, `rag_documents`, `rag_chunks`, `meeting_sessions`, `meeting_memos`, `meeting_documents`.
@ -187,12 +191,14 @@ DB schema (`src/main/db/schema.ts`, drizzle SQLite): `history`, `dictionary`, `s
## 6. Desktop status summary
- Core dictation/LLM/history pipeline: **implemented + tested** (~590 desktop tests; vitest + playwright).
- Core dictation/LLM/history pipeline: **implemented + tested**. Measured 2026-09-21: 1314 vitest cases in `apps/desktop`, 1311 passing; playwright e2e is separate. The failures are environment-dependent rather than regressions — two need a local sidecar venv or embedding server, one pins an error message that has since changed (`11` GAP-QA-02). These numbers hold with `better-sqlite3` built for the host Node ABI; rebuilding it for Electron to run the app invalidates them until you rebuild back (`11` GAP-INFRA-06).
- Cross-platform packaging: Windows NSIS (signed, `forceCodeSigning`), macOS DMG/ZIP arm64 (ad-hoc signing); auto-update via canonical Forgejo feed with update policy (`release/update-policy.json`).
- Local-first AI (SoX + faster-whisper sidecar + bundled Ollama) and cloud paths both present.
- **Local STT is packaged** (`1.3.0`): `electron-builder.yml` `extraResources` copies `sidecar-dist/sidecar` → `resources/sidecar` and `resources/ffmpeg` → `resources/ffmpeg`; `scripts/ci/verify-sidecar-bundle.mjs` gates packaging. Build locally with `npm --prefix apps/desktop run sidecar:setup && npm --prefix apps/desktop run sidecar:build`. The sidecar stays in console mode so `stdout`/`stderr` reach the app log (UTF-8, line-buffered); a packaged sidecar **must** exist or startup fails loudly instead of silently falling back to a system Python.
- All local engine URLs (`LocalSTTService`, `LocalLLMService`, `RAGService`, `OnlineLLMService`, `STTManager`) pass through `src/main/utils/loopback.ts`, which rewrites `localhost` to `127.0.0.1`, because some Windows hosts resolve `localhost` to IPv6 only and local engines bind IPv4.
- Meeting intelligence, RAG, voice conversation (local + Realtime), captions, file transcription: implemented.
- **Key bindings: implemented and verified on Windows.** Every global shortcut now comes from one contract (`@d3ro/core/keybinding`) with multiple bindings per action, mouse-button support, and no hardcoded accelerators left in `bootstrap.ts`. A manual run on 2026-09-21 confirmed legacy migration (custom values preserved), 6 actions loaded, the uiohook keyboard **and** mouse hook active with zero boot errors, and multi-binding working; contract side is `packages/core` 117 tests GREEN with no type errors in the key-binding files (`11` GAP-KEY-01 `[x]`). Two things remain open: `KeyBindingService` has no unit test of its own, and macOS/Linux mouse behavior is unconfirmed (`11` GAP-KEY-02). The rewrite also fixed a dead hands-free double-press path, an order-dependent reserved-combo check, a `globalShortcut.unregisterAll()` that wiped the popup accelerators, and a `setEnabled(true)` that re-enabled hooking with an empty binding set.
- The same pass fixed an unrelated pre-existing dashboard bug: `caption.onStateChanged` delivers `{ state }`, but `DashboardPage` passed the whole object into `setCaptionState`, so the caption status readout never showed the right value (`DashboardPage.tsx:148`).
- **Ad mediation**: `DirectHouseSponsorAdapter` performs real configurable REST bids; the other 9 adapters remain fail-closed stubs pending official SDKs (see `11-gap-backlog.md` GAP-ADS-01/02).
- Tier resolution now routes through `@d3ro/core/entitlement` (`resolveEntitlement`, `normalizeEntitlementTier`); `useLicenseState.isPro` includes `pro_plus`.
- No `TODO`/`FIXME` markers found in `src` (grep clean). `src/main/types/` is an empty directory.
@ -207,6 +213,8 @@ DB schema (`src/main/db/schema.ts`, drizzle SQLite): `history`, `dictionary`, `s
| Bootstrap order | `src/main/bootstrap.ts` |
| IPC registry | `src/main/ipc/index.ts` |
| IPC channel SSOT | `packages/core/src/ipc-channels.ts` |
| Key-binding contract SSOT | `packages/core/src/keybinding.ts` (catalog, actions, validation, conflicts, formatting, parsing) |
| Key-binding service / IPC / UI | `src/main/services/KeyBindingService.ts`, `src/main/ipc/keybinding-handlers.ts`, `src/renderer/components/keybinding/` |
| Preload API | `src/preload/index.ts` |
| Windows | `src/main/windows/WindowManager.ts` |
| Voice orchestrator | `src/main/services/VoiceModeService.ts` |

View file

@ -14,8 +14,8 @@ Status quick-reference: `[x]` done+verified · `[~]` partial/unverified · `[ ]`
| ID | Feature | D | W | M | B | Anchors / notes |
|---|---|---|---|---|---|---|
| CAP-01 | Push-to-talk dictation (hold/release) | [x] | [-] | [x] | [-] | Desktop `VoiceModeService`; mobile RecordScreen via app CTA/notification action (no global hotkey) |
| CAP-02 | Hands-free toggle dictation | [x] | [-] | [x] | [-] | Desktop double-press; mobile toggle |
| CAP-01 | Push-to-talk dictation (hold/release) | [x] | [-] | [x] | [-] | Desktop `VoiceModeService`; the trigger is now the rebindable `dictation` action of CAP-16 (several bindings per action, keyboard or mouse) rather than a single stored shortcut. The pipeline itself is unchanged and tested; the rewritten entry layer was confirmed in the 2026-09-21 manual run (CAP-16). Mobile RecordScreen via app CTA/notification action (no global hotkey) |
| CAP-02 | Hands-free toggle dictation | [x] | [-] | [x] | [-] | Desktop double-press shares the `dictation` binding and is split by the action's `doublePress` flag (`KeyBindingService.ts:680`). This path was **dead in shipped builds**: the previous lookup returned only the first matching action, so with both actions on the same binding double-press never reached hands-free. Fixed and confirmed in the 2026-09-21 manual run (CAP-16); core tests cover the contract side (same binding is not a conflict, `keybinding.test.ts:499`/`:734`). `KeyBindingService` still has no unit test of its own (GAP-KEY-01 evidence). Mobile toggle |
| CAP-03 | Live partial transcript while recording | [x] | [ ] | [ ] | [-] | Desktop `voice:partialTranscript` + recording-tip; producer added in `1.3.0` (`VoiceModeService._runPartial` → `LocalSTTService.transcribePartial`, 1.5 s cadence / 7.5 s window, never inserted). The row was `[x]` before any producer existed. |
| CAP-04 | Recording waveform + level meter | [x] | [x] | [x] | [-] | Desktop 9-bar cos distribution; mobile audio level; the recording-tip popup bundle and its on-disk assets are verified by `scripts/ci/verify-desktop-renderer-bundles.mjs` |
| CAP-05 | Device/mic selection | [x] | [ ] | [~] | [-] | Desktop config; mobile uses system default |
@ -29,6 +29,7 @@ Status quick-reference: `[x]` done+verified · `[~]` partial/unverified · `[ ]`
| CAP-13 | Live captions overlay | [x] | [-] | [-] | [-] | Desktop `CaptionService` + caption-overlay popup; popup assets verified in the packaged build (`scripts/ci/verify-desktop-renderer-bundles.mjs`); GAP-INFRA-05 |
| CAP-14 | Recording persistence / crash recovery | [x] | [ ] | [x] | [-] | Desktop WAV persist; mobile durable queue + process-kill WAV recovery |
| CAP-15 | Android foreground recording service | [-] | [-] | [x] | [-] | Mobile API 34 FGS + persistent notification (SSOT R-005 GREEN) |
| CAP-16 | Rebindable global key bindings (keyboard + mouse) | [x] | [-] | [-] | [-] | Contract SSOT `packages/core/src/keybinding.ts`: `KEY_CATALOG` (10 groups, `:615`), `KEYBINDING_ACTIONS` (6 actions, `:719`), `validateBinding` (`:953`), `detectBindingConflicts` (`:1016`). Multiple bindings per action persist as one `AppConfig.keyBindings` map (`packages/core/src/types.ts:459`), replacing the four singular `*Shortcut` fields; `ConfigService` migrates legacy values once (`ConfigService.ts:142`). `KeyBindingService` hooks keyboard **and** mouse via uiohook (`KeyBindingService.ts:387`) — MB1 is not bindable, MB2/MB3 need a modifier, MB4/MB5 are free, and no mouse button can be suppressed, so the original click still fires (warning surfaced in the UI). Selection is either key-recording or a searchable grouped dropdown (`KeyBindingPicker.tsx:536`). `history-popup`/`command-popup` were hardcoded in `bootstrap.ts` and are now rebindable actions (`bootstrap.ts:159`). **Verified 2026-09-21 on Windows by a manual run** (`%APPDATA%/d3ro-voice/logs/main.log`, 12:53–13:06): `ConfigService` migrated the four legacy shortcuts with the user's non-default values preserved exactly, `KeyBindingService` loaded 6 bindings for 6 actions and started the uiohook keyboard **and** mouse hook with zero boot errors, and keyboard plus mouse (MB4/MB5) bindings were exercised through the UI. A `Loaded 7 key binding(s) … for 6 action(s)` line later in the same session shows multi-binding working end to end. The migrated map was read back from `d3ro-voice-config.json`: legacy `*Shortcut` fields gone, no `displayLabel` left. Contract evidence: `packages/core` 117 tests GREEN, no renderer type errors in the key-binding files. **Still open:** `KeyBindingService` has no unit test of its own, macOS/Linux mouse behavior is unconfirmed (`11` GAP-KEY-02), and `command` still falls back to the dictation pipeline (GAP-KEY-03). W/M `[-]`: no OS-level global binding surface exists there (browser sandbox; mobile has no global hotkey, see CAP-01). B `[-]`: device-local setting, nothing server-side. See `11` GAP-KEY-02/03 (open), GAP-KEY-01 (`[x]`), and `11` §7 CONSTRAINT-I18N-01. |
---
@ -158,9 +159,9 @@ Status quick-reference: `[x]` done+verified · `[~]` partial/unverified · `[ ]`
| ID | Feature | D | W | M | B | Anchors / notes |
|---|---|---|---|---|---|---|
| SHELL-01 | Settings / preferences | [x] | [~] | [x] | [x] | Desktop tabbed modal; web theme/i18n; mobile `SettingsScreen` |
| SHELL-01 | Settings / preferences | [x] | [~] | [x] | [x] | Desktop tabbed modal; the General tab hosts the whole key-binding editor (CAP-16: global on/off switch + one `KeyBindingField` per action, grouped voice/window — `SettingsModal.tsx:239`), which is also the first settings entry point the `command` action ever had; web theme/i18n; mobile `SettingsScreen` |
| SHELL-02 | Theme system (6 themes) | [x] | [x] | [x] | [-] | `theme.ts` SSOT |
| SHELL-03 | i18n (12 locales) | [x] | [x] | [x] | [-] | `@d3ro/i18n`; ko/en fully translated, others partial |
| SHELL-03 | i18n (12 locales) | [x] | [x] | [x] | [-] | `@d3ro/i18n`; ko/en fully translated, others partial. Measured 2026-09-21: `ko` 1716 keys / `en` 1709 / the other ten 327 each, so ~1,380 keys fall back for non-English locales — tracked as `11` GAP-I18N-01 |
| SHELL-04 | Onboarding / first-run | [x] | [ ] | [x] | [-] | Desktop model bootstrap; mobile audience/theme/locale |
| SHELL-05 | Accessibility / reduced motion | [~] | [~] | [~] | [-] | Desktop reduced-motion honored; mobile a11y rows pending |
| SHELL-06 | System tray / background | [x] | [-] | [-] | [-] | Desktop tray |
@ -203,7 +204,7 @@ Status quick-reference: `[x]` done+verified · `[~]` partial/unverified · `[ ]`
| Surface | `[x]` | `[~]` | `[ ]` | Notable strength | Notable weakness |
|---|---|---|---|---|---|
| Desktop | ~40 | 3 | ~8 | Local AI pipeline, meetings, RAG, conversation, hotkeys | Ads stubs, no team admin, no email account |
| Desktop | ~40 | 3 | ~8 | Local AI pipeline, meetings, RAG, conversation, key bindings | Ads stubs, no team admin, no email account |
| Web | ~22 | 6 | ~14 | Server-shared data UX, billing, meetings, teams | No local AI, limited knowledge upload/search |
| Mobile | ~40 | 12 | ~18 | Cloud + native recording, portability, admin, IAP/ads | External store/console gates, a11y, deep E2E pending |
| Backend | ~45 | 6 | ~4 | RLS, Edge functions, billing, fail-closed AI | Payple webhook signature, some external provider keys |

View file

@ -13,6 +13,7 @@ Legend: `[ ]` open · `[~]` in progress · `[!]` blocked externally · `[x]` res
- An item here is **not** a failure. It is a known state with an owner and a next step.
- When you close an item, flip it to `[x]`, add the date + evidence path, and also update `10-feature-catalog.md`.
- Grandfathered detail lives in `docs/v3/MOBILE_APP_COMPLETION_SSOT.md`; this file is the cross-surface roll-up. When the two disagree, the SSOT wins for mobile and must be reconciled here.
- **Not everything imperfect is a gap.** Trade-offs that were reviewed and deliberately kept live in §7 as constraints, not in §1. Check §7 before opening a row for one.
---
@ -58,6 +59,13 @@ Legend: `[ ]` open · `[~]` in progress · `[!]` blocked externally · `[x]` res
| GAP-REL-04 | Release | canonical feed는 Cloudflare 뒤에 있어 업로드 본문이 **100MiB**를 넘으면 HTTP 413으로 거부한다. | `scripts/ci/publish-updater-release.mjs`, `docs/deployment/unsigned-distribution.md` | `[~]` 2026-09-18: 설치본을 90.6MiB로 줄여 업데이트 피드 게시를 복구했다(GAP-STT-07). 휴대용/Scoop 채널은 여전히 95MiB 분할이 필요하다. |
| GAP-STT-07 | Local STT | 진(사이드카)을 앱 번들에 넣으면 설치본이 100MiB를 넘고 매 업데이트마다 162MiB를 다시 받는다. | `apps/desktop/src/main/services/RuntimeProvisioner.ts`, `apps/desktop/electron-builder.yml` | `[x]` 2026-09-18: 설치본에서 엔진/ffmpeg를 제거하고 처음 필요할 때 `runtime-latest`에서 내려받는다(부품별 + 결합본 SHA-256 검증). 설치본 189MB → 90.6MiB, 런타임 1회 116MiB(엔진 94.4 + ffmpeg 21.7). 실제 feed로 통합 검증 완료. |
| GAP-STT-06 | Local STT | Decode settings were untuned: previous-text conditioning let repeated hallucinations compound, and no VAD parameters meant slow, uneven segments. | `apps/desktop/sidecar/main.py` | `[x]` 2026-09-18: `condition_on_previous_text=false`, bounded low-temperature fallback, `no_speech`/`compression_ratio`/`log_prob` thresholds, 300 ms silence trimming. Same transcript, ~5x faster on the reference machine (7.7 s audio: 1609 ms → 303 ms). |
| GAP-KEY-01 | Key bindings | 키바인딩 전면 개편(CAP-16)이 실앱 구동으로 검증되지 않은 상태였다. uiohook 전역 후킹, 더블프레스 타이밍, 마우스 버튼 수신을 실행 중인 Electron 에서 확인한 적이 없었고, 증거가 계약 수준(`packages/core` 117개 통과, 키바인딩 파일 타입 에러 0건)뿐이었다. 에이전트는 데스크톱 GUI 를 띄울 수 없다(`AGENTS.md` §3). | `apps/desktop/src/main/services/KeyBindingService.ts`, `packages/core/__tests__/keybinding.test.ts`, `%APPDATA%/d3ro-voice/logs/main.log`, `%APPDATA%/d3ro-voice/d3ro-voice-config.json` | `[x]` 2026-09-21: 사용자가 Windows 에서 앱을 띄워 검증 완료. 로그(12:53–13:06)에 `Migrated 4 legacy shortcut(s) to keyBindings` → `Loaded 6 key binding(s) from config for 6 action(s)` → `uiohook started, global keyboard/mouse hook active` → `key-bindings initialized` → `Key binding events connected` 가 순서대로 남았고 부팅 에러 0건, STT 모델 로딩까지 정상. 키보드·마우스(MB4/MB5) 바인딩을 UI 로 실제 조작해 동작을 확인했고, 같은 세션의 `Loaded 7 key binding(s) … for 6 action(s)` 가 다중 바인딩이 실동작함을 보인다. 설정 파일 되읽기로 마이그레이션 결과를 확인 — 사용자 커스텀 값이 그대로 보존됐고(dictation `code:49,alt` / hands-free `code:49,alt+shift` / command `code:165,ctrl` / caption `code:49,ctrl+alt+shift`, 팝업 2종은 신규 기본값), 구 `*Shortcut` 4개와 `displayLabel` 은 모두 사라졌다. **남은 것**: `KeyBindingService` 자체의 유닛 테스트는 여전히 없다(실행 검증이 유닛 테스트를 대체하지 않는다). macOS/Linux 는 GAP-KEY-02 로 계속 열려 있다. |
| GAP-KEY-02 | Key bindings | 마우스 버튼 지원이 **Windows 기준으로만** 설계·확인됐다. `KeyBindingService` 에는 마우스 관련 플랫폼 분기가 없고(`process.platform` 은 meta 수정자 라벨 표기에만 쓰인다), macOS/Linux 에서 uiohook 이 보고하는 X1/X2 버튼 번호와 OS 기본 "뒤로/앞으로" 동작과의 간섭은 확인하지 않았다. 마우스 이벤트는 suppress 가 불가능하므로 원래 동작이 항상 함께 실행된다. | `KeyBindingService.ts:235`(`readMouseButton`), `:301`(meta 라벨 분기), `packages/core/src/keybinding.ts:561-612`(마우스 카탈로그 5종). 2026-09-21 실앱 검증(GAP-KEY-01)은 **Windows 에서만** 이뤄졌고 거기서는 MB4/MB5 가 정상 동작했다. | macOS/Linux 에서 MB2~MB5 수신 여부와 버튼 번호 매핑을 확인하고, 다르면 카탈로그를 플랫폼별로 분기한다. |
| GAP-KEY-03 | Key bindings | `command` 액션에 전용 핸들러가 없다. 이번에 처음으로 설정 UI 에 노출됐지만, 트리거되면 dictation 파이프라인으로 fallback 하며 `KEYBINDING_ACTIONS` 의 `holdMode:false` 대신 dictation 과 같은 hold-to-talk 로 강제된다. 개편 이전부터 같은 동작이었고 이번 작업은 그 사실을 코드에 명시화만 했다(기능 변화 없음). | `apps/desktop/src/main/services/VoiceModeService.ts:1071`(`_resolveHoldMode`), `packages/core/src/keybinding.ts:740`(액션 정의) | `command` 전용 동작을 정의하고 `_resolveHoldMode` 의 예외를 제거하거나, 액션을 카탈로그에서 뺀다. |
| GAP-QA-02 | Quality | 캡션 테스트 2건이 **개발 머신에 사이드카 venv 가 있는지에 따라 결과가 갈린다**. `LocalSTTService.initialize()`(`:239`) → `_ensureSidecarRunning()`(`:583`) → `_spawnSidecar()`(`:650`) → `_waitForHealth()`(`:794`) 경로에서 venv 가 존재하면 실제 Python 프로세스를 띄우고 health 폴링이 vitest 기본 타임아웃 10초를 넘긴다. venv 가 없으면 `getSidecarCommand()`(`apps/desktop/src/main/utils/paths.ts:174`)가 즉시 throw 해서 같은 테스트가 빠르게 통과한다. 테스트가 로컬 환경을 격리하지 못한 것이 결함이다. | `tests/red/ipc-surfaces.usecase.test.ts`(`캡션 시작 실패는 success:false 로 나온다`), `tests/red/silent-errors.usecase.test.ts:48`. **키바인딩 개편의 회귀가 아니다** — 2026-09-21 에 HEAD(`0ca9e24`) 무수정 코드를 같은 환경(venv 연결)에서 돌려 동일하게 재현했다. 같은 날 같은 머신에서도 실행 방식에 따라 결과가 갈렸다: 전체 실행은 `3 failed / 1311 passed (1314)`(`rag.usecase` + `silent-errors` 캡션 + `paths.test`)이고 `ipc-surfaces` 캡션 케이스는 통과했는데, 그 파일만 단독 실행하면 같은 케이스가 10초 타임아웃으로 실패한다. 테스트 총수 1314 는 어느 실행에서나 같고, 새로 깨진 테스트는 0건이다. | 사이드카 기동을 테스트 경계에서 주입·모킹해 환경 의존을 끊는다. 함께 실패하는 `rag.usecase`(임베딩 서버 부재)도 같은 성격이다. `tests/main/utils/paths.test.ts:78` 은 성격이 다르다 — 기대 정규식이 `사이드카를 찾을 수 없습니다` 인데 실제 메시지는 `로컬 음성 엔진이 아직 설치되지 않았습니다…` 로 바뀌어 테스트가 문구를 따라가지 못한 것이다. |
| GAP-I18N-01 | i18n | 로케일별 키 수가 크게 어긋난다. 2026-09-21 실측: `ko` 1716 / `en` 1709 / 나머지 10개 로케일 각 327. `keybinding.*` 55개는 12개 로케일 전부에 동일하게 들어갔지만, 그 밖 약 1,380개 키가 비영어 로케일에 없어 폴백 체인(locale → `en` → `ko`)으로 표시된다. 키바인딩 작업 이전부터 있던 부채이며 그 작업 범위 밖이었다. | `packages/i18n/src/locales/*.json`, 카탈로그 SHELL-03 | 로케일 간 키 diff 를 내는 커버리지 게이트를 만들어 회귀를 막고, 누락 키를 채운다. |
| GAP-I18N-02 | i18n | 렌더러가 `ko.json` 에 없는 `license.*` 키를 쓴다. `TranslationKey` 가 `ko.json` 에서 파생되므로 누락은 타입 에러로 드러난다. 타입 에러로만 끝나지 않는다 — 폴백 체인이 `locale → en → ko → 키 문자열` 이므로 마스터 로케일에도 없으면 **`license.team` 같은 키가 화면에 그대로 노출된다**. 2026-09-21 실측: `license.feature.premium_llm`·`license.team`·`license.enterprise` 가 없고 이로 인한 TS2345 가 4건이다. HEAD 에서도 없던 키이므로 선재 결함이며 키바인딩 작업과 무관하다. | `apps/desktop/src/renderer/components/UpgradePromptModal.tsx:47`·`:192`, `apps/desktop/src/renderer/pages/DashboardPage.tsx:481`·`:529`, `packages/i18n/src/locales/ko.json` | 세 키를 `ko.json` 에 추가하고 12개 로케일에 반영한다. 같은 타입체크에 잡히는 `LicenseTab.tsx`(6건)·`LicenseModal.tsx`(2건)는 원인이 다르다 — `TFunction` 을 `(k: string) => string` 에 넘기는 TS2322 4건과 `currentTier` 미정의 TS2304 2건으로, 후자는 컴파일이 깨지는 별개 결함이다(GAP-INFRA-04 범위). |
| GAP-INFRA-06 | Dev env | `better-sqlite3` 네이티브 ABI 가 **앱 실행과 로컬 테스트에서 서로 다른 값을 요구**한다. Electron 33 은 ABI 130, 호스트 Node 23 은 ABI 131 이라 한쪽에 맞추면 다른 쪽이 깨진다. 2026-09-21 실측: `electron-rebuild -f -w better-sqlite3` 직후 vitest 가 `366 failed / 948 passed` 로 무너졌고, 리빌드 전에는 `1311 passed` 였다. 같은 날 확인한 현재 워크스페이스는 Node ABI 쪽(호스트 `node -e "require('better-sqlite3')"` 성공)이라 테스트는 돌고 앱 실행에는 재리빌드가 필요하다. **배포 차단 이슈가 아니다** — `node_modules/` 는 gitignore(`.gitignore:1`)이고 패키징 경로는 `scripts/ci/verify-native-abi.mjs` 가 이미 막는다(GAP-REL-07 `[x]`). 순수하게 로컬 개발 환경 전환 비용 문제다. | `scripts/ci/verify-native-abi.mjs`, `scripts/ci/fix-native-abi.mjs`, `package.json`(현재 리빌드용 스크립트 없음) | 두 ABI 를 오가는 npm 스크립트를 둔다(예: `rebuild:app` = Electron ABI, `rebuild:test` = Node ABI). 지금은 전환 방법이 문서화도 스크립트화도 되어 있지 않아 매번 수동으로 알아내야 한다. |
| GAP-STT-08 | Local STT | 1.3.5 설치본에서 엔진 설치가 "런타임 아카이브 해시 불일치 (sidecar)"로 항상 실패했다. 부품 검증은 **메모리 스트림**에서 센 값으로, 결합 검증은 **디스크 파일**에서 계산해 기준이 서로 달랐다. 디스크 쓰기가 잘려도 부품 검사를 통과하고 결합 단계에서만 터지므로 원인 파악도 불가능했다. 재시도가 없어 전송이 한 번 끊기면 곧바로 설치 실패였다. | `apps/desktop/src/main/services/RuntimeProvisioner.ts` | `[x]` 2026-09-18: 부품 크기·해시를 디스크 파일 기준으로 통일하고, 결합본은 크기를 먼저 검사한 뒤 해시를 본다(오류 메시지에 실제/기대값 포함). 부품 다운로드는 실패 시 해당 파일을 지우고 최대 3회 재시도한다. 서버 아티팩트는 무결함을 확인했고(부품 2개 해시 일치, 결합본 `e203aa53…` = 인덱스 기대값), 실제 feed로 설치를 재현해 18초 만에 성공. **1.3.6으로 게시 완료** — `latest.yml`이 1.3.6/90.6MiB를 서빙하고 설치본 sha512가 피드 메타데이터와 일치. 설치본 asar에 수정 코드가 포함되고 구버전 `archiveHash` 경로는 제거됨을 확인. |
| GAP-INFRA-05 | Build | 패키징된 렌더러 팝업 스크립트가 번들에 없었다. 팝업 HTML이 classic `<script src="./script.js">`를 참조해 Vite가 처리하지 않았고, dev에서는 로드되지만 설치본에는 파일이 없었다. 그래서 녹음 오버레이가 0:00에서 멈추고 웨이브 바가 뜨지 않았으며 실시간 자막이 렌더되지 않았다. 로드 전 `webContents.send`가 조용히 버려지는 문제와 `hide()` 이후 재표시의 z-order/repaint 유실도 함께 있었다. | `apps/desktop/src/renderer/popups/*/index.html`, `apps/desktop/src/main/windows/WindowManager.ts`, `scripts/ci/verify-desktop-renderer-bundles.mjs` | `[x]` 2026-09-19: 팝업 5종을 `type="module"`로 전환해 Vite가 해시된 번들로 방출하도록 고쳤고, 빌드 HTML이 참조하는 모든 로컬 asset이 디스크에 있는지 검사하는 `verify-desktop-renderer-bundles.mjs`(+ self-test)를 `.forgejo`/`.github` 패키징 파이프라인에 연결했다. WindowManager는 렌더러 준비 전 IPC를 `did-finish-load`까지 보관하고, 팝업을 표시할 때마다 topmost 재선언 + 강제 repaint를 수행하며, 팝업 렌더러 콘솔/로드 실패를 main 로그로 승격한다. |
@ -119,6 +127,11 @@ These are the mobile SSOT rows still `[ ]` / `[~]`. Do not duplicate the full te
- **Desktop local dictation / LLM / history / meetings / RAG / conversation:** local
dictation works in dev **and** in packaged builds as of `1.3.0` (engine bundled, paths
fixed, IPv4 loopback). Ads: one real adapter, rest stubs.
- **Desktop key bindings (CAP-16):** rewritten onto one SSOT with multiple bindings per
action and mouse-button support; **verified on Windows** by a manual run on 2026-09-21
(legacy migration, 6 actions loaded, keyboard + mouse hook live, multi-binding exercised
— GAP-KEY-01 `[x]`). macOS/Linux mouse behavior is still unconfirmed (GAP-KEY-02) and
the service has no unit test of its own.
- **Web console:** yes, feature-complete for server-shared data; knowledge upload/search and team feed implemented.
- **Mobile:** code complete for most flows and tested locally; blocked mainly by external store/console gates, plus a11y and some E2E depth.
- **Backend:** fail-closed AI proxies, RLS, billing, ads SSV implemented; push transports (FCM + webpush + APNs) and cron drain implemented.
@ -155,3 +168,14 @@ Actionable checklist for the work started this session. Fields to fill are blank
**Docs**
- [ ] Keep `docs/deployment/push-transport-without-firebase.md` and this file in sync when transports or clients change.
---
## 7. 알려진 설계 제약 (수용됨 — 결함 아님)
여기 있는 항목은 고쳐야 할 갭이 아니라 **대안을 검토한 뒤 의도적으로 유지하기로 한 절충**이다.
§1 에 갭으로 재등록하지 마라.
| ID | 제약 | 왜 이대로 두는가 | 완화 장치 |
|---|---|---|---|
| CONSTRAINT-I18N-01 | `packages/core/src/keybinding.ts` 는 i18n 키를 평범한 `string` 으로 노출한다. 렌더러가 `asTranslationKey()`(`apps/desktop/src/renderer/components/keybinding/translation-key.ts:7`)로 경계에서 캐스팅하므로, 존재하지 않는 키를 넘겨도 컴파일러가 잡지 못한다. | core 가 로케일 패키지에 의존하지 않게 하려는 의도적 설계다. 검토한 대안 둘 다 성립하지 않는다 — (A) 키 필드를 리터럴 유니온으로 좁히는 방식은 `KEY_CATALOG` 가 `letterEntries()` 같은 함수 생성부를 포함해 불가능하고, (B) core 가 `@d3ro/i18n` 의 타입 가드를 쓰는 방식은 의존 방향을 core → i18n 으로 역전시켜 `03-shared-packages.md` §6 의 전제를 깬다. 2026-09-21 결정: 현행 유지. | `packages/core/__tests__/keybinding-i18n.test.ts` (14 케이스). core 가 참조하는 키가 12개 로케일 전부에 있는지, 값이 빈 문자열이 아닌지, core 가 렌더러 전용 `keybinding.ui.*` 를 참조하지 않는지 검사한다. 거부 사유 키는 하드코딩 목록이 아니라 실제 `validateBinding` 경로를 태워 수집하므로 새 사유가 생기면 자동으로 커버된다. |