d3ro-voice/docs/map/10-feature-catalog.md
Yun Chan 4ad1ae6ed4 feat(keybinding): several shortcuts per action, mouse buttons, searchable picker
Shortcuts were defined in four places that drifted apart: per-action IPC channel
pairs, a hand-written VK table in the service, a second one in the renderer, and
three copies of the keycap styling. Adding an action meant editing all of them,
so two shortcuts stayed hardcoded in bootstrap and one had no settings entry at
all.

packages/core/src/keybinding.ts is now the single source for the binding type,
the selectable key catalog, the action catalog, normalization, validation,
conflict detection, display labels, search and deserialization. Main, preload
and renderer all read from it; nothing redefines keys or rules locally.

- Each action holds a list of bindings instead of one. AppConfig's four
  *Shortcut fields collapse into a single keyBindings map, migrated on launch.
- Mouse buttons can be bound. Left click is refused, right/middle need a
  modifier, side buttons are free. uiohook cannot swallow events, so the
  original click still fires and the UI says so.
- Keys can be picked from a grouped dropdown with a search box, not only by
  recording a keypress.
- HOTKEY's 14 channels become KEYBINDING's 9, taking the action as a parameter,
  so actions no longer multiply channels. The history and command popups moved
  out of bootstrap into ordinary actions.
- displayLabel is gone; labels derive from the binding and follow the app
  language and platform.

Fixes found on the way:
- Double-press hands-free was unreachable: lookup returned only the first
  matching action, and dictation shares its default binding.
- Reserved-combination checks compared joined key names, so a different modifier
  order let Ctrl+C through.
- Disabling shortcuts released every global registration in the process,
  including the popup ones, and never restored them.
- Enabling shortcuts after starting disabled left nothing registered.
- The dashboard stored the caption event payload instead of the state in it.
2026-09-21 13:41:47 +09:00

22 KiB
Raw Blame History

10 — Feature Catalog (Feature Map)

The canonical feature map. One row = one user-facing capability. Status is per surface. Legend in 00-index.md §2. Surfaces: D = desktop (apps/desktop), W = web (apps/web), M = mobile (apps/mobile-rn), B = backend (Supabase/.NET).

How to use an ID: cite it in commits, plans, and backlog. Example: "CAP-03 is [~] on mobile (external OAuth pending)".

Status quick-reference: [x] done+verified · [~] partial/unverified · [ ] planned/absent · [!] blocked externally · [-] N/A.


CAP — Capture & Transcribe

ID Feature D W M B Anchors / notes
CAP-01 Push-to-talk dictation (hold/release) [x] [-] [x] [-] Desktop VoiceModeService; the trigger is now the rebindable dictation action of CAP-16 (several bindings per action, keyboard or mouse) rather than a single stored shortcut. The pipeline itself is unchanged and tested; the rewritten entry layer was confirmed in the 2026-09-21 manual run (CAP-16). Mobile RecordScreen via app CTA/notification action (no global hotkey)
CAP-02 Hands-free toggle dictation [x] [-] [x] [-] Desktop double-press shares the dictation binding and is split by the action's doublePress flag (KeyBindingService.ts:680). This path was dead in shipped builds: the previous lookup returned only the first matching action, so with both actions on the same binding double-press never reached hands-free. Fixed and confirmed in the 2026-09-21 manual run (CAP-16); core tests cover the contract side (same binding is not a conflict, keybinding.test.ts:499/:734). KeyBindingService still has no unit test of its own (GAP-KEY-01 evidence). Mobile toggle
CAP-03 Live partial transcript while recording [x] [ ] [ ] [-] Desktop voice:partialTranscript + recording-tip; producer added in 1.3.0 (VoiceModeService._runPartial → LocalSTTService.transcribePartial, 1.5 s cadence / 7.5 s window, never inserted). The row was [x] before any producer existed.
CAP-04 Recording waveform + level meter [x] [x] [x] [-] Desktop 9-bar cos distribution; mobile audio level; the recording-tip popup bundle and its on-disk assets are verified by scripts/ci/verify-desktop-renderer-bundles.mjs
CAP-05 Device/mic selection [x] [ ] [~] [-] Desktop config; mobile uses system default
CAP-06 System/loopback audio capture [x] [-] [ ] [-] Desktop only (caption source); mobile policy-limited
CAP-07 Local Whisper STT [x] [-] [x] [-] Desktop ships the faster-whisper sidecar (resources/sidecar, built by sidecar:build, verified by scripts/ci/verify-sidecar-bundle.mjs), warms it up at app start, and connects over IPv4 loopback; mobile on-device Whisper (supported devices)
CAP-08 Cloud STT (multi-provider) [x] [x] [x] [x] Desktop 6 providers + D3RO Cloud; web/mobile via stt-proxy; .NET internal gateway
CAP-09 STT auto-fallback + fail-closed [x] [x] [x] [x] STTManager; SSOT R-021/R-022 GREEN
CAP-10 STT model download/management UI [x] [-] [~] [-] Desktop model manager + onboarding; mobile bundled model
CAP-11 File transcription (audio/video) [x] [ ] [x] [~] Desktop ffmpeg chunking; mobile import picker; web deferred
CAP-12 Audio import from other apps (share intent) [-] [-] [x] [-] Mobile Android ACTION_SEND/ACTION_VIEW (SSOT R-016 GREEN)
CAP-13 Live captions overlay [x] [-] [-] [-] Desktop CaptionService + caption-overlay popup; popup assets verified in the packaged build (scripts/ci/verify-desktop-renderer-bundles.mjs); GAP-INFRA-05
CAP-14 Recording persistence / crash recovery [x] [ ] [x] [-] Desktop WAV persist; mobile durable queue + process-kill WAV recovery
CAP-15 Android foreground recording service [-] [-] [x] [-] Mobile API 34 FGS + persistent notification (SSOT R-005 GREEN)
CAP-16 Rebindable global key bindings (keyboard + mouse) [x] [-] [-] [-] Contract SSOT packages/core/src/keybinding.ts: KEY_CATALOG (10 groups, :615), KEYBINDING_ACTIONS (6 actions, :719), validateBinding (:953), detectBindingConflicts (:1016). Multiple bindings per action persist as one AppConfig.keyBindings map (packages/core/src/types.ts:459), replacing the four singular *Shortcut fields; ConfigService migrates legacy values once (ConfigService.ts:142). KeyBindingService hooks keyboard and mouse via uiohook (KeyBindingService.ts:387) — MB1 is not bindable, MB2/MB3 need a modifier, MB4/MB5 are free, and no mouse button can be suppressed, so the original click still fires (warning surfaced in the UI). Selection is either key-recording or a searchable grouped dropdown (KeyBindingPicker.tsx:536). history-popup/command-popup were hardcoded in bootstrap.ts and are now rebindable actions (bootstrap.ts:159). Verified 2026-09-21 on Windows by a manual run (%APPDATA%/d3ro-voice/logs/main.log, 12:53–13:06): ConfigService migrated the four legacy shortcuts with the user's non-default values preserved exactly, KeyBindingService loaded 6 bindings for 6 actions and started the uiohook keyboard and mouse hook with zero boot errors, and keyboard plus mouse (MB4/MB5) bindings were exercised through the UI. A Loaded 7 key binding(s) … for 6 action(s) line later in the same session shows multi-binding working end to end. The migrated map was read back from d3ro-voice-config.json: legacy *Shortcut fields gone, no displayLabel left. Contract evidence: packages/core 117 tests GREEN, no renderer type errors in the key-binding files. Still open: KeyBindingService has no unit test of its own, macOS/Linux mouse behavior is unconfirmed (11 GAP-KEY-02), and command still falls back to the dictation pipeline (GAP-KEY-03). W/M [-]: no OS-level global binding surface exists there (browser sandbox; mobile has no global hotkey, see CAP-01). B [-]: device-local setting, nothing server-side. See 11 GAP-KEY-02/03 (open), GAP-KEY-01 ([x]), and 11 §7 CONSTRAINT-I18N-01.

AI — AI Processing

ID Feature D W M B Anchors / notes
AI-01 Local LLM (Ollama) [x] [-] [ ] [-] Desktop bundled Ollama
AI-02 Cloud LLM (Claude/OpenAI) [x] [x] [x] [x] Desktop PremiumLLMService; web/mobile via llm-proxy; .NET LlmProxyService
AI-03 Auto Polish (cleanup/filler removal) [x] [~] [~] [x] Desktop built-in; web/mobile via commands
AI-04 Translate / summarize / rephrase [x] [x] [x] [x] Built-in instructions
AI-05 Custom instructions (user commands) [x] [x] [x] [x] Desktop CommandsPage (Red Team RT-03 verified); web commands; mobile CommandsScreen
AI-06 Voice keyword commands [x] [-] [ ] [-] Desktop VoiceCommandService + command popup
AI-07 LLM Chains (multi-step pipelines) [x] [ ] [ ] [-] Desktop ChainService
AI-08 Screen/context capture for prompts [x] [-] [ ] [-] Desktop ScreenContextService
AI-09 Streaming responses [x] [x] [x] [x] SSE/NDJSON streaming
AI-10 Dictation templates (voice form fill) [x] [ ] [x] [~] Desktop DictationTemplateService; mobile TemplatesScreen

MEM — Memory & Knowledge

ID Feature D W M B Anchors / notes
MEM-01 History list + search [x] [x] [x] [x] Desktop SQLite; web/mobile Supabase
MEM-02 History detail + edit [x] [x] [x] [x] Mobile HistoryDetailScreen
MEM-03 History favorites [x] [x] [x] [x]
MEM-04 History export / share [x] [~] [x] [x] Desktop export; web limited; mobile share sheet
MEM-05 History audio playback [x] [x] [x] [x] Signed URLs on web/mobile
MEM-06 Dictionary (custom vocabulary) [x] [x] [x] [x] All surfaces CRUD; Desktop Red Team RT-02 & RT-18 fuzzed/verified
MEM-07 Dictionary import/export [x] [x] [x] [-] Desktop dictionary:import/export JSON+CSV (file dialogs, DictionaryService); web serializeDictionary/importDictionaryFile + header buttons; mobile CSV/TXT export + CSV/JSON/TXT import via data-portability
MEM-08 Memos (tags over history) [x] [ ] [x] [x] Desktop MemoService; mobile MemosScreen; web none
MEM-09 Knowledge base / RAG add+index [x] [x] [x] [x] Desktop local RAG (DEF-008 infinite chunking loop resolved, RT-08 verified); web/mobile cloud RAG
MEM-10 Semantic search over knowledge [x] [x] [x] [x] Web KnowledgeSearch calls search-knowledge (was mislabeled deferred); mobile + Edge search-knowledge
MEM-11 Knowledge file upload [x] [x] [x] [x] Web .txt/.md picker + newline-aware chunking + embed-chunks; desktop txt/md/pdf/docx; mobile file picker
MEM-12 Voice actions (OS automation) [x] [x] [x] [x] Desktop VoiceActionService; web ActionRunner (simulated); mobile ActionsScreen
MEM-13 Cross-surface data sync [~] [x] [x] [x] Desktop Supabase sync (V2-4); web/mobile native
MEM-14 Memo tag search [x] [ ] [x] [x]

MTG — Meetings

ID Feature D W M B Anchors / notes
MTG-01 Meeting recording + live transcript [x] [~] [x] [x] Desktop MeetingModeService; web realtime view; mobile
MTG-02 Meeting creation (title/attendees/language/template) [x] [x] [x] [x] SSOT H-006 GREEN
MTG-03 Timestamped memos during meeting [x] [x] [x] [x]
MTG-04 AI summary generation [x] [x] [~] [x] Desktop MeetingSummaryService; web generate-document-button; mobile via Edge
MTG-05 Document generation (minutes/report/idea-note/mindmap) [x] [x] [x] [x] Edge generate-meeting-document; SSOT F-011 GREEN
MTG-06 Document edit (Markdown) [x] [x] [x] [x] Desktop editor; web document-editor
MTG-07 Export PDF/DOCX/TXT/Markdown [x] [~] [x] [x] Desktop ExportMenu; web markdown; mobile print/DOCX chooser (SSOT data portability GREEN)
MTG-08 Speaker diarization [~] [ ] [ ] [ ] Desktop phase-15.5 (LLM estimate + pyannote prep); mobile SSOT H-014 pending
MTG-09 Audio seek ↔ transcript timestamp [ ] [ ] [ ] [ ] SSOT H-011 pending
MTG-10 Meeting list search/filter/sort [~] [ ] [~] [x] Basic lists; advanced filters pending
MTG-11 Meeting AI chat over transcript [x] [ ] [ ] [x] Desktop MeetingChatPanel + MEETING_CHAT
MTG-12 Content reporting for generated docs [-] [ ] [x] [x] content-report Edge + generation receipts

CV — Conversation

ID Feature D W M B Anchors / notes
CV-01 Local duplex voice conversation (STT→LLM→TTS) [x] [ ] [x] [-] Desktop VoiceConversationService; mobile Talk
CV-02 Realtime voice (OpenAI gpt-realtime, Premium) [x] [ ] [ ] [x] Desktop useRealtimeConversation + realtime-token Edge
CV-03 Text AI chat [x] [x] [x] [x] Desktop chat, web chat-panel, mobile TalkScreen
CV-04 TTS playback + controls [x] [ ] [x] [-] Desktop SAPI/say; mobile Android TTS
CV-05 Voice selection / backend selection [x] [ ] [~] [-] Desktop settings conversationBackend

ACC — Accounts, Sync, Devices, Portability

ID Feature D W M B Anchors / notes
ACC-01 Email sign-up/login [ ] [x] [x] [x] Desktop uses separate online-auth; SSOT A-001..A-005 [ ] rows are mobile checklist granularity
ACC-02 OAuth Google [~] [x] [~] [x] Mobile full consent→callback pending (external)
ACC-03 OAuth GitHub / Apple [~] [~] [~] [~] GitHub/Apple provider secrets pending
ACC-04 Password reset / recovery deep link [ ] [ ] [x] [x] Mobile ForgotPassword/UpdatePassword
ACC-05 Account profile / identity management [~] [~] [x] [x] Desktop LicenseTab; mobile AccountScreen
ACC-06 Logout + local sensitive purge [x] [x] [x] [x] Mobile central purge GREEN
ACC-07 Account deletion (server cascade + local purge) [ ] [ ] [~] [x] account-delete Edge; mobile device E2E pending
ACC-08 Cloud sync (per-user data) [x] [x] [x] [x] Desktop CloudSyncService; SSOT D-* largely [ ] granular
ACC-09 Device registration + revocation [-] [ ] [x] [x] Mobile DevicesScreen
ACC-10 Offline queue + retry [~] [ ] [x] [x] Mobile durable queue
ACC-11 Data export/import (portability) [~] [ ] [x] [x] Mobile canonical JSON E2E GREEN; desktop has export files
ACC-12 Notification / push [ ] [~] [~] [x] Backend transports for FCM + Web Push (VAPID) + APNs (.p8) + outbox cron drain implemented. Web/mobile client registration for webpush/apns still pending; Android delivery needs FCM project.

TEAM — Teams & Admin

ID Feature D W M B Anchors / notes
TEAM-01 Team create / rename / delete [-] [x] [x] [x] Desktop N/A
TEAM-02 Invite by email / link + accept deep link [-] [x] [x] [x] team-invite/team-accept
TEAM-03 Members + roles (admin/member/viewer) + leave [-] [x] [x] [x]
TEAM-04 Team meetings/docs sharing + RLS isolation [-] [x] [~] [x] Cross-user isolation tested
TEAM-05 Team comments / activity feed [-] [x] [x] [x] team_activities migration + create_team_activity RPC + realtime; web ActivityFeed, mobile TeamDetail activity card (2026-09-13)
TEAM-06 Admin back office (users/subs/models/usage/audit) [-] [-] [x] [x] apps/admin + mobile AdminScreen
TEAM-07 Role-based destructive action confirm + audit [-] [x] [x] [x] SSOT T-006..T-008 GREEN
TEAM-08 Desktop admin surface [ ] [-] [-] [-] None; N/A by design

MON — Monetization & Ads

ID Feature D W M B Anchors / notes
MON-01 Tier gating (Free/Pro/Pro+/Team/Enterprise) [x] [x] [x] [x] LicenseService, entitlement provider
MON-02 Usage quotas (daily_usage) [x] [x] [x] [x]
MON-03 Desktop offline license (Ed25519) [x] [-] [-] [x] crypto-license + admin issuer
MON-04 Web checkout (Stripe) [-] [x] [~] [x] Stripe checkout/portal/webhook
MON-05 Web checkout (Payple) [-] [x] [~] [~] Payple checkout/manage/renew/webhook; webhook signature pending
MON-06 Paywall / upgrade prompts [x] [x] [x] [x] UpgradePromptModal, ProPaywallScreen
MON-07 Mobile IAP purchase + restore [-] [-] [~] [x] iap-verify + billing-context; live store E2E blocked
MON-08 Billing catalog / pricing display [x] [x] [x] [x] Server catalog SSOT; hardcoded prices removed
MON-09 Free-tier banner ads [~] [-] [x] [x] Desktop adapters fail-closed; mobile AdMob test GREEN, prod serving blocked
MON-10 Rewarded ads → quota credits [~] [-] [x] [x] Desktop RewardedQuotaModal (stub adapters); mobile SSV GREEN
MON-11 Ad mediation engine + settlement [~] [-] [~] [x] Engine + settlement built. DirectHouseSponsorAdapter is now a real configurable REST adapter (bid/impression/click/reward via endpointUrl, fail-closed when unconfigured, unit-tested). Other 9 networks remain UnavailableAdAdapter stubs pending official SDKs.
MON-12 Subscription management (portal/store) [-] [x] [x] [x] Stripe portal / Payple manage / Play manage

SHELL — Platform Shell, Settings, Onboarding, Support

ID Feature D W M B Anchors / notes
SHELL-01 Settings / preferences [x] [~] [x] [x] Desktop tabbed modal; the General tab hosts the whole key-binding editor (CAP-16: global on/off switch + one KeyBindingField per action, grouped voice/window — SettingsModal.tsx:239), which is also the first settings entry point the command action ever had; web theme/i18n; mobile SettingsScreen
SHELL-02 Theme system (6 themes) [x] [x] [x] [-] theme.ts SSOT
SHELL-03 i18n (12 locales) [x] [x] [x] [-] @d3ro/i18n; ko/en fully translated, others partial. Measured 2026-09-21: ko 1716 keys / en 1709 / the other ten 327 each, so ~1,380 keys fall back for non-English locales — tracked as 11 GAP-I18N-01
SHELL-04 Onboarding / first-run [x] [ ] [x] [-] Desktop model bootstrap; mobile audience/theme/locale
SHELL-05 Accessibility / reduced motion [~] [~] [~] [-] Desktop reduced-motion honored; mobile a11y rows pending
SHELL-06 System tray / background [x] [-] [-] [-] Desktop tray
SHELL-07 Auto-launch on login [x] [-] [-] [-] Desktop only
SHELL-08 Auto-update [x] [-] [!] [-] Desktop electron-updater; Forgejo canonical feed; channels + mandatory/major-vs-delta policy (release/update-policy.json); mobile store updates
SHELL-09 Support / diagnostics [x] [ ] [ ] [~] Desktop SupportModal; admin /support stub
SHELL-10 Download center / releases [-] [x] [-] [x] Web /download, admin /releases, Forgejo
SHELL-11 Landing site / legal pages [-] [-] [-] [-] site/ — privacy/terms/delete-account live
SHELL-12 Notifications (in-app / desktop) [x] [ ] [x] [x] Desktop events; mobile push

INFRA — Build, CI, Release, Quality

ID Feature Status Anchors
INFRA-01 Monorepo + workspaces + turbo [x] package.json, turbo.json
INFRA-14 Unified entitlement resolver (@d3ro/core/entitlement) [x] EntitlementSnapshot + resolveEntitlement map Supabase/desktop-license/.NET sources to one contract; desktop isPro fixed, syncFromCloud normalized; mobile/web adoption incremental (see 11 GAP-ID-02)
INFRA-02 Shared packages [x] packages/*
INFRA-03 Desktop build + signed packaging [x] electron-builder.yml, GitLab package-windows/macos
INFRA-04 Mobile CI (debug/E2E/release) [~] .github/workflows/ci.yml, .gitlab-ci.yml; production AAB external
INFRA-05 .NET API tests [x] apps/api-server.Tests
INFRA-06 Edge function tests (Deno) [x] server/supabase/functions, tests/
INFRA-07 E2E desktop (playwright) [x] apps/desktop/tests, apps/desktop/playwright.config.ts
INFRA-08 E2E web (playwright) [x] apps/web/e2e
INFRA-09 E2E mobile (Maestro + instrumentation) [~] .maestro/, androidTest; emulator API 35 gate in CI
INFRA-10 Secret scanning / release boundaries [x] scripts/ci/check-no-hardcoded-secrets.mjs, verify-mobile-release-*.mjs
INFRA-11 Docker + NAS deploy [x] docker-compose.nas.yml, scripts/deploy-nas.*
INFRA-12 Cloudflare edge + tunnel [x] server/cloudflare-worker, Cloudflare Tunnel kd-nas
INFRA-13 Site deploy (Cloudflare Pages + GitHub Pages) [x] .forgejo/workflows/deploy-site.yml, .github/workflows/deploy-site.yml
INFRA-15 Update & release system [x] Canonical Forgejo feed + channels/policy (release/update-policy.json, src/main/update-policy.ts), canonical publisher scripts/ci/publish-forgejo-release.mjs, legacy GitLab mirror; npm run release:metadata:test. v1.1.0 was published to Forgejo on 2026-09-15; product version moved to 1.2.0 as a forward-fix with CI-only publication, a same-version re-release guard, and download centers that link the feed instead of repository paths. 1.3.0 (2026-09-18) carries the local-STT fixes; Windows publication still needs the CI signing secrets (11 GAP-REL-02).
INFRA-16 Desktop STT engine packaging [x] apps/desktop/scripts/setup-sidecar.mjs + build-sidecar.mjs, electron-builder.yml extraResources (sidecar-dist/sidecar → resources/sidecar, resources/ffmpeg), and scripts/ci/verify-sidecar-bundle.mjs run in package-windows/package-macos before electron-builder. Verified on the real bundle: sidecar.exe + _internal including faster_whisper/assets/silero_vad_v6.onnx, plus a packaged-engine transcription round-trip on GPU.
INFRA-17 서명 없는 배포 채널 (portable + Scoop) [x] scripts/ci/build-portable.mjs (95MiB 7z 분할 볼륨 + Scoop 매니페스트), scripts/ci/publish-portable-release.mjs, scripts/local/install-d3ro-voice.ps1, bucket/ 버킷, .forgejo/workflows/portable.yml; 7z 분할 볼륨(Scoop, 162MiB) + zip 분할 부품(수동 설치, 243MiB, 7-Zip 불필요); updater feed와 분리. 2026-09-18 portable-1.3.1 게시 + 실제 설치 검증.
INFRA-18 로컬 런타임 온디맨드 설치 [x] RuntimeProvisioner(부품 다운로드 + SHA-256 검증 + tar 해제, %APPDATA%/d3ro-voice/runtime), POST runtime:ensure / runtime:progress IPC, 설정 > STT 상태/내려받기 UI. 설치본에서 엔진/ffmpeg를 분리해 189MB → 90.6MiB, 업데이트 피드 게시 복구. 검증은 전부 디스크에 기록된 파일 기준이며(부품 크기·해시 → 결합본 크기·해시), 부품 다운로드는 최대 3회 재시도한다. 2026-09-18 실제 feed 통합 검증(엔진 94.4MiB/18초, ffmpeg 21.7MiB/5초).
INFRA-19 네이티브 ABI + updater 설정 게이트 [x] scripts/ci/verify-native-abi.mjs(패키징된 better_sqlite3.node가 Electron ABI인지 호스트 Node 로드 거부로 판별) + scripts/ci/fix-native-abi.mjs(로컬 잠금 우회용 주입). GitLab/Forgejo/GitHub 패키징 단계에 검증 삽입. 2026-09-18: Node ABI 모듈로 앱이 시작 즉시 죽은 사고 + 누락으로 자동 업데이트가 죽은 사고를 함께 방지().

Coverage summary (by surface)

Surface [x] [~] [ ] Notable strength Notable weakness
Desktop ~40 3 ~8 Local AI pipeline, meetings, RAG, conversation, key bindings Ads stubs, no team admin, no email account
Web ~22 6 ~14 Server-shared data UX, billing, meetings, teams No local AI, limited knowledge upload/search
Mobile ~40 12 ~18 Cloud + native recording, portability, admin, IAP/ads External store/console gates, a11y, deep E2E pending
Backend ~45 6 ~4 RLS, Edge functions, billing, fail-closed AI Payple webhook signature, some external provider keys