d3ro-voice/docs/map/10-feature-catalog.md
Yun Chan 72e9b7c18b fix(meeting): stop learned vocabulary from rewriting ordinary Korean
The first cut replaced any Hangul word within one jamo of a learned term.
Measured against the real local corpus (three stored meeting transcripts)
it changed 286 of 3,005 words: 하면 became 라면, 이제 became 이게, 매일 became
메일. Most of what the app learns from typing is ordinary grammar, so
distance alone cannot separate a misheard word from a correct one.

Deterministic correction now covers only dictionary pronunciations and the
Latin identifiers the user types, preserving their casing. Korean terms
still bias recognition through the prompt hint and guide the polish
prompt. Re-measured on the same transcripts: 0 of 3,005 words changed.
Hangul-layout typing artifacts (dkssudgktpdy) are dropped from the hint.
2026-11-06 13:40:09 +09:00

252 lines
86 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# 10 — Feature Catalog (Feature Map)
> The canonical feature map. One row = one user-facing capability.
> Status is per surface. Legend in [`00-index.md`](./00-index.md) §2.
> Surfaces: D = desktop (`apps/desktop`), W = web (`apps/web`), M = mobile (`apps/mobile-rn`), B = backend (Supabase/.NET).
**How to use an ID:** cite it in commits, plans, and backlog. Example: "CAP-03 is `[~]` on mobile (external OAuth pending)".
Status quick-reference: `[x]` done+verified · `[~]` partial/unverified · `[ ]` planned/absent · `[!]` blocked externally · `[-]` N/A.
---
## CAP — Capture & Transcribe
| ID | Feature | D | W | M | B | Anchors / notes |
|---|---|---|---|---|---|---|
| CAP-01 | Push-to-talk dictation (hold/release) | [x] | [-] | [x] | [-] | Desktop `VoiceModeService`; the trigger is now the rebindable `dictation` action of CAP-16 (several bindings per action, keyboard or mouse) rather than a single stored shortcut. The pipeline itself is unchanged and tested; the rewritten entry layer was confirmed in the 2026-09-21 manual run (CAP-16). Mobile RecordScreen via app CTA/notification action (no global hotkey) |
| CAP-02 | Hands-free toggle dictation | [x] | [-] | [x] | [-] | Desktop double-press shares the `dictation` binding and is split by the action's `doublePress` flag (`KeyBindingService.ts:680`). Fixed and confirmed in the 2026-09-21 manual run (CAP-16); unit-tested in `KeyBindingService.test.ts` (11 cases, 2026-10-30, GAP-KEY-01 `[x]`) and `keybinding.test.ts`. Mobile toggle |
| CAP-03 | Live partial transcript while recording | [x] | [ ] | [ ] | [-] | Desktop `voice:partialTranscript` + recording-tip; producer added in `1.3.0` (`VoiceModeService._runPartial` → `LocalSTTService.transcribePartial`, 1.5 s cadence / 7.5 s window, never inserted). The row was `[x]` before any producer existed. |
| CAP-04 | Recording waveform + level meter | [x] | [x] | [x] | [-] | Desktop 9-bar cos distribution; mobile audio level; the recording-tip popup bundle and its on-disk assets are verified by `scripts/ci/verify-desktop-renderer-bundles.mjs` |
| CAP-05 | Device/mic selection | [x] | [ ] | [~] | [-] | Desktop config; mobile uses system default |
| CAP-06 | System/loopback audio capture | [x] | [-] | [ ] | [-] | Desktop only (caption source); mobile policy-limited |
| CAP-07 | Local Whisper STT | [x] | [-] | [x] | [-] | Desktop ships the faster-whisper sidecar (`resources/sidecar`, built by `sidecar:build`, verified by `scripts/ci/verify-sidecar-bundle.mjs`), warms it up at app start, and connects over IPv4 loopback; mobile on-device Whisper (supported devices) **2026-10-30: sidecar/cpu_budget.py applies the same bounded worker policy to primary/auxiliary Whisper models and CPU fallback; budget and constructor regression tests pass. 2026-11-01: rebuilt sidecar startup/health passed and runtime 1.16.0 is published; actual high-load typing remains open (GAP-INPUT-11).** |
| CAP-08 | Cloud STT (multi-provider) | [x] | [x] | [x] | [x] | Desktop 6 providers + D3RO Cloud; web/mobile via `stt-proxy`; .NET internal gateway |
| CAP-09 | STT auto-fallback + fail-closed | [x] | [x] | [x] | [x] | `STTManager`; SSOT R-021/R-022 GREEN |
| CAP-10 | STT model download/management UI | [x] | [-] | [~] | [-] | Desktop model manager + onboarding; mobile bundled model |
| CAP-11 | File transcription (audio/video) | [x] | [ ] | [x] | [~] | Desktop ffmpeg chunking; mobile import picker; web deferred |
| CAP-12 | Audio import from other apps (share intent) | [-] | [-] | [x] | [-] | Mobile Android `ACTION_SEND`/`ACTION_VIEW` (SSOT R-016 GREEN) |
| CAP-13 | Live captions overlay | [x] | [-] | [-] | [-] | Desktop `CaptionService` + caption-overlay popup; popup assets verified in the packaged build (`scripts/ci/verify-desktop-renderer-bundles.mjs`); GAP-INFRA-05 **2026-09-24 (unreleased, 1.7.0 candidate):** fixed 6 s batches replaced by streaming (`StreamingCaptionTrack` + core `caption-streaming.ts`): the uncommitted buffer is re-recognised every 1 s (greedy `partial`) and sent as `caption:delta` {text, stable} where `stable` is the LocalAgreement-2 prefix; 0.7 s of silence finalises the line with a full pass; unbroken audio over 12 s commits segments ending before the last 1.5 s using Whisper segment times; idle audio is trimmed to 1.5 s. Captions always use the local engine; `auto` language is pinned after the first final. Finished lines are refined by the local LLM (`buildCaptionRefinePrompt`, `acceptCaptionRefinement` rejects >35 % change) and replaced via `caption:segmentUpdated`; toggle `captionRefineEnabled` (Settings). Overlay: draggable handle with remembered position, waiting notice until the first caption. Separate caption model `captionSttModelId` (null = dictation model): the sidecar keeps one aux model next to the primary (`/load` `slot:"aux"`, `/transcribe` `model_id`, 409 `model_not_loaded` → reload + one retry); runtime minimum raised to 1.7.0. |
| CAP-14 | Recording persistence / crash recovery | [x] | [ ] | [x] | [-] | Desktop WAV persist; mobile durable queue + process-kill WAV recovery |
| CAP-15 | Android foreground recording service | [-] | [-] | [x] | [-] | Mobile API 34 FGS + persistent notification (SSOT R-005 GREEN) |
| CAP-16 | Rebindable global key bindings (keyboard + mouse) | [x] | [-] | [-] | [-] | Contract SSOT `packages/core/src/keybinding.ts`: `KEY_CATALOG` (10 groups), `KEYBINDING_ACTIONS` (6 actions), `validateBinding`, `detectBindingConflicts`. Multiple bindings per action persist as one `AppConfig.keyBindings` map (`packages/core/src/types.ts`), replacing the four singular `*Shortcut` fields; `ConfigService` migrates legacy values once. `KeyBindingService` hooks keyboard **and** mouse via uiohook — MB1 is not bindable, MB2/MB3 need a modifier, MB4/MB5 are free. **Verified 2026-09-21 on Windows by a manual run** and **2026-10-30 unit-tested** in `KeyBindingService.test.ts` (11 tests, GAP-KEY-01 `[x]`), with dedicated `command` action toggle/hold + forced voice-command matching in `VoiceModeService.ts` / `voice-intent.ts` (`voice-intent.test.ts` 19 tests, GAP-KEY-03 `[x]`). **Still open:** macOS/Linux mouse behavior (`11` GAP-KEY-02). W/M `[-]`: no OS-level global binding surface exists there. B `[-]`: device-local setting. |
---
## AI — AI Processing
| ID | Feature | D | W | M | B | Anchors / notes |
|---|---|---|---|---|---|---|
| AI-01 | Local LLM (Ollama) | [~] | [-] | [ ] | [-] | Desktop bundled Ollama. 2026-09-22 guard contract: no boot warmup; suggestion `keep_alive: 2m`; request-owned cancellation; generate/stream 2048 tokens / 120 s and chat 512 / 60 s; `done` frame required with incomplete-stream cleanup; deduplicated Ollama spawn/poll plus lifecycle disposal. GAP-LLM-03 has targeted evidence (6 test files / 69 passed / 0 failed; changed code/tests ESLint and diff check exit 0). Raw Ollama confirms cold timeout cleanup, a 553 ms `num_predict=1`/`keep_alive='2m'` response, and after the about-119.9 s expiry a 19:48:59 +09:00 `/api/ps` HTTP 200 / 45.8 ms `{models:[]}` observation with no intervening generate/unload/kill/retry; `/api/version` was HTTP 200 / 7.3 ms / `0.32.13`. This is raw API expiry/unload evidence, not app-restart or GUI/runtime proof (GAP-LLM-04). **2026-10-11 residency policy (supersedes 2m / later 30m):** single `LOCAL_LLM_KEEP_ALIVE='10m'` for every local request (`services/llm/model-residency.ts`), no boot warmup (first keystroke warms via `SuggestionService._keepWarm`), keep-warm refresh only when <4 min left, and `keep_alive: 0` unload of this app's resident models on quit (`LocalLLMService.unloadResidentModels`). Reason: gemma4:e4b holds ~11.6 GB RAM (5.9 GB pinned host buffer, no mmap on Windows CUDA) — boot warmup took free RAM 7.9 GB→32 MB within 20 s (GAP-PERF-01). **2026-10-30: loopback Ollama generate/stream/chat use max(1,min(4,available CPUs-2)) inference workers via inference-cpu-budget.ts; remote hosts retain their defaults. This is worker headroom, not a total CPU cap; CPU-only throughput and actual typing latency need measurement (GAP-INPUT-11).** |
| AI-02 | Cloud LLM (Claude/OpenAI) | [x] | [x] | [x] | [x] | Desktop `PremiumLLMService`; web/mobile via `llm-proxy`; .NET `LlmProxyService` |
| AI-03 | Auto Polish (cleanup/filler removal) | [x] | [~] | [~] | [x] | Desktop built-in; web/mobile via commands. Desktop Auto Polish combines deterministic zero-latency speech disfluency cleanup (`cleanDictatedText` in `@d3ro/core/dictation-cleaner`, wired into `VoiceModeService._transcribe`) with the plain `refine` LLM action (`llm-prompts.ts:28`) and `MeetingModeService.polishTranscript`: strips spoken fillers (`어`, `음`, `그 쓸데없는` while keeping quoted `어라든가`/`아라든가`), mid-sentence self-corrections (`검색 아니 검색이 아니라 타이핑을`→`타이핑을`), redundancies (`쓸데없는 의미없는`→`불필요한`), colloquial suffixes (`예를 들자면은`→`예를 들어`, `보면은`→`보면`), and contextual STT homophones (`스시가 영어로 변경되지`→`PC가 영어로 변경되지`, `그 길을 쓸 이유가`→`그 기능을 쓸 이유가`) while strictly preserving the speaker's original tone/register (반말/해요체/합쇼체). Regression cases pin `refine`/`summarize`/`grammar`/`expand` (`VoiceModeService.test.ts:435`, `llm-prompts.test.ts:153`, `dictation-cleaner.test.ts`). |
| AI-04 | Translate / summarize / rephrase | [~] | [x] | [x] | [x] | Built-in instructions. Plain-action path (`SettingsModal.tsx` → `defaultLLMAction`) and built-in instruction presets (`CustomInstructionService.ts`) both use `llm-prompts.ts`. **2026-10-30 (`GAP-LLM-01` `[x]`):** added `AppConfig.targetLanguage` (default `'English'`), 12-language selector in `SettingsModal.tsx`, and `resolveTargetLanguage(explicit)` reading `ConfigService` (`llm-prompts.test.ts` 43/43 PASS). `[~]` on desktop only until interactive GUI confirmation of custom-instruction insertion (`11` GAP-LLM-02). |
| AI-05 | Custom instructions (user commands) | [~] | [x] | [x] | [x] | Web `commands`; mobile `CommandsScreen` — both go through Edge Functions and are unaffected. **Desktop: the custom-instruction path never worked in any shipped release.** The instruction was passed in the `text` argument of `processText(text, action, targetLanguage, customPrompt)` with the system-prompt argument left empty; `BASE_SYSTEM_PROMPTS` has no `custom` key, so resolution fell back to `refine` silently and the model polished the instruction it was handed — the transcript never reached it. Separately, only `{{text}}` was substituted and **none of the built-ins use it** (`{{targetLanguage}}`, `{{userPrompt}}`, or no placeholder), so the substitution was a no-op from the day it was written. Introduced `fea923d` (2026-04-05); present `v0.1.0-alpha`..`v1.4.0`; **not a regression**. Three entry points were affected: commands-page activation (`CommandsPage.tsx:102-117`), command-popup selection (`bootstrap.ts:377-381`), voice keyword match (`VoiceModeService.ts:681-686`). Fixed in `9c2b4d4` (2026-09-21) — `llm-prompts.ts` is now the SSOT for placeholder substitution and argument placement (`renderInstructionPrompt:78`, `buildInstructionInvocation:101`, `resolveSystemPrompt:118`). **2026-10-29:** added a 6th built-in preset `builtin-clean-speech` (`구어·추임새·중복 정리`, `CustomInstructionService.ts`) and voice command triggers (`정리해줘`, `문장 정리해줘`, `깔끔하게 정리해줘`, `추임새 빼줘`, `중복 정리해줘`, `clean this up` in `voice-command-policy.ts`) that remove speech disfluencies, self-corrections, and redundancies while keeping the speaker's original tone (반말/존댓말). **2026-10-31:** upgraded `CommandsPage.tsx`, `tools-model.ts`, `VoiceModeService._processWithInstruction`, and `@d3ro/core/instruction-template` to support multi-command ordered pipelines (`parseActiveInstructionIds` / `composeInstructionPipelinePrompt`, checkbox multi-select with `1단계`·`2단계` badges and `↑`/`↓` reordering, e.g., `1단계: 구어·추임새·중복 정리 → 2단계: 격식체 변환`). The earlier "Red Team RT-03 verified" claim did not catch this and its cited evidence file (`red_team_log.md`) is not in the repo. `[~]` because unit tests pass (`llm-prompts.test.ts` 42, `llm-handlers.test.ts` 7, `VoiceModeService.test.ts` 22, `ChainService.test.ts` 5, `CustomInstructionService.test.ts` 12) but there is **no running-app verification** (`11` GAP-LLM-02). See `11` GAP-LLM-01/02 **Sync (2026-09-27):** desktop user commands (not the desktop presets) mirror Supabase `custom_instructions` both ways, deletions included; server presets (`builtin_key`) are not copied to desktop. **Presets (2026-09-27):** prompt edits of the four desktop presets that have a phone counterpart (translate/summarize/formal/explain-code) update the server preset row through `sync_set_builtin_instruction_prompt_v1` (NULL restores the default; `{{targetLanguage}}` is sent as English because both sides translate to English); the phone executes that prompt. `자유 프롬프트` and `구어·추임새·중복 정리` stay local |
| AI-06 | Voice keyword commands | [~] | [-] | [ ] | [-] | Desktop `VoiceCommandService` + command popup. Keyword matching itself works (`VoiceModeService.ts:681-686`), but execution went through the broken custom-instruction path (AI-05), and a **second, separate defect** made the shortcut skip LLM processing entirely whenever `defaultLLMAction === 'none'` — which is exactly the value the commands UI and the command popup write when no command is active (`CommandsPage.tsx:116`, `bootstrap.ts:385`), so an explicitly named instruction was nullified by the default setting. Both fixed in `9c2b4d4` (`VoiceModeService.ts:779`/`:785`); **not verified in a running app** (`11` GAP-LLM-02) |
| AI-07 | LLM Chains (multi-step pipelines) | [~] | [ ] | [ ] | [-] | Desktop `ChainService`. The argument placement was already correct here, but chain steps **never substituted placeholders**, so `{{targetLanguage}}` / `{{userPrompt}}` reached the model verbatim as the system prompt. Now shares the same resolution function as the other two paths (`ChainService.ts:196`). Fixed in `9c2b4d4` (2026-09-21); 5 unit tests GREEN, but `tests/red/chain.usecase.test.ts` could not execute (`11` GAP-INFRA-06) and there is **no running-app verification** (`11` GAP-LLM-02) |
| AI-08 | Screen/context capture for prompts | [x] | [-] | [ ] | [-] | Desktop `ScreenContextService` |
| AI-09 | Streaming responses | [x] | [x] | [x] | [x] | SSE/NDJSON streaming |
| AI-10 | Dictation templates (voice form fill) | [x] | [ ] | [x] | [~] | Desktop `DictationTemplateService`; mobile `TemplatesScreen` **Sync (2026-09-27):** desktop custom dictation and meeting-document templates keep their id on the server through `sync_upsert_user_template_v1`/`sync_delete_user_template_v1`, which reuse the revision-checked mobile RPCs; builtins exist on both sides and are not copied. |
---
## MEM — Memory & Knowledge
| ID | Feature | D | W | M | B | Anchors / notes |
|---|---|---|---|---|---|---|
| MEM-01 | History list + search | [x] | [x] | [x] | [x] | Desktop SQLite (`HistoryService.list`/`getAll` with SQLite `mode`, `isFavorite`, `tag`, `search`, and `beforeCreatedAt` cursor pagination — 2026-10-30 `GAP-UX-02` `[x]`); web/mobile Supabase |
| MEM-02 | History detail + edit | [x] | [x] | [x] | [x] | Desktop `HistoryService.update` + `history:update` IPC + inline edit in `HistoryEntryCard.tsx` (2026-10-30 `GAP-UX-02` `[x]`); Mobile `HistoryDetailScreen` |
| MEM-03 | History favorites | [~] | [x] | [x] | [x] | Desktop had **no** favorites until 2026-09-27 (the old `[x]` was wrong): `history.is_favorite` column, `HistoryService.setFavorite` + `history:setFavorite` IPC, star toggle in `HistoryEntryCard`; the value is the same Supabase `history.is_favorite` mobile filters on, so it syncs both ways (`apps/desktop/tests/main/sync/SyncEngine.test.ts` (20) + `apps/desktop/tests/integration/cross-device-sync.supabase.test.ts` (5, local Supabase stack)). `[~]` until a GUI run toggles it and mobile shows it |
| MEM-04 | History export / share | [x] | [~] | [x] | [x] | Desktop `MemoService.exportMarkdown` exports both tagged and untagged history with filter support and interactive save dialog (2026-10-30 `GAP-UX-02` `[x]`); web limited; mobile share sheet |
| MEM-05 | History audio playback | [~] | [x] | [x] | [x] | Signed URLs on web/mobile. **Desktop had no playback at all** (the old `[x]` was wrong); since 2026-09-27 the history card plays this device's recording from the file (`history:getAudio` → bytes → Blob) and a phone/web recording through a 5-minute signed URL of its `audio_files` row. Desktop recordings upload to the same `audio` bucket contract (`{userId}/desktop/...`, `audio_files`, ≤50 MiB, Settings > Cloud toggle `cloudSyncAudio`) so the phone plays them too (`tests/main/sync/SyncExtensions.test.ts` (9) + integration case "지식 문서·설정·녹음" against the local Supabase stack). `[~]` until a GUI run plays both kinds |
| MEM-06 | Dictionary (custom vocabulary) | [x] | [x] | [x] | [x] | All surfaces CRUD; Desktop Red Team RT-02 & RT-18 fuzzed/verified |
| MEM-07 | Dictionary import/export | [x] | [x] | [x] | [-] | Desktop `dictionary:import/export` JSON+CSV (file dialogs, `DictionaryService`); web `serializeDictionary`/`importDictionaryFile` + header buttons; mobile CSV/TXT export + CSV/JSON/TXT import via `data-portability` |
| MEM-08 | Memos (tags over history) | [x] | [ ] | [x] | [x] | Desktop `MemoService`; mobile `MemosScreen`; web none. **Desktop tags sync with mobile since 2026-09-27**: pushed through the mobile RPCs (`mobile_add/remove_memo_tag_v1`) and reconciled by natural key `historyId` + normalized tag because the server owns tag ids and renames rows (`services/sync/memo-tag-sync.ts`). Tag normalisation (trim, collapse spaces, lower-case) is now identical on both sides |
| MEM-09 | Knowledge base / RAG add+index | [x] | [x] | [x] | [x] | Desktop local RAG (DEF-008 infinite chunking loop resolved, RT-08 verified); web/mobile cloud RAG **Sync (2026-09-27):** desktop documents mirror `knowledge_documents` + `knowledge_chunks` as source text; each surface embeds with its own model (desktop nomic 768-d, server 1536-d), the server index is requested through `embed-chunks`. Documents synced from the phone are stored without a file and re-indexed from their stored chunks (`RAGService.applyRemoteDocument`). Chunk text is now kept even when local embedding fails |
| MEM-10 | Semantic search over knowledge | [x] | [x] | [x] | [x] | Web `KnowledgeSearch` calls `search-knowledge` (was mislabeled deferred); mobile + Edge `search-knowledge` |
| MEM-11 | Knowledge file upload | [x] | [x] | [x] | [x] | Web `.txt`/`.md` picker + newline-aware chunking + `embed-chunks`; desktop txt/md/pdf/docx; mobile file picker **Sync (2026-09-27):** desktop documents mirror `knowledge_documents` + `knowledge_chunks` as source text; each surface embeds with its own model (desktop nomic 768-d, server 1536-d), the server index is requested through `embed-chunks`. Documents synced from the phone are stored without a file and re-indexed from their stored chunks (`RAGService.applyRemoteDocument`). Chunk text is now kept even when local embedding fails |
| MEM-12 | Voice actions (OS automation) | [x] | [x] | [x] | [x] | Desktop `VoiceActionService`; web `ActionRunner` (simulated); mobile `ActionsScreen` |
| MEM-13 | Cross-surface data sync | [~] | [x] | [x] | [x] | **Desktop rewritten 2026-09-27** (`services/sync/SyncEngine.ts`): persistent outbox (offline/restart safe), server-clock keyset cursors per user DB with pagination, pending local edits never overwritten by pull, deletions both ways through `sync_tombstones`, per-row failure isolation, Realtime that actually connects (`ws` transport). Covers history (+title/favorite), dictionary, meetings, meeting memos/documents, memo tags, custom commands, dictation + meeting-doc templates; plus device registration. Evidence: `apps/desktop/tests/main/sync/SyncEngine.test.ts` (20) + `apps/desktop/tests/integration/cross-device-sync.supabase.test.ts` (5, local Supabase stack). `[~]` until migration `20260927000034` is deployed to production and a desktop build is exercised against a phone (GAP-SYNC-05); knowledge/audio/settings still local (GAP-SYNC-06) **2026-09-27 second batch:** knowledge base, recordings (upload + remote playback) and the mobile `user_settings` fields that mean the same thing on both sides (ko/en locale, system/light/dark theme, auto-polish on/off, active user command) now sync too; remote setting changes reach the open window (`app:dataChanged`, `config:changed`). Production DB is at migration `0035` and all Edge Functions are redeployed from the repo |
| MEM-14 | Memo tag search | [x] | [ ] | [x] | [x] | |
| MEM-15 | Unified record + meeting search | [~] | [ ] | [x] | [-] | Shared `@d3ro/core/types` result contract. **2026-10-30 (`GAP-UX-05`):** Mobile (`unified-search-service.ts`, `UnifiedSearchScreen.tsx`) now supports `AbortSignal`/`AbortController` in-flight cancellation, `transcripts` segment table search (`searchTranscriptSegments`), and bounded payload projection (`boundPayloadText`), verified by `unified-search-service.test.ts` + 320dp dark/411dp light runtime journeys (`[x]`). Desktop (`UnifiedSearchService.ts`) searches `history`, `meetings`, `meetingMemos`, and timed transcript segments (`UnifiedSearchService.test.ts` 12/12 PASS; `[~]` pending Electron `Ctrl+K` manual check). |
---
## MTG — Meetings
| ID | Feature | D | W | M | B | Anchors / notes |
|---|---|---|---|---|---|---|
| MTG-01 | Meeting recording + live transcript | [x] | [~] | [x] | [x] | Desktop `MeetingModeService`; web realtime view; mobile **Sync (2026-09-27):** the phone renders `transcripts` segments in preference to `edited_transcript`, so desktop edits, auto-polish and diarization never reached it; the desktop now rewrites the meeting's segments from its `[MM:SS] [speaker] text` lines on every transcript change (upsert by `meeting_id,segment_index`, trailing segments trimmed; parser SSOT `@d3ro/core/meeting-transcript`, `tests/main/sync/SyncTranscriptsPresets.test.ts` (4) + local-stack integration case "전사 구간·프리셋") |
| MTG-02 | Meeting creation (title/attendees/language/template) | [x] | [x] | [x] | [x] | SSOT H-006 GREEN |
| MTG-03 | Timestamped memos during & after meeting | [x] | [x] | [x] | [x] | **2026-10-30 (`GAP-UX-01` `[x]`):** `MeetingModeService.addMemo(content, meetingId?)` supports ended meetings and syncs `MeetingNotesTab` notepad notes via `CloudSyncService.pushOne('meeting_memos', id)` (`desktop-ux-gaps.test.ts` 13/13, `meeting.usecase.test.ts` 23/23 PASS). |
| MTG-04 | AI summary generation | [x] | [x] | [~] | [x] | Desktop `MeetingSummaryService` generates `[MM:SS]` timestamp citations (`parseCitationTimestampMs`, `extractTranscriptCitations`) with clickable jump-to-segment badges in `MarkdownRenderer`/`MeetingSummaryTab` (2026-10-30 `GAP-UX-03` `[x]`); web `generate-document-button`; mobile via Edge |
| MTG-05 | Document generation (minutes/report/idea-note/mindmap) | [x] | [x] | [x] | [x] | Edge `generate-meeting-document`; SSOT F-011 GREEN |
| MTG-06 | Document edit (Markdown) | [x] | [x] | [x] | [x] | Desktop editor; web `document-editor` |
| MTG-07 | Export PDF/DOCX/TXT/Markdown | [x] | [~] | [x] | [x] | Desktop `ExportMenu`; web markdown; mobile print/DOCX chooser (SSOT data portability GREEN) |
| MTG-08 | Speaker diarization | [~] | [ ] | [ ] | [ ] | Desktop `phase-15.5` (LLM estimate + pyannote prep); mobile SSOT H-014 pending |
| MTG-09 | Audio seek ↔ transcript timestamp | [x] | [x] | [x] | [-] | **2026-10-30 (`GAP-MTG-01` `[x]`):** Desktop (`TranscriptTab.tsx` / `EditableSegment.tsx`), Web (`meeting-audio-player.tsx` ↔ `live-transcript-list.tsx`), and Mobile (`meetings-service.ts`, `MeetingDetailScreen.tsx`) parse `[MM:SS]` / `timestamp_ms` segments, seek audio on badge tap, and highlight the active transcript segment (`__tests__/meetings.test.ts` PASS). |
| MTG-10 | Meeting list search/filter/sort | [~] | [ ] | [~] | [x] | Basic lists; advanced filters pending |
| MTG-11 | Meeting AI chat over transcript | [x] | [ ] | [ ] | [x] | Desktop `MeetingChatPanel` + `MEETING_CHAT` |
| MTG-12 | Content reporting for generated docs | [-] | [ ] | [x] | [x] | `content-report` Edge + generation receipts |
| MTG-13 | Existing audio/video import into a meeting | [x] | [ ] | [~] | [x] | **2026-11-01:** Desktop `meetingMode:importRecording` reuses `FileTranscriptionService` without duplicate history, saves canonical timestamped transcript + normalized WAV, then opens existing minutes generation. Mobile `MeetingsScreen` creates the workspace then routes `meetingId` + `importRequest` to the existing durable Record meeting pipeline. **2026-11-02:** installed Windows 1.16.0 passed the actual Electron picker→timestamped transcript→local summary/action-item journey, so desktop is `[x]`. Fold6 `.3` import entry/editor rendered, but authenticated device file-to-minutes E2E remains (`GAP-MTG-03`). [Evidence](../evidence/2026-11-02-verification.md). |
| MTG-14 | Learning-informed transcript correction (user dictionary + typed vocabulary) | [~] | [ ] | [ ] | [ ] | **2026-11-05 (root cause + rebuild of the transcript-correction seam):** the learned corpus was only wired to input suggestions. Meeting/live captions ran STT with a hardcoded `다음은 한국어 대화입니다.` prompt (no dictionary, no learned terms), the refine prompt had no glossary, the refine result updated only `CaptionService`'s own copy (meetings and the renderer kept the raw line), and the persisted `rawTranscript` was a plain join of raw segments. Now: core SSOT `@d3ro/core/transcript-correction` (glossary build, dictionary-variant replacement, learned-identifier correction with a unique-best rule, LLM reference block) + desktop `TranscriptGlossaryService` (dictionary + `personal_phrases`, 15 s cache invalidated on dictionary writes, fail-closed) are wired into `CaptionService` (hint on every recognition, deterministic correction per final line, glossary-aware refine, `segment-updated` event), `MeetingRecordingSession`/`MeetingModeService` (refined lines reach the meeting transcript and renderer via `meetingMode:segmentUpdated`; auto-polish gets the glossary block), `FileTranscriptionService` (per-chunk hint + correction) and `VoiceModeService` (learned-term pass). **2026-11-05 (v1.16.3 correction policy):** the first cut also rewrote ordinary Hangul words by 1-jamo distance. Measured against the real local corpus (users/_local/d3ro.db, 3 stored meeting transcripts) it changed **286 of 3,005 words** (하면 to 라면, 이제 to 이게, 매일 to 메일, 정보 to 정도). Deterministic correction is now limited to dictionary pronunciations and typed Latin identifiers (canonical casing preserved); Korean terms reach recognition through the STT hint and the polish prompt only. Re-measured on the same corpus: **0 of 3,005**. Real-engine check (sidecar large-v3-turbo, Korean TTS fixture containing the user term): baseline 하람로그에 to hinted transcript haramlog에, and a dictionary pronunciation entry still maps 디에로 to D3RO. Hangul-layout typing artifacts (dkssudgktpdy) are dropped from the hint. Tests: core transcript-correction.test.ts (16), desktop transcript-glossary.test.ts (10); live meeting with real jargon still open (GAP-MTG-04). |
---
## CV — Conversation
| ID | Feature | D | W | M | B | Anchors / notes |
|---|---|---|---|---|---|---|
| CV-01 | Local duplex voice conversation (STT→LLM→TTS) | [x] | [ ] | [x] | [-] | Desktop `VoiceConversationService`; mobile Talk |
| CV-02 | Realtime voice (OpenAI gpt-realtime, Premium) | [x] | [ ] | [ ] | [x] | Desktop `useRealtimeConversation` + `realtime-token` Edge |
| CV-03 | Text AI chat | [x] | [x] | [x] | [x] | Desktop chat, web `chat-panel`, mobile `TalkScreen` |
| CV-04 | TTS playback + controls | [x] | [ ] | [x] | [-] | Desktop SAPI/`say`; mobile Android TTS |
| CV-05 | Voice selection / backend selection | [x] | [ ] | [~] | [-] | Desktop settings `conversationBackend` |
---
## ACC — Accounts, Sync, Devices, Portability
| ID | Feature | D | W | M | B | Anchors / notes |
|---|---|---|---|---|---|---|
| ACC-01 | Email sign-up/login | [ ] | [x] | [x] | [x] | Desktop uses separate online-auth; SSOT A-001..A-005 `[ ]` rows are mobile checklist granularity |
| ACC-02 | OAuth Google | [~] | [x] | [~] | [x] | Mobile full consent→callback pending (external) |
| ACC-03 | OAuth GitHub / Apple | [~] | [~] | [~] | [~] | GitHub/Apple provider secrets pending |
| ACC-04 | Password reset / recovery deep link | [ ] | [ ] | [x] | [x] | Mobile ForgotPassword/UpdatePassword |
| ACC-05 | Account profile / identity management | [~] | [~] | [x] | [x] | Desktop `LicenseTab`; mobile `AccountScreen` |
| ACC-06 | Logout + local sensitive purge | [x] | [x] | [x] | [x] | Mobile central purge GREEN |
| ACC-07 | Account deletion (server cascade + local purge) | [ ] | [ ] | [~] | [x] | `account-delete` Edge; mobile device E2E pending. **2026-11-04 backend retention proof:** isolated local Supabase verified `audio-purge` authorization and both direct and pg_cron/Vault dispatch; the queued DB row, deleted `audio_files` row, and private Storage object were removed together. Production deployment/schedule remains `GAP-OPS-04`. |
| ACC-08 | Cloud sync (per-user data) | [~] | [x] | [x] | [x] | Desktop `CloudSyncService` + `services/sync/`. The earlier `[x]` hid shipped defects fixed on 2026-09-27 (GAP-SYNC-01/02): first pull after sign-in fetched nothing, only the first meeting's memos/documents were pushed, deletions never propagated, team meetings were pulled into the personal DB and their `team_id` cleared on re-push, Realtime never connected. Anonymous local-mode data is now imported into the first account that signs in (`db/index.ts importLocalModeData`). Production deploy pending (GAP-SYNC-05) |
| ACC-09 | Device registration + revocation | [~] | [~] | [x] | [x] | Mobile `DevicesScreen`. **Desktop registers itself since 2026-09-27** (`services/sync/device-registration.ts`, platform `windows`/`macos`, installation id in `AppConfig.deviceInstallationId`); revoking it from the phone signs the desktop out (Realtime on `devices` + 5-min heartbeat), signing out on desktop unregisters it. Web creates/checks in a stable `platform='web'` device only when the user opts into browser notifications, refuses to resurrect a revoked row, and revalidates an existing local Push subscription through the register RPC before showing it connected (`web-device.ts`, `web-push-session.ts`, `web-push.spec.ts`). An isolated throwaway local-browser run produced exactly one active web device and one device-bound token; no web device-management UI or production proof. Mobile parser accepts all five platforms and names them on the card. Local-stack desktop evidence exists; production pending GAP-SYNC-05 |
| ACC-10 | Offline queue + retry | [~] | [ ] | [x] | [x] | Mobile durable queue |
| ACC-11 | Data export/import (portability) | [~] | [ ] | [x] | [x] | Mobile canonical JSON E2E GREEN; desktop has export files. **2026-09-27 backend fix:** `export_account_portability` serialised whole rows, so columns added after the v1 archive (meeting creation metadata, meeting-document template/idempotency) made any account with meetings fail the exact-key check in both the server restore and the mobile parser; the export now projects onto v1 keys (migration `20260927000034`, integration test asserts memo/meeting keys) |
| ACC-12 | Notification / push | [ ] | [~] | [~] | [x] | Backend transports for FCM + Web Push (VAPID) + APNs (.p8) + outbox cron drain implemented. **2026-11-04 (`GAP-PUSH-03` `[~]`):** Web production sidebar calls the explicit user-gesture permission → stable web device → `subscribeWebPush` → `register_push_registration` path. Opt-out records disabled intent before bounded parallel server/local cleanup; a stale local subscription is re-registered through the server RPC before it may appear connected, and logout never waits on cleanup (`web-push.spec.ts` 17 contract tests). The standalone image copies `public/sw.js`, and missing build-time `NEXT_PUBLIC_VAPID_PUBLIC_KEY` fails closed. Isolated local evidence reached active device + token + pending outbox + completed drain + `push_deliveries.status='delivered'`; this is provider acceptance, not browser receipt. The final production-standalone browser attempt timed out at login navigation before the dashboard, so production permission/subscription/render/receipt and deployed cron/provider configuration remain open (`EXT-FIREBASE-01` for Android). |
---
## TEAM — Teams & Admin
| ID | Feature | D | W | M | B | Anchors / notes |
|---|---|---|---|---|---|---|
| TEAM-01 | Team create / rename / delete | [-] | [x] | [x] | [x] | Desktop N/A |
| TEAM-02 | Invite by email / link + accept deep link | [-] | [x] | [x] | [x] | `team-invite`/`team-accept` Edge Functions; mobile deep link; **2026-10-30 (`GAP-TEAM-02` `[x]`):** `/app/accept-invite?token=...` browser flow + landing `/accept-invite/` web CTA. |
| TEAM-03 | Members + roles (admin/member/viewer) + leave | [-] | [x] | [x] | [x] | |
| TEAM-04 | Team meetings/docs sharing + RLS isolation | [-] | [x] | [~] | [x] | Cross-user isolation tested |
| TEAM-05 | Team comments / activity feed | [-] | [x] | [x] | [x] | `team_activities` migration + `create_team_activity` RPC + realtime; web `ActivityFeed`, mobile TeamDetail activity card (2026-09-13) |
| TEAM-06 | Admin back office (users/subs/models/usage/audit) | [-] | [-] | [x] | [x] | `apps/admin` + mobile `AdminScreen` |
| TEAM-07 | Role-based destructive action confirm + audit | [-] | [x] | [x] | [x] | SSOT T-006..T-008 GREEN |
| TEAM-08 | Desktop admin surface | [ ] | [-] | [-] | [-] | None; N/A by design |
---
## MON — Monetization & Ads
| ID | Feature | D | W | M | B | Anchors / notes |
|---|---|---|---|---|---|---|
| MON-01 | Tier gating (Free/Pro/Pro+/Team/Enterprise) | [x] | [x] | [x] | [x] | `LicenseService`, entitlement provider |
| MON-02 | Usage quotas (daily_usage) | [x] | [x] | [x] | [x] | `packages/core/src/plan-catalog.ts` is the quota SSOT, including `knowledge_embedding`; Deno consumers use the generated `core-contract.generated.ts` copy guarded by `npm run contract:check`. |
| MON-03 | Desktop offline license (Ed25519) | [x] | [-] | [-] | [x] | `crypto-license` + admin issuer |
| MON-04 | ~~Web checkout (Stripe)~~ — **삭제됨 2026-09-26** | [-] | [-] | [-] | [-] | 사용자 결정: 결제는 웹 Payple(KRW) + 모바일 Google Play만. `stripe-checkout`/`stripe-portal`/`stripe-webhook` 함수, 웹 `checkout-button`/`portal-button`, 데스크톱 `CheckoutModal`·`payment-handlers`·`payment:*` IPC 삭제. 과거 `provider='stripe'` 구독 행은 읽기 전용 표시만(마이그레이션 불변). 운영 정리는 GAP-BILL-03 |
| MON-05 | Web checkout (Payple) | [-] | [x] | [~] | [~] | Payple checkout/manage/renew/webhook; webhook signature pending |
| MON-06 | Paywall / upgrade prompts | [x] | [x] | [x] | [x] | `UpgradePromptModal`, `ProPaywallScreen` |
| MON-07 | Mobile IAP purchase + restore | [-] | [-] | [~] | [x] | `iap-verify` + `billing-context`; live store E2E blocked |
| MON-08 | Billing catalog / pricing display | [x] | [x] | [x] | [x] | Server catalog SSOT; hardcoded prices removed |
| MON-09 | Free-tier banner ads | [~] | [-] | [x] | [x] | Desktop adapters fail-closed; mobile AdMob test GREEN, prod serving blocked |
| MON-10 | Rewarded ads → quota credits | [x] | [-] | [x] | [x] | **2026-10-30 (`GAP-ADS-02` `[x]`):** Desktop `AdMediationEngine.claimReward` verifies adapter `reportRewardCompletion`, impression, duration, cooldown, and nonce uniqueness before calling `LicenseService.grantRewardQuota` (`AdMediationEngine.test.ts` 23/23 PASS); mobile SSV GREEN |
| MON-11 | Ad mediation engine + settlement | [~] | [-] | [~] | [x] | Engine + settlement built. `DirectHouseSponsorAdapter` is now a **real configurable REST adapter** (bid/impression/click/reward via `endpointUrl`, fail-closed when unconfigured, unit-tested). Other 9 networks remain `UnavailableAdAdapter` stubs pending official SDKs. |
| MON-12 | Subscription management (portal/store) | [-] | [x] | [x] | [x] | Payple manage / Play manage (Stripe portal 삭제 2026-09-26) |
---
## INPUT — Input Intelligence (typing telemetry & next-sentence suggestions)
Everything here is **desktop-only** (Windows today) and **opt-in**: `inputTelemetryEnabled` and
`inputLearnTypedText` default to `false`. D=desktop, W=web, M=mobile, B=backend.
Status is `[~]` on desktop for one shared reason: the code and its 48 unit cases exist, but the
end-to-end behaviour has **not been verified by typing in a real app** (`11` GAP-INPUT-01).
| ID | Feature | D | W | M | B | Anchors / notes |
|---|---|---|---|---|---|---|
| INPUT-01 | Keyboard/mouse telemetry capture (opt-in) | [~] | [-] | [-] | [-] | `InputTelemetryService` — keystroke/click/scroll counters, mouse travel as the Manhattan sum of per-axis pixel deltas, active time, per-hour×app buckets flushed every 5 s. Key **contents** are never stored (ActivityWatch `aw-watcher-input` data-minimisation policy, adopted deliberately — see `11` §7). Hook ownership is ref-counted so `KeyBindingService` keeps working (`global-input-hook.ts`). |
| INPUT-02 | Foreground-app attribution | [~] | [-] | [-] | [-] | `utils/win32-foreground.ts` via `koffi` FFI (title/pid/exe/bounds), sampled at most 1×/s. `get-windows` was rejected: it needs an install script this repo does not run. |
| INPUT-03 | Weekly input insights | [~] | [-] | [-] | [-] | `INPUT_TELEMETRY.getSummary` aggregates `input_activity` into totals, daily series, top hours and top apps; rendered in Settings → Input and as a dashboard card. Daily average mouse travel is converted px → m using the display scale factor. |
| INPUT-04 | Typed-text learning (UIA, password-excluded) | [~] | [-] | [-] | [-] | Text is read from the focused field via `GET /uia/focus` (sidecar UIA bridge) and diffed longest-common-prefix/suffix, so **IME-committed Hangul/kana is counted correctly** — keycodes cannot reconstruct CJK text. UIA sends `hasSelection` only, derived by TextPattern range Start/End comparison without calling `GetText` on the selection range or adding a selected-text payload; the existing focused-field text can still include a selection. A non-collapsed selection immediately clears suggestions as `selection-active`. `IsPassword` is checked before any read (fail-closed); IME composition suppresses both stats and suggestions. **2026-09-24:** text equal to the control name (empty-field placeholder, e.g. "메시지 입력") is treated as empty (`withoutPlaceholderText`); a suggestion requires `editedSinceFocus` + typing within `recentTypingWindowMs` (8 s), otherwise `not-typing` — clicking into a pre-filled field no longer triggers suggestions. **2026-10-29:** `_typedInCurrentFocus` and `_lastTypedHwnd` track whether a typing keystroke occurred after the latest click/focus rebase (`_rebaseFocusText`), and `_sampleForeground` forces immediate foreground sampling on that first keystroke, so typing immediately after clicking into a search box or new window sets `editedSinceFocus = true` on the very first snapshot. **2026-10-24:** typed deltas are no longer saved per snapshot (that stored mid-word fragments every 0.4–0.8 s). They are buffered per field and saved as one baseline→latest diff on 2 s idle, sentence-ending punctuation, 150-char growth, field clear (send), focus change or stop. Typed samples under 4 non-space chars, URL/query-like lines, trailing incomplete jamo and PUA/lone-surrogate chars (emoji truncated to 16 bits upstream, root cause unconfirmed) are dropped. Rows already in the DB are pruned on start by the same rule (`isLearnableTypedText` in `_pruneUnlearnableCorpus`, typed source only; dry count on the author machine 2026-10-24: 1,235/2,664 samples, 285/1,209 phrases). Expressive jamo runs (ㅋㅋ, ㅠㅠ, ㅇㅇ) and space-less questions are kept. Mid-word fragments of 4+ chars cannot be detected and remain until the 30-day retention. **2026-10-30..31: UIA reads are single-flight and discarded after input/focus/lifecycle changes, stale capture times, or changed native GetLastInputInfo tokens; contexts carry inputRevision/nativeInputTick for replacement-time validation (`InputTelemetryService`, `win32-input-activity`). Standalone modifier keydowns/key-repeats (`Ctrl`/`Alt`/`Shift`/`Win`) and `Ctrl+Alt` suggestion navigation chords are ignored in `_handleKeyDown` without advancing `_inputRevision`/`_lastKeyAt` or scheduling UIA snapshots (`classifyKeyStroke` checks modifier VKs before modifier flags), and `_settleTimer` (780 ms) is skipped on non-typing keys and auto-cancelled when the 380 ms `_textTimer` snapshot cannot trigger a suggestion (`prefixLen < minPrefixChars`). See GAP-INPUT-11.** |
| INPUT-05 | Personal phrase corpus (typed + voice) | [~] | [-] | [-] | [-] | Sentence-level phrases from typed text and from voice history (`HistoryService.create` feeds `recordExternalText`), ranked by frequency/recency as prompt hints; users can delete individual phrases. **2026-09-24:** typed text from `LEARNING_EXCLUDED_APPS` (terminals, code editors, Agent Switchboard) is never learned, and every phrase must pass `isLearnablePhrase` (no box/block/geometric glyphs, ≥60 % letters). `_pruneUnlearnableCorpus` re-applies both rules to existing phrases/samples/edges on start and every retention cycle (first run on the author machine removed 69 phrases / 91 samples of terminal status lines and agent chats). **2026-10-25:** Agent Switchboard moved from `LEARNING_EXCLUDED_APPS` to `APP_SCOPED_LEARNING_APPS` — its text is kept only as `typing_samples` (with app name), never in shared phrases/graph, and is used only for suggestions inside that same app (`findContinuationHints(..., { appName })` feeds `continuationHints`). It is the user's most-typed app; the 2026-09-24 exclusion was about dev instructions leaking into KakaoTalk suggestions, which scoping prevents. |
| INPUT-06 | Next-sentence suggestion (ghost text) | [~] | [-] | [-] | [-] | `SuggestionService` + `buildSuggestionPrompt` (`llm-prompts.ts` SSOT, instruction stays in the system prompt). The 2026-09-22 guard contract is 600 ms debounce, ≥5 s between requests, 6 requests/min by default (hard-config maximum 12), 3 candidates, 64 output tokens, 12-character growth before regeneration, 8 s request timeout and `keep_alive: 2m`; boot warmup is removed. Each request has its own cancellation signal. Presentation-active includes candidates, `generating`, `warmingUp` and `partialText`; clear/dismiss aborts, invalidates the generation token, clears TTL state and emits `cleared`/hide, and a final success resets `generating=false`/`partialText=null`. Focused evidence for the lifecycle and Windows-child-process change: five test files / 80 tests passed; desktop typecheck/lint, Python `py_compile`, and `git diff --check` exited 0 (core 131-test pass was independently verified earlier). Status remains `[~]`: this is not app-restart, GUI overlay, or real automatic-typing evidence. **2026-09-24 (supersedes the numbers above):** `keep_alive: 10m`, warm-up on boot, on Ollama reconnect and whenever the model is presumed cold (cold reload measured 11.4 s > 8 s timeout); a session asks for 1 candidate, then a sequential fill loop appends one unique candidate at a time up to `maxCandidatesTotal` 12 (avoid-list in the prompt, stops after 2 empty fills); only the first request consumes rate/daily budget; stale aborts refund the budget; any further typing ends the session (`matchesSessionPrefix`, IME last-syllable tolerant); prompt forbids assistant-style questions. Terminals (`TERMINAL_APPS`) never get suggestions. **2026-10-24 accept fix:** `waitForModifiersReleased` now trusts OS `GetAsyncKeyState` (`utils/win32-key-state.ts`) over the uiohook-tracked set. A missed keyup made acceptance wait the full 1.5 s and then paste Ctrl+V while modifiers were still considered held; in the logged case the field length stayed 15 after "inserted" (6 chars). New suggestions are also suppressed while an accept is in flight, so the overlay no longer reappears during the wait. **2026-10-24~25 experiment, reverted:** an n-gram-first / confidence-gated / short-candidate rework raised offline next-word precision (3%→33%) but made suggestions rare, and the user judged the app much worse. Suggestion behaviour is back to v1.10.1. Kept: suggestions that copy on-screen text are rejected (`copiesContext`), no suggestions while typing an address/path (`isAddressInput`, skip reason `address-input`). Lessons and data: `docs/evidence/2026-10-24-suggestion-accuracy-research.md` §9; replay harness `tests/eval/`. **2026-10-25 additive upgrades (frequency unchanged):** (1) `suggestion-accept-word` (default Ctrl+Alt+Space) inserts only the first word and keeps the rest on screen (`splitFirstWord`; partial accepts are not learned as full sentences). (2) a confident personal n-gram candidate (`PersonalNgramIndex`, strict `PERSONAL_NGRAM_SETTINGS`) is shown instantly as candidate #1 while the model generates; model candidates are appended after it, and it survives an empty/failed/timed-out model run (no "no suggestions" notice then). Agent Switchboard text never enters this shared n-gram. Hotkey accept now force-releases held Ctrl/Alt/Shift after 200 ms (`prepareModifiersForPaste`). **2026-10-28 extreme personalization & n-gram upgrades (frequency unchanged):** `PersonalNgramModel` now weights repeated `personalPhrases` (`count >= 2`) and falls back to a Korean stemmed-word bigram index (`_stemBigram` via `stemTerm`, e.g. matching `"배포를 완료했습니다"` when typing `"배포가 "`), and `SuggestionService._fillOne` reranks newly generated LLM candidates by `scoreCandidatePersonalization` (overlap with personal graph `continuations`/`followUps`/`related`/`phrases` + register match) while preserving the instant n-gram slot at index 0. **2026-10-31 (`v1.15.2`) `Ctrl+Alt+Enter` reliability & terminal accept fix:** (1) `ChordStateMachine.keyUp` releases all held bindings sharing a released modifier (`Ctrl`/`RightAlt`) so `suggestion-accept` never stays stuck in `_isDown` after `command` (`Ctrl+RightAlt`), (2) `KeyBindingService` maps `UiohookKey.NumpadEnter` and adds a 25 ms `globalShortcut` fallback trigger while `isOverlayOrCaptureActive` is true, (3) `SuggestionService.accept()` waits up to 2.5 s (`_waitForFirstCandidate`) when invoked while candidate #1 is still generating/pending, and (4) in `terminalMode` / `isTerminalApp`, `_insertAccepted` uses `'keyboard'` (`KEYEVENTF_UNICODE`) instead of `'clipboard'` (`Ctrl+V`), syncs `TerminalKeystrokeTracker` via `applyProgrammaticReplacement`, and stops `_fillRemainingCandidates` immediately (`suggestion-accept-hold.test.ts` 4/4, `KeyBindingService.test.ts` 12/12, `keybinding-runtime.test.ts` 29/29 PASS). |
| INPUT-07 | Caret-anchored suggestion overlay | [~] | [-] | [-] | [-] | `suggestion-overlay` popup placed by `anchorFloatingPanel` (caret → element → cursor fallback, flip above when the caret is near the bottom, clamped to the work area). Non-focusable; click-through unless `suggestionOverlayInteractive`. While actually visible it continues periodic UIA validation after 5 s and revalidates 120 ms after mouse-up; unavailable/non-editable focus or a non-collapsed selection aborts and hides it. X first hides the renderer panel, then main IPC directly hides `BrowserWindow` and dismisses the service, so late tokened results cannot revive it. Accept/next/prev/dismiss are four global key bindings (`suggestion-accept`/`next`/`prev`/`dismiss`, default `Ctrl+Alt+→/↓/↑/←`), and the overlay has a mouse close button. Up to three candidates are shown in a scrollable list with a warm-up/generating spinner. **2026-09-24:** one page of 3 fixed-height (2-line) numbered items, footer = range/progress + source + key guide derived from the live bindings (`buildSuggestionKeyHints`, shared modifier shown once). Keys: Ctrl+Alt+↑/↓ move (the page follows the active item; at the last item `next` waits while filling, otherwise wraps; the 1.6.0 Left/Right page actions were removed as unreachable before a page filled and colliding with Intel display rotation), Ctrl+Alt+Enter accept, plain Esc closes while visible (overlay-scoped, not a binding), Ctrl+Alt+Backspace secondary dismiss; old default bindings migrate at tuning revision 5. Streaming updates never reposition/re-present the window (`decideSuggestionOverlayAction`); placement uses `anchorKind` — caret: below the line, element: outside the element (below → above → right → left → inner bottom-right). Navigation re-arms the visible TTL. Window 460×208. |
| INPUT-08 | Per-app exclusions & consent controls | [~] | [-] | [-] | [-] | `inputExcludedApps` (executable names, case-insensitive) blocks both collection context and suggestions; telemetry master switch, pause, text-learning toggle and "delete collected data" all live in Settings → Input. 30-day retention prune runs on start. |
| INPUT-09 | Flow Radar | [~] | [-] | [-] | [-] | `rankFlowWindows` ranks hourly aggregate activity density, character volume and edit stability into potential-focus time windows. It is not a real-session detector or session record. |
| INPUT-10 | Edit Friction | [~] | [-] | [-] | [-] | `calculateFrictionInsight` derives friction from char/backspace quantities and reports edits per 100 chars; it does not infer sentiment or productivity. |
| INPUT-11 | App DNA | [~] | [-] | [-] | [-] | Personal phrase ranking passes `appName` context and applies a 1.75× same-app ranking bonus (`selectPhraseHints`); learned text remains opt-in. |
| INPUT-12 | App Quality | [~] | [-] | [-] | [-] | `InputTelemetryService.getSummary` aggregates per-app suggestion total, accepted count, accept rate and average latency from local suggestion history. |
| INPUT-13 | Privacy Receipt | [~] | [-] | [-] | [-] | Local-only receipt shows actual row counts. Raw individual key events/key codes/content stream are not retained; opt-in learned text can reside in `typing_samples`/`personal_phrases`. `input_activity`, `typing_samples` and `suggestions` retain 30 days; `personal_phrases` has no age-based automatic expiry and is removed by individual deletion, delete-all, or consent withdrawal. Receipt read/delete storage failures are IPC errors. |
| INPUT-14 | Smart Exclusion | [~] | [-] | [-] | [-] | Password fields add no evidence. Only the current app may receive a one-click recommendation after ≥4 observations, readable=0 and problematic ratio ≥75%; it never auto-excludes. |
| INPUT-15 | Why This Suggestion | [~] | [-] | [-] | [-] | Overlay exposes only local-model/local-memory provenance and continuation/related/phrase/appPhrase counts; raw evidence text is not shown. |
| INPUT-16 | Memory Decay | [~] | [-] | [-] | [-] | Phrase ranking combines a 30-day half-life, frequency and the App DNA 1.75× same-app bonus. |
| INPUT-17 | Instant Recall | [~] | [-] | [-] | [-] | Local-memory fallback (`provenance=local-memory`) occurs only for unavailable model, non-cancellation error, actual timeout or empty model result; it spends no additional model budget and is blocked on dismiss, new typing, token/context mismatch or staleness. |
| INPUT-18 | Shortcut Safety Audit | [~] | [-] | [-] | [-] | Settings UI applies core `auditKeyBindingMap` to surface invalid/conflict issues while hold/double-press exceptions remain unchanged. |
| INPUT-19 | Typo auto-correct while typing (built-in automatic Han/Eng conversion) | [~] | [-] | [-] | [-] | **2026-10-03; upgraded 2026-10-26..31 (2-Stage Hybrid Auto-Correction: Araso-caliber instant layout engine + Smart Background Gemma Pass + Atomic Single-Syscall SendInput + Delta Rebasing + Terminal & Vim & xterm.js zero-swallow IME protection + live Windows GUI E2E verified).** `TypoCorrectionService` (main, singleton + EventEmitter) runs on each `typed-context` before `SuggestionService`. **Active by default out of the box:** `InputTelemetryService` starts the UIA context observer whenever `typoAutoCorrectEnabled !== false` (even when `inputTelemetryEnabled` is `false`), and skips persisting `input_activity` rows to SQLite unless `inputTelemetryEnabled === true`; in Settings → Input (`InputConsentPanel.tsx`), the `typoAutoCorrect` switch works independently of telemetry consent. **2-Stage Hybrid Auto-Correction (`2026-10-27..28`):** `bootstrap.ts` calls `suggestion.ensureWarmForTyping(context)` on every `typed-context` (keeping `gemma4:e4b` warm even when `suggestionEnabled` is off) and runs the full 2-stage `typo.handleTypingContext(context)` in-place: Stage 1 (Instant `<1 ms` deterministic rule engine at 25 ms after word boundary / 650 ms at token tail) immediately converts known Han/Eng layout errors and dictionary misspellings, while Stage 2 (Background Gemma Pass at 350 ms after word boundary for suspicious tokens or 600 ms idle for general sentences, anchored to the trailing 2 words via `trailingWordOnly: true`) queries the warm local LLM (`gemma4:e4b`) with 2-set keyboard hints (`buildModelTypoHints` in `[한영 자판 변환 참고]`) and validates cross-script layout+typo fixes (`isPlausibleTypoFix` + `validateModelCorrections`, e.g. `ㅗ디ㅐ`→`hello`, `ㅡㅁ가둥`→`markdown`, `네갸ㅜㅎ에서`→`spring에서`, `dkssudgktpd`→`안녕하세요`, `회으에서`→`회의에서`, `아ㄴ녀ㅇ`→`안녕`) before handing off to `SuggestionService` via `'settled'`. **Ultra-fast atomic replacement & delta-rebasing (`2026-10-28`):** `TextInsertService.replaceBeforeCaret` (`method === 'keyboard'`) executes a single atomic `SendInput` call via `sendWin32UnicodeReplacement` (`utils/win32-unicode-input.ts`) packing `(optional VK_SPACE) + N × VK_BACK + M × KEYEVENTF_UNICODE` with `0 ms` JS sleep (native batch ordering only; concurrent input still requires burst deferral), and `TypoCorrectionService` uses `rebaseTypoReplacementPlan` (`packages/core/src/typo-correction.ts`) to adjust `deleteCount` and append any newly typed suffix characters (`<= 12` chars) when the user keeps typing during the 40 ms UIA verification or LLM check. **Terminal, Vim & xterm.js support (`2026-10-27..31`):** `TypoCorrectionService` keeps `layoutOnly: true` active inside `TERMINAL_APPS` (`WindowsTerminal`, `wt`, `pwsh`, `cmd`, `Kitedesk`, `wezterm`, `alacritty`, `kitty`, `ghostty`, `putty`, `MobaXterm`, `nvim-qt`, `neovide`, `gvim`, etc.) and developer/xterm.js surfaces (`isTerminalOrDeveloperSurface`, `isXtermHelperControl`) while keeping spelling/model corrections off. **2026-10-31 terminal character-swallow fix (`v1.15.1`):** removed per-keydown `isForegroundKoreanIme` (`SendMessageTimeoutW(WM_IME_CONTROL)`) from `InputTelemetryService._handleKeyDown` (syncing IME once at idle snapshot via `TerminalKeystrokeTracker.syncKoreanIme`), added an `xterm-helper-textarea` fast path in `sidecar/uia_bridge.py` that skips `GetTopLevelControl()` and `TextPattern` / `TextEditPattern.GetActiveComposition()` calls (which stalled Chromium's renderer up to 1,500 ms and stuck `comp=True`), suppressed 40 ms boundary snapshots on normal Korean words (`_shouldRunFastTerminalBoundarySnapshot`) and 800 ms active-typing polls on terminals, and blocked `TypoCorrectionService` from touching trailing composed Hangul syllables (`/[가-힣]$/u`) in terminals before `Space` is pressed. `TerminalKeystrokeTracker` (`packages/core/src/terminal-prompt.ts`) + `extractTerminalActiveContext` + `isForegroundKoreanIme` (`utils/win32-foreground.ts`, checking `IMC_GETCONVERSIONMODE` `IME_CMODE_NATIVE` on Windows TSF/Chromium) reconstruct the active line across (a) UIA screen-buffer terminals (`WindowsTerminal`/`pwsh`/`cmd`, restoring trailing spaces stripped by grid buffers and reading the bottom viewport via `GetVisibleRanges()` / `[-MAX_TEXT_CHARS:]` and `ExpandToEnclosingUnit(..., waitTime=0)` in `sidecar/uia_bridge.py`), (b) `xterm.js` hidden `<textarea class="xterm-helper-textarea" aria-label="Terminal input">` controls in `Agent Switchboard`, VS Code, Cursor, Hyper, and Tabby (`reconcileWithXtermValue`, preventing the infinite re-correction loop caused by xterm.js keeping `.value` until Enter/Ctrl+C), and (c) canvas/remote terminals with `len=0` (`Kitedesk`/`WezTerm`/`Alacritty`), tracking Vim modes (`insert`, `normal`, `command`). In Vim Insert mode, converts EN↔KO (`dkssudgktpdy `→`안녕하세요 `, `ㅗ디ㅣㅐ `→`hello `) and strips leading mode-entry keys (`i/I/a/A/o/O/s/S`) from raw keystroke buffers; in Vim Command-line mode, converts Hangul ex-commands (`:ㅈㅂ`→`:wq`, `:ㅈㅂ!`→`:wq!`, `:ㅈ`→`:w`, `:ㅂ`→`:q`, `:ㅂ!`→`:q!`, `:ㅌ`→`:x`, `:ㅍ네`→`:vsp`, `:네`→`:sp`, `:ㅜㅐㅗ`→`:noh`) while preserving `:`; in Vim Normal mode (`vimMode === 'normal'`), suppresses all conversions so normal-mode motions (`hjkl`, `dd`, `ciw`) are never touched; in shell prompts, converts CLI/code tokens (`퍄ㅡ `→`vim `, `ㅜ퍄ㅡ `→`nvim `, `ㅎ샤 `→`git `, `ㅣㄴ `→`ls `, `채ㅜㄴㅅ `→`const `). Layout conversions fire after 25 ms idle on word boundaries (Space/punctuation, with a 140 ms quick snapshot timer), 135 ms idle in Vim command mode (`:ㅈㅂ`), or 650 ms idle at token tail (with a 680 ms quick snapshot timer so compound words like `some`→`sometime`/`somebody` are never split mid-word), work even when Chromium/Discord reports `isComposing=true` (passing `commitComposition: true` only when the character before the caret is a composed Hangul syllable `/[가-힣]$/u`, never on single compatibility jamo `[ㄱ-ㅎㅏ-ㅣ]`), ignore synthetic `KEYEVENTF_UNICODE` events (`keycode=0`) while restricting `_programmaticKeys` to synthetic `Backspace`/`Space`/`Ctrl+V`, preserve keys typed during async replacement via `baseLength` in `TerminalKeystrokeTracker.applyReplacement`, retain `_lastCheckedRegion` on successful `_apply` (and only reset on cancellation/failure) while programmatic keys no longer advance `_lastKeyAt`, guard `_watchRevert` against false-positive revert blacklisting across new sentences/cleared fields (and never blacklist loose-jamo strings), preserve `caretReliable=true` after space-terminated replacements (`isEditAtTextEnd` strips trailing whitespace), and reset `UiaContextService._failureStreak` on successful UIA snapshots (`BACKOFF_STEPS_MS` capped at 2.5 s instead of sticky 30 s). Layer 1 (`packages/core/src/typo-correction.ts` + `packages/core/src/araso-layout.ts`): 2-set Korean layout mistypes both ways (`dkssudgktpdy`→`안녕하세요`, `ㅗ디ㅣㅐ`→`hello`, `재깅`→`world`, `ㅗ디ㅣㅐ 재깅`→`hello world`, `ㅗ디ㅣㅐ재깅`→`hello world`, `내ㅡㄷ샤ㅡㄷ`/`some샤ㅡㄷ`→`sometime`, `ㅎ샤ㅗㅕㅠ`→`github`, `ㅎ샤ㅗㅕㅠ에`→`github에`, `ㅠㅕㅎ가`→`bug가`, `ㄴ가`→`s가`, `ㄴ은`→`s은`, `메ㅔ을`→`app을`, `ㅖㅇㄹ파일`→`PDF파일`, `ㅁㅔㅑ키`→`API키`), Hangul-typed English contractions & fuzzy 1-edit typos (`ㅊ무ㅅ`/`ㅊ무'ㅅ`/`ㅊ무ㅜ'ㅅ`→`can't`, `ㅊ무'ㅅ는`→`can't는`, `애ㅜ'ㅅ`→`don't`, `ㅑ'ㅡ`/`ㅑㅡ`→`I'm`, `ㅑㅅ'ㄴ`→`it's`, `ㅗ디ㅐ`/`ㅗ디ㅣㅐㅐ`→`hello`, `ㅔㅐㄴ냐ㅠㅣ`→`possible`, `세갸ㅜㅎ`→`spring`, `ㅡㅁ가애주`→`markdown`, `help[`→`help`, `my self`→`myself`), standalone sentence-initial `ㅑ `→`I ` (`ㅑ ㅊ무'ㅅ `→`I can't `), Araso batchim-agreement & rare-stem Hangul-typed acronyms (`츤를`→`CMS를`, `먀를`→`AI를`, `틔을`→`XML을`, `뗘를`→`EU를`, `넴를`→`SPA를`, `젳를`→`VPC를`, `눈를`→`SNS를` vs `눈을` kept, `운를`→`DNS를` vs `운이` kept, `긴를`→`RLS를`, `쓰은`→`TM은` vs `쓰는` kept, `꾸은`→`RN은` vs `꾸는` kept), 2-letter loose-jamo English words & acronyms (`ㅑㄹ`→`if`, `ㅑㅅ`→`it`, `ㅑㅜ`→`in`, `ㅑㄴ`→`is`, `ㅐㄹ`→`of`, `ㅐㅜ`→`on`, `ㅣㄴ`→`ls`, `ㅎ샤`→`git`, `ㅔㅊ`/`ㅖㅊ`→`PC`, `ㅐㄴ`/`ㅒㄴ`→`OS`, `ㅔㄱ`/`ㅖㄱ`→`PR`, `ㅊㅑ`→`CI`, `ㅍㅡ`→`VM`, `ㅔㅡ`→`PM`, `ㅔㅐ`→`PO`, `ㅗㄱ`→`HR`, `ㅑㄱ`→`IR`, `ㅇㅡ`→`DM`, `ㅡㅇ`→`MD`, `ㅔㅇ`→`PD`, `ㅍㅔ`→`VP`, `ㅠㅡ`→`BM`, `ㅆㅍ`→`TV`, `ㅑㅖ`→`IP`, plus Korean-prefix + loose-jamo compounds `내ㅔㅊ`→`내PC`, `회사ㅔㅊ에서`→`회사PC에서`, `한글로ㅔㅊ라고`→`한글로PC라고`, `윈도우ㅐㄴ`→`윈도우OS`, `새ㅁㅔㅑ를`→`새API를`), contextual `ㅑ`→`I` / `ㅁ`→`a` and `새`→`to` / `애`→`do` / `무`→`an` / `뭉`→`and` / `랙`→`for` / `좨`→`who` / `양`→`did` / `햇`→`got` / `둥`→`end` + lookahead phrase agreement (`새 ㅠㄷ `→`to be `, `새 솓 `→`to the `, `새 애 this `→`to do this `, `랙 솓 `→`for the `, `뭉 솓 `→`and the `, `I want 새 애 this`→`I want to do this`, `into english 새 애`→`into english to do` while preserving `오늘 새 API를 만들었어` and `이거 commit 해`) plus `소두`→`then` / `조두`→`when` / `견`→`run` / `재가`→`work` / `쟈소`→`with` / `소드`→`them` / `소무`→`than` / `미내`→`also` / `소됴`→`they` / `생묘`→`today` / `갸홋`→`right`, ~1,500+ common & extended English stems (`EXTENDED_ENGLISH_STEMS`) with inflectional (`-s`/`-es`/`-ed`/`-ing`/`-ly`/`-er`/`-est`) and derivational morphology (`matchesInflectedOrDerivedEnglish`: `-tion`/`-sion`/`-ation`/`-ment`/`-ness`/`-able`/`-ible`/`-ful`/`-less`/`-ive`/`-ous`/`-al`/`-ic`/`-ize` and prefixes `un-`/`re-`/`pre-`/`non-`/`sub-`/`over-`/`under-`/`auto-`/`multi-`/`micro-`/`inter-`, e.g. `ㅏㅕㅠㄷ구ㄷㅅㄷㄴ`→`kubernetes`, `ㅡㅑㅊ개낸ㅅ`→`microsoft`), partially-converted token recomposition (`안녕gktpdy`→`안녕하세요`, `안녀d`→`안녕`, `github에tj`→`github에서`, `hellㅐ`→`hello`, `hello재깅`→`hello world`), mixed English/Acronym/CamelCase + Korean words/particles with batchim agreement (`githubdp`→`github에`, `PDFvkdlf`→`PDF파일`, `Reactzjavhsjsxm`→`React컴포넌트`, `VSCodetjfwjd`→`VSCode설정`, `dhsmfdmsgoogledptj`→`오늘은google에서`, `1:1ansdml`→`1:1문의`, `TMdms`→`TM은` vs `TMsms`→`쓰는`), lingering Shift on double consonants (`djEJs`→`어떤`, `dlTDj`→`있어`, `qkRNjTdj`→`바꿨어`), chat jamo (`ㅋㅋ`/`ㅎㅎ`/`ㅠㅠ`/`ㅜㅜ`), number+counter conversion (`3tl`→`3시`), sentence-level Korean context propagation (`Stay hungry, stay foolishsms wkqtmdml akfdlek.`→`Stay hungry, stay foolish는 잡스의 말이다.`, `dlrj go`→`이거 해`), URL/path/email run protection (`isUrlOrPathRun`), always-wrong spellings (됬→됐, ㄹ께→ㄹ게, 몇일→며칠, teh→the, dont→don't …), and 1-edit near-misses of the user's frequent English words. Layer 2: warm suggestion model check (`buildTypoCheckPrompt` + `buildModelTypoHints`, re-validated by `validateModelCorrections` + `isPlausibleTypoFix`). Evidence: `packages/core/__tests__/typo-correction.test.ts` 53 tests, `terminal-prompt.test.ts` 13 tests, `suggestion-continuation.test.ts` 26 tests, desktop `TypoCorrectionService.test.ts` 29 tests, `input-flow-services.test.ts` 14 tests, `suggestion-typo-correction.test.ts` 7 tests, plus **live Windows GUI E2E tests (`E2ETestBox.exe` 8/8 + `neovide.exe` Terminal & Vim 6/6 on `WinSta0\Default`)** (GAP-INPUT-08). **2026-10-30 load-safety fix: validate keyboard revision, native input token and foreground again when the insertion queue drains; register synthetic keys only immediately before injection; skip automatic Windows corrections without atomic input support. Partial/unknown SendInput results throw without replay. Automated regressions pass; reported missing Hangul under real CPU/model load is not yet reproduced/verified (GAP-INPUT-11).** **Deterministic A/B evidence (2026-10-30):** Only the three changed services come from HEAD `f49acf0`; dependencies/test doubles/tests stay current in both runs. HEAD 1/17 vs current 17/17 comprises three independent delayed-queue text-corruption cases, 13 snapshot/metadata/coalescing contracts, and one unchanged-input control that passes in both. This is not a full historical-release A/B. `scripts/verification/typing-race-proof.mjs`; `test-results/typing-load/race-proof-2026-10-30T00-50-32-284Z/report.json`. **2026-11-02:** installed 1.16.0 native Korean IME passed 7/7 under bounded 100% CPU and 7/7 with direct `gemma4:e4b` cold-load overlap. The Latin-layout control reproduced next-key/backspace interleaving, so boundary snapshots now wait 140 ms and short-idle partial chains are deferred; after a 650 ms pause the current trailing correction region (up to the core 120-character region) is replaced together. Desktop 2,773 tests/build PASS. Updated-app Latin GUI and physical/Chromium/terminal proof remain open (GAP-INPUT-11). [Evidence](../evidence/2026-11-02-verification.md). |
| INPUT-20 | Context-aware suggestion prompt & graph retrieval | [~] | [-] | [-] | [-] | **2026-10-03; upgraded 2026-10-28.** Prompt (`buildSuggestionPrompt`) splits the prefix with core `writing-context.ts` into earlier text / last sentence / unfinished current sentence and states the mode (finish this sentence vs. write the next one), language and Korean register (formal/polite/plain/casual); adds the text after the caret (no overlap). Graph (`PersonalGraphService`): `indexText` now incrementally creates `shares_terms` edges against recent existing nodes (not only within the same call) and links consecutive short messages within the same app (`_lastIndexedByScope`, 10-min window) via `follows` edges so single-sentence chat messages form a sequence chain; `retrieveContext` accepts `recentWrites` (from `RecentWritesBuffer`) to recover `followUps` even after Enter clears a chat box, augments exact substring continuations (`continuationsFrom`) with particle-invariant stemmed continuations (`stemContinuationsFrom`, e.g. `"오늘 회의록을 정리해서"` → `"내일 오전까지 공유드리겠습니다"` from `"회의록 정리해서 내일 오전까지 공유드리겠습니다"`), and expands 2-hop graph paths (`rankTwoHopGraphTargets` across `shares_terms→follows` and `follows→follows`). Evidence: `personal-graph-context.test.ts` (real SQLite) 7/7, `packages/core/__tests__/personal-graph.test.ts` 19/19, `packages/core/__tests__/personal-ngram.test.ts` 9/9. |
| INPUT-21 | Prefill continuation + document/conversation context for suggestions | [~] | [-] | [-] | [-] | **2026-10-03.** For models whose chat template we know (gemma 4 `<\|turn>`, gemma 2/3 `<start_of_turn>`; core `chat-template.ts`), the suggestion request is sent `raw` with the user's own unfinished sentence (or last sentence) pre-filled into the model turn, so the model literally continues the text: mid-word joins (`회의실`+`에서`) and spacing come out exact (`extractContinuation` keeps joint `attach`/`separate`; trailing space is not pre-filled — it destabilised output — and an attach after a finished word is rejected). Ollama's chat-API prefill was measured and rejected (drops the leading space, closes the turn: `검토해 보시`→`부탁드립니다.`). Other models keep the instruction prompt. Context added to both paths: whole text before the caret (`TypingContext.documentBefore`, ≤6000 chars; null for document-like/terminal/password) → opening lines outside the 1,200-char window and a fact list (names+titles, attendee lines, dates, times, amounts, versions, acronyms, quoted terms — `analyzeDocumentContext`); **same-window messages the user just sent** (chat boxes empty on send — `RecentWritesBuffer`, memory only, 30 min, cleared when input observation stops, terminals excluded); search-box detection (keywords, not sentences); plain `-다` register split from casual `-어`; no emoji unless the user uses them. Output guards: first sentence/line only, stream cut as soon as it ends, restated-prefix stripping (also fixes the instruction path, where `CI는 통과했고…` was inserted after `CI는`), repeat-of-written-sentence drop, and **unsupported-fact rejection** (`findUnsupportedFacts`: dates/numbers/names+titles/lone Latin identifiers absent from document, window title, recent messages and personal hints) with one retry for the first candidate. Fill loop: temperature 1.0, prefill only (the 2026-10-17 change removed the alternating instruction prompt — see INPUT-22). Evidence: real `gemma4:e4b` probe `tests/integration/suggestion-quality.ollama.test.ts` (15 scenarios, opt-in `D3RO_SUGGESTION_PROBE=1`) — first-candidate warnings 0/15 vs 1/15 for the old path (old path invented a launch date instead of the 11월 15일 stated 1,200 chars earlier), mean ≈0.68 s both; core `suggestion-continuation.test.ts` 25, `llm-prompts.test.ts` +7. Not yet verified by typing in a real app (GAP-INPUT-09). |
| INPUT-22 | Typo fixes proposed inside the suggestion popup + snappier suggestions | [~] | [-] | [-] | [-] | **2026-10-17.** While suggestions are on, typos are no longer replaced silently: `TypoCorrectionService.propose()` builds a plan without applying it and `SuggestionService` shows it in the popup as a `오타 고침` row (`del` struck-through original → `ins` fix) with the corrected tail prefixed to every candidate; the model continues from the corrected text. Accept = backspace the typo span + paste fix + candidate in one `replaceBeforeCaret` (if the caret char is still IME-composing, a real Space commits it first and is deleted too); if the text before the caret no longer ends with the typo span, only the candidate is inserted. Silent auto-correct (INPUT-19) still runs when suggestions are off. New core rule `detectJamoNoise`: a lone stray jamo inside/before Hangul syllables (`그ㅓ런`, `ㅇ다른`, `가쓴ㄷ거`) or a lone jamo word mid-sentence is removed; `ㅋㅋ`/`ㅠㅠ`/trailing `ㅎ`/first-word `ㅇ` are kept. Model typo check runs before the first candidate only when `hasSuspiciousTypo` (stray jamo, rare syllable); otherwise once in the background after the first candidate (`thorough`, ≥2 words) and the row is attached late. Proposals also work in learning-excluded apps (agent windows) but not terminals/excluded apps. Responsiveness: `minIntervalMs` 5000→1200, per-minute cap 6→20 (config migration rev 7, IPC clamp 30), composing idle ×1.5→×1.2, settle snapshot fires right after the required idle (≈780 ms instead of 1 s), accept relaxes the interval so the next pause after Space suggests again, visible TTL 20→45 s. Fill loop no longer mixes instruction prompts (instruction-path gemma *answered* the user: "설명해 주실 수 있을까요?"). **Round 2 (same day, live test: "panel should appear the moment I pause; it still closes mid-generation / flashes"):** 15 of 18 first requests returned an empty model output — a prefill ending mid-sentence with no trailing space (`…잠시 쉬면`) makes gemma close its turn at once; each empty result closed the panel and the next snapshot regenerated the same text ~1 s later (flash loop). Fixes: empty raw output → one retry with a trailing space appended to the prefill (`ModelRequest.spaced`, output re-parsed with a leading space so the joint is separate; probe: 4/4 sentences continue; fills reuse the spaced variant); a prefix whose first request failed is not regenerated until the text changes (`_failedPrefix`); a lone trailing jamo is removed with the *following* space so the text still ends "word + space"; snapshot debounce 700→380 ms and a "생성 중" placeholder (`presenceDelayMs` 350, self-clears after `presenceMaxMs` 3 s or when typing resumes, only when a request will follow — warm model, budget, not dismissed, not a failed prefix) shown before generation starts; it is not counted as "visible" by the policy. Tests: `suggestion-presence.test.ts` 5. Recent same-window writes are observed even while IME-composing (Korean chat boxes are almost always composing, so sent messages were never remembered). Diagnosis (installed v1.9.0 log + history DB, 1,122 suggestions / 11 accepted): v1.9.0 predates INPUT-19/21 and cleared a visible popup once `typedRecently` expired 8 s after the last key — ≈ the time the 12-candidate fill takes ("closes by itself when all candidates are done"). Evidence: real `gemma4:e4b` — `안ㄴ뇽하세여.?`→`안녕하세요.?` (1.1 s), `회으에서`→`회의에서`, `괜찬으세요`→`괜찮으세요`, `ㅇ다른`/`그ㅓ런` (0.8 s); desktop `suggestion-typo-correction.test.ts` 7, `TypoCorrectionService.test.ts` +4, core `typo-correction.test.ts` +4; desktop 2556 pass (2 known caption-env failures). Real-app check pending (GAP-INPUT-10). |
---
## SHELL — Platform Shell, Settings, Onboarding, Support
| ID | Feature | D | W | M | B | Anchors / notes |
|---|---|---|---|---|---|---|
| SHELL-01 | Settings / preferences | [x] | [~] | [x] | [x] | **2026-09-28 개편:** 데스크톱 설정은 모달이 아니라 메인 창 전체 화면 라우트(`settings`) — 왼쪽 세로 탭 8개 + 본문 `components/settings/SettingsRows`(구획 h3 + 라벨·설명/컨트롤 행). 일반 탭에 키 바인딩 편집기(CAP-16). 이전 기록: Desktop tabbed modal; the General tab hosts the whole key-binding editor (CAP-16: global on/off switch + one `KeyBindingField` per action, grouped voice/window — `SettingsModal.tsx:239`), which is also the first settings entry point the `command` action ever had; web theme/i18n; mobile `SettingsScreen` |
| SHELL-02 | Theme system (6 themes) | [x] | [x] | [x] | [-] | `theme.ts` SSOT |
| SHELL-03 | i18n (12 locales) | [x] | [~] | [~] | [-] | `@d3ro/i18n`; all 12 catalogs (`ko`, `en`, `ja`, `zh`, `zh-TW`, `es`, `fr`, `de`, `pt`, `ru`, `vi`, `th`) are synchronized to 2,742 keys and guarded by `i18n-locale-parity.test.ts` (15/15 PASS), including the updater dialog keys added on 2026-11-04. Catalog parity is not surface exposure: web has no locale selector and mobile currently exposes ko/en only (`GAP-I18N-03`, SSOT `O-012`), so those surfaces remain `[~]`. |
| SHELL-04 | Onboarding / first-run | [x] | [ ] | [~] | [-] | Desktop model bootstrap (`onboarding.llmModelSize` aligned to 4.0GB `gemma4:e4b`, `GAP-UX-07` `[x]`); mobile audience/theme/locale. **2026-11-01:** 신규 사용자 목적지를 persistence 전에 준비하도록 수정하고 Fold6에서 `.2` fresh auth/signup/invite/small-screen 4/4 PASS (`fold6-20261101-fix1`); latest `.3` first-run auth and retry fail at onboarding visibility (`fold6-20261101-mtg13`). Exact `.3` APK now passes isolated API 34 emulator fresh auth 37 s / signup 50 s with restart persistence; personal Fold6 data was preserved and its separate failures remain open. |
| SHELL-05 | Accessibility / reduced motion | [~] | [~] | [~] | [-] | Desktop reduced-motion honored. Mobile source labels/roles and 48dp tests plus 2026-11-04 inspected-route UIAutomator/font-scale/rotation evidence are partial GREEN; the AdMob SDK descendant exposed one unlabeled clickable WebView. Manual TalkBack/focus order, reduced-motion/app-wide coverage, and iOS remain ([evidence](../evidence/2026-11-04-mobile-runtime-a11y.md)). |
| SHELL-06 | System tray / background | [x] | [-] | [-] | [-] | Desktop tray |
| SHELL-07 | Auto-launch on login | [x] | [-] | [-] | [-] | Desktop only |
| SHELL-08 | Auto-update | [x] | [-] | [!] | [-] | Desktop electron-updater; Forgejo canonical feed; channels + mandatory/major-vs-delta policy (`release/update-policy.json`); mobile store updates **2026-10-24 UI:** updates are now visible in the app, not only through native dialogs. Settings → About opens with an Updates section (`components/settings/UpdateSection.tsx`): status, progress bar, last check, and Check / Download / Restart-to-install buttons. A title-bar chip (`components/shell/UpdateChip.tsx`) appears only while an update is available, downloading or ready, and opens that tab. IPC `appUpdate:getStatus|check|download|install` plus the `appUpdate:statusChanged` event; state comes from `UpdateService.getStatus()`. A skipped version still shows, and downloading it manually clears the skip. Dev and portable builds show "disabled". **2026-11-04:** native consent/restart dialogs resolve `getPopupTranslator()` at each prompt, so a runtime locale change is not cached; ko→en behavior is covered by `update-dialog-i18n.test.ts`. The isolated `--preview-update-dialogs` packaged-QA entry calls the real prompter without updater/feed/install side effects and is covered by entry/prompt tests. The current-source package/ABI passed, but the native dialog appearance and interaction remain an external-desktop gate (`GAP-QA-04`). |
| SHELL-09 | Support / diagnostics | [x] | [ ] | [ ] | [x] | Desktop `SupportModal` (`support-handlers.ts` strict-typed, `GAP-REL-15` `[x]`); admin `/support` fail-closed `content_reports` & support console (`GAP-SHELL-01` `[x]`). |
| SHELL-10 | Download center / releases | [-] | [x] | [-] | [x] | Web `/download`, admin `/releases`, Forgejo |
| SHELL-11 | Landing site / legal pages | [-] | [-] | [-] | [-] | `site/` — the only landing/download/legal/invite/assetlinks surface (Wave 3, 2026-09-26: copies in api-server wwwroot and apps/web removed; 404.html; hero loops a copy of the app recording capsule). Prices/URLs from `packages/core` plan-catalog/web-urls — see `design.md`. **2026-10-28 designpaca audit:** flat header (`Header.tsx` blur removed), WCAG AA contrast verified across all 14 routes (`ko`..`vi`, `privacy`, `terms`, `delete-account`, `accept-invite`), `legal.css` explicit `font-weight: 500/600` (no `>=700` fallback), `accept-invite` polite Korean register (`해요체`) and `--faint`/`--accent` AA contrast. |
| SHELL-12 | Notifications (in-app / desktop) | [x] | [ ] | [x] | [x] | Desktop events; mobile push |
| SHELL-13 | Desktop/mobile information architecture (2026-09-28) | [x] | [-] | [x] | [-] | 1급 홈·기록·회의 + 2급 도구. Desktop: `renderer/navigation/app-navigation.tsx`, `AppSidebar`, status bar removed. Mobile: visible Home/Records/Meetings/More tabs, hidden Record route, floating three-action Record FAB, More tool/account grouping; full Jest and the canonical 320dp dark/411dp light runtime journeys are GREEN (`11` GAP-UX-04). `DashScreen`/`WorksHubScreen` were removed; WorksHub-only local filter/pinning was intentionally retired while every destination remains in tabs/More and user content is untouched. Decision SSOT: root `design.md`. |
| SHELL-14 | Home ("오늘") | [x] | [-] | [x] | [-] | Desktop `pages/HomePage.tsx`: recording key, starts, blockers, summary, recent 6/4, file transcription, bounded usage. Mobile `HomeScreen`: today summary, recent records 5/meetings 3, unified-search and plan entry; canonical 320dp dark and 411dp light runtime journeys GREEN. |
| SHELL-15 | Design system v4 "조용한 계기" | [x] | [x] | [~] | [-] | `packages/ui/src/theme-raw.ts` raw-value SSOT (`--d3-text-inverse` mapped to `accent.ink`); flat surfaces, three text levels with AA contrast, `accent.solid/ink`, `InstrumentPanel`/`SegmentControl` aligned with v4. **2026-10-28 designpaca audit:** `apps/web`, `apps/admin`, `apps/desktop/src/renderer` (main 16 routes/tabs + 6 popups), `site/` (14 routes), and `apps/mobile-rn` audited and fixed (`npm run check:design` 0 violations; Playwright headless Chromium 36/36 routes PASS for contrast, `<11px` floor, `<=600` weight cap, zero blur, single `h1`, labeled controls, and zero horizontal overflow across 320–1440px). `packages/ui-native` mirrors dark/light semantics and exposes 48dp control/numeral/motion tokens. **2026-11-04 Android follow-up:** inspected app-owned Home/Login actions were labeled and >=48dp; API 34 Login was reachable at font scale 2.0 in portrait/landscape with scroll. Mobile stays `[~]` because this is automated route-level evidence, while physical TalkBack/focus order, an SDK-owned unlabeled AdMob WebView descendant, app-wide coverage, and iOS remain ([evidence](../evidence/2026-11-04-mobile-runtime-a11y.md)). |
---
## INFRA — Build, CI, Release, Quality
| ID | Feature | Status | Anchors |
|---|---|---|---|
| INFRA-01 | Monorepo + npm workspaces | [x] | `package.json`; unused `turbo.json` and `pnpm-workspace.yaml` removed in Wave 4 |
| INFRA-14 | Unified entitlement resolver (`@d3ro/core/entitlement`) | [x] | `EntitlementSnapshot` + `resolveEntitlement` map Supabase/desktop-license/.NET sources to one contract; adopted across desktop (`LicenseService`), web (`apps/web/src/lib/entitlement.ts`), and mobile (`apps/mobile-rn/src/lib/entitlement-context.tsx`) (`11` GAP-ID-01/02 `[x]`). |
| INFRA-02 | Shared packages | [x] | `packages/*` |
| INFRA-03 | Desktop build + signed packaging | [x] | `electron-builder.yml`, GitLab `package-windows/macos` |
| INFRA-04 | Mobile CI (debug/E2E/release) | [~] | `.forgejo/workflows/ci.yml`, `.gitlab-ci.yml`, root `test:mobile` → mobile `test:ci` (`jest --runInBand`); production AAB remains external |
| INFRA-05 | .NET API tests | [x] | `apps/api-server.Tests` |
| INFRA-06 | Edge function tests (Deno) | [x] | `server/supabase/functions/**/*.test.ts` (root `tests/` was not an Edge test home and was removed 2026-11-04) |
| INFRA-07 | E2E desktop (playwright) | [x] | `apps/desktop/tests/e2e` (5 protected journeys + shared `support/electron-app.ts` fixture), `apps/desktop/playwright.config.ts`. 2026-11-04: red-team cycle specs and 4 stale feature specs deleted, duplicated launch code consolidated. Policy: `docs/tdd-red/06-repo-application.md`. |
| INFRA-08 | Web tests (vitest + playwright) | [x] | `apps/web/tests` (vitest pure-logic contracts, now part of root `npm test`); `apps/web/e2e` (playwright; `smoke.spec.ts` owns the unauthenticated surface contract — fail-closed redirects + `/download` handoff). Web declares its own `@playwright/test`/`vitest` devDeps instead of using the desktop workspace's hoisted binaries. |
| INFRA-09 | E2E mobile (Maestro + instrumentation) | [~] | `.maestro/`, `androidTest`; emulator API 35 gate in CI. 2026-11-04 current-source dual-ABI E2E APK verifier + state-preserving `adb install -r`/cold launch passed on Fold6 Android 16 and API 34 AVD; physical Notifications status/privacy/recent flow passed 1/1 without a permission command, and automated UIA/font-scale/rotation checks are recorded. Manual TalkBack, permission transition/real FCM receipt, full journey, iOS, and production AAB remain ([evidence](../evidence/2026-11-04-mobile-runtime-a11y.md)). |
| INFRA-10 | Secret scanning / release boundaries | [x] | `scripts/ci/check-no-hardcoded-secrets.mjs`, `verify-mobile-release-*.mjs` |
| INFRA-11 | Docker + NAS deploy | [x] | `docker-compose.nas.yml`, `scripts/deploy-nas.*` |
| INFRA-12 | Cloudflare edge + tunnel | [x] | `server/cloudflare-worker`, Cloudflare Tunnel `kd-nas` |
| INFRA-13 | Site deploy (Cloudflare Pages) | [x] | `.forgejo/workflows/deploy-site.yml`; the unused GitHub Pages workflow was removed in Wave 3 |
| INFRA-17 | Test hygiene gate (TDD policy) | [x] | `scripts/ci/check-test-hygiene.mjs` (`npm run test:hygiene` + `:test`), wired into `.forgejo/workflows/ci.yml` quality and `.gitlab-ci.yml` validate. Rules: `.only`, reason-less `.skip()`, trivial assertions, `isVisible()`-gated assertions, machine paths, `waitForTimeout`, `.e2e.test.ts`, `e2e/` 밖의 `*.spec.ts`, assertion-free files. Applied policy + deletion/consolidation log: `docs/tdd-red/06-repo-application.md`. |
| INFRA-15 | Update & release system | [x] | Canonical Forgejo feed + channels/policy (`release/update-policy.json`, `src/main/update-policy.ts`), canonical publisher `scripts/ci/publish-forgejo-release.mjs`, legacy GitLab mirror; `npm run release:metadata:test`. v1.1.0 was published to Forgejo on 2026-09-15; product version moved to `1.2.0` as a forward-fix with CI-only publication, a same-version re-release guard, and download centers that link the feed instead of repository paths. `1.3.0` (2026-09-18) carries the local-STT fixes; Windows publication still needs the CI signing secrets (`11` GAP-REL-02). |
| INFRA-16 | Desktop STT engine packaging | [x] | `apps/desktop/scripts/setup-sidecar.mjs` + `build-sidecar.mjs`, `electron-builder.yml` `extraResources` (`sidecar-dist/sidecar` → `resources/sidecar`, `resources/ffmpeg`), and `scripts/ci/verify-sidecar-bundle.mjs` run in `package-windows`/`package-macos` before electron-builder. Verified on the real bundle: `sidecar.exe` + `_internal` including `faster_whisper/assets/silero_vad_v6.onnx`, plus a packaged-engine transcription round-trip on GPU. |
| INFRA-17 | 서명 없는 배포 채널 (portable + Scoop) | [x] | `scripts/ci/build-portable.mjs` (95MiB 7z 분할 볼륨 + Scoop 매니페스트), `scripts/ci/publish-portable-release.mjs`, `scripts/local/install-d3ro-voice.ps1`, `bucket/` 버킷, `.forgejo/workflows/portable.yml`; 7z 분할 볼륨(Scoop, 162MiB) + zip 분할 부품(수동 설치, 243MiB, 7-Zip 불필요); updater feed와 분리. 2026-09-18 `portable-1.3.1` 게시 + 실제 설치 검증. |
| INFRA-18 | 로컬 런타임 온디맨드 설치 | [x] | `RuntimeProvisioner`(부품 다운로드 + SHA-256 검증 + tar 해제, `%APPDATA%/d3ro-voice/runtime`), `POST runtime:ensure` / `runtime:progress` IPC, 설정 > STT 상태/내려받기 UI. 설치본에서 엔진/ffmpeg를 분리해 189MB → 90.6MiB, 업데이트 피드 게시 복구. 검증은 전부 디스크에 기록된 파일 기준이며(부품 크기·해시 → 결합본 크기·해시), 부품 다운로드는 최대 3회 재시도한다. 2026-09-18 실제 feed 통합 검증(엔진 94.4MiB/18초, ffmpeg 21.7MiB/5초). |
| INFRA-19 | 네이티브 ABI + updater 설정 게이트 | [x] | `scripts/ci/verify-native-abi.mjs`는 패키징된 `better_sqlite3.node`를 host Node가 로드하면 잘못된 ABI로 실패시키고, 대상별 실제 `uiohook-napi/prebuilds/<platform>-<arch>/uiohook-napi.node`도 필수로 검사한다. `--self-test`가 N-API prebuild 누락/존재 fixture를 고정하며 상대·절대 `--dir` 모두 같은 package를 검증한다. `fix-native-abi.mjs`는 better-sqlite3 주입만 담당한다. GitLab desktop jobs와 Forgejo release/portable scripts가 검증기를 실행하고, installer/portable build는 `resources/app-update.yml`도 필수로 확인한다. 2026-09-18의 잘못된 Node ABI 즉시 종료와 updater 설정 누락 사고를 함께 방지한다. |
| INFRA-20 | Desktop typing race and load verification | [~] | `typing-race-proof.mjs` deterministic baseline 2/17 vs current 17/17. 2026-11-02: installed 1.16.0 native Korean IME passed 7/7 under bounded 100% CPU and 7/7 with direct `gemma4:e4b` cold-load overlap. The Latin-layout control reproduced next-key/backspace interleaving; source now defers bursts for 140 ms and batches the current trailing region after a pause, with desktop 2,773 tests/build GREEN. The fix is published in Windows v1.16.1; installed-app Latin GUI, physical keyboard and Chromium/terminal runs remain. See [evidence](../evidence/2026-11-02-verification.md), GAP-INPUT-11. |
---
## Coverage summary (by surface)
| Surface | `[x]` | `[~]` | `[ ]` | Notable strength | Notable weakness |
|---|---|---|---|---|---|
| Desktop | ~40 | 6 | ~8 | Local AI pipeline, meetings, RAG, conversation, key bindings | Ads stubs, no team admin, no email account; custom-instruction/chain LLM path (AI-04..07) fixed 2026-09-21 but unverified in a running app; cross-device sync rewritten 2026-09-27, production migration deploy pending (GAP-SYNC-05) |
| Web | ~22 | 6 | ~14 | Server-shared data UX, billing, meetings, teams | No local AI, limited knowledge upload/search |
| Mobile | ~40 | 12 | ~18 | Cloud + native recording, portability, admin, IAP/ads | External store/console gates, a11y, deep E2E pending |
| Backend | ~45 | 6 | ~4 | RLS, Edge functions, billing, fail-closed AI | Payple webhook signature, some external provider keys |