yunchan/d3ro-voice
1
0
Fork 0
Code Issues Pull requests Projects Releases 3 Packages Wiki Activity Actions

release: ship v1.5.0 with on-device writing suggestions
Some checks failed
deploy-site / deploy (push) Failing after 33s
Details
portable-unsigned / portable-windows (push) Failing after 4m7s
Details
release / release-windows (push) Failing after 3m16s
Details

Browse source 
Adds next-sentence suggestions while typing, weekly input insights and a
personal phrase memory to the desktop app, and fixes custom instructions so
they process the text instead of inserting the instruction's own wording.
Local model requests are now bounded and individually cancellable.

Bumps the product version to 1.5.0 (Android/iOS build 1050000), refreshes the
landing and web download links, and records the new INPUT feature rows and the
open verification gaps in the infrastructure map.
This commit is contained in:
Yun Chan 2026-09-23 16:04:27 +09:00
parent 99f06c253c
commit 5c11ee2fde
104 changed files with 14410 additions and 174 deletions
Download patch file Download diff file Expand all files Collapse all files

8
docs/map/00-index.md
View file

@ -2,8 +2,12 @@
> Status: ACTIVE
> Last full audit: 2026-09-13
> Last update: 2026-09-21 — LLM instruction-prompt fix (`9c2b4d4`): the custom-instruction path inserted the instruction's own wording instead of the processed result and had **never worked in any shipped release** (`v0.1.0-alpha`..`v1.4.0`, introduced `fea923d` 2026-04-05, not a regression). `llm-prompts.ts` is now the SSOT for prompt resolution and placeholder substitution, shared by `VoiceModeService` / `ChainService` / `LLM.PROCESS`. AI-04/05/06/07 are demoted to `[~]` on desktop — fixed with unit tests, but **not verified in a running app** and the four related `tests/red/*.usecase.test.ts` could not execute (`better-sqlite3` ABI). New: GAP-LLM-01 (no target-language setting), GAP-LLM-02 (this fix unverified); GAP-INFRA-06 amended (the ABI masks verification, not just dev-env switching cost); GAP-I18N-01 amended (`popup.error.default` missing in 10 locales). Earlier the same day: CAP-16 (desktop key bindings rebuilt on one `@d3ro/core/keybinding` SSOT — multiple bindings per action, mouse buttons, `HOTKEY` → `KEYBINDING` IPC group), verified on Windows by a manual run, so CAP-16 and CAP-02 are `[x]` and GAP-KEY-01 is closed. Still open: GAP-KEY-02/03, GAP-QA-02, GAP-I18N-01/02, GAP-INFRA-06, GAP-LLM-01/02; `11` §7 holds accepted design constraints (things deliberately kept, not gaps)
> Scope: entire monorepo `D:/workspace/D3ROVoice` at product version `1.4.0` (`release/product-version.json`, released 2026-09-21)
> Last update: 2026-09-23 — **v1.5.0 릴리스.** CHANGELOG `[1.5.0]` 을 확정하고 버전 SSOT를 1.5.0(android/iOS 1050000)으로 올렸으며, `site/src/release.ts` 와 `apps/web/src/lib/desktop-release.ts` 다운로드 링크를 1.5.0으로 동기화했다. 이 릴리스는 입력 인텔리전스(`INPUT-01`~`INPUT-18`, 전부 데스크톱 `[~]`), 커스텀 인스트럭션 수정(AI-04/05), `LocalLLMService` 요청별 취소·상한을 포함한다. 인증서가 없어 무서명 업데이터 게시 예외(GAP-REL-06)를 유지한다.
>
> Previous update: 2026-09-22 — **Gemma/Ollama 폭주 방어 계약을 기록했다.** 19:14:12 부팅 워밍업이 `keep_alive: 30m`으로 `gemma4:e4b`를 19:44:12까지 VRAM 3,226,342,521 bytes / context 4096으로 강제 상주시킨 것이 관측됐으며, 같은 시점 Windows GPU Engine PID 표본에는 Ollama의 활성 compute가 없었다. 즉 당시 상태는 무한 추론이 아니라 강제 residency였다. 19:11:17~19:11:58의 자동 제안 연속 생성은 기존 900 ms·12/min·5 candidates·128 tokens·한 글자 재생성 정책이 허용한 burst였다. 현재 구현 계약은 부팅 warmup 제거, `keep_alive: 2m`, 제안 600 ms / 최소 5 s 간격 / 기본 6회·hard max 12회 per min / 3 candidates / 64 tokens / 12-char growth / 8 s timeout, 그리고 요청별 취소·상한·종료 정리다. 독립 표적 검증은 6 test files / 69 tests passed / 0 failed, 변경 코드·테스트 ESLint와 `git diff --check`도 exit 0이다. Raw Ollama에서는 cold bounded 요청이 client hard timeout 15.044 s에 취소된 뒤 `/api/ps`가 비었고 `/api/version`은 80 ms에 회복했다. 명시 warmup은 HTTP 200 / 16.639 s, 후속 warm 요청은 body `options.num_predict=1`, `keep_alive='2m'`로 553 ms HTTP 200 / `done:true` / `eval_count:1` / `done_reason:length`였고 `/api/ps` expiry는 약 119.9 s였다. **19:48:59 +09:00에는 새 generate/unload/kill/retry 없이 충분히 지난 뒤 한 번의 `/api/ps`가 HTTP 200 / 45.8 ms / `{models:[]}`였고 `/api/version`은 HTTP 200 / 7.3 ms / `0.32.13`이었다.** 이는 raw API 수준의 expiry 뒤 unload 확인일 뿐 앱 재시작·GUI·실제 타이핑 증거는 아니므로 상태는 `[~]`로 유지한다 (`11` GAP-LLM-04, GAP-INPUT-06).
>
> Previous update: 2026-09-21 — **Input intelligence (입력 레메트리 + 다음 문장 제안) 신규**. 데스크톱에 입력 수집기(`InputTelemetryService`), UIA 컨텍스트 브리지(사이드카 `GET /uia/focus`), 제안 서비스(`SuggestionService`), 어렛 커 오버레이, 설정 > 입력 탭(동의·정책·주간 인사이트·개인 문구)을 추가했다. 카탈로그에 `INPUT-01`~`INPUT-08`(전부 데스크톱 `[~]` — 유닛 48건은 GREEN 이지만 **실앱 타이핑 검증 전**), 백로그에 GAP-INPUT-01~05 + GAP-LLM-03, §7 에 CONSTRAINT-INPUT-01(키 내용 미저장 — ActivityWatch 정책 채택)을 기록. 설계 근거는 조사 기반이다: 어렛은 `GetGUIThreadInfo` 가 아니라 UIA `TextPattern.GetSelection`(Chromium 은 `TextPattern2` 미구현), 타이핑 스트는 키코드 복원이 아니라 UIA 스냅샷 diff(한/일 IME 대응), 디바운스/토큰 한도는 인라인 컴플리션 실측값(Continue 350 / Tabby 250 / twinny 300 ms, 출력 64~256 토큰). 의존성: `koffi` 3.3.1(포그라운드 창 FFI), 사이드카 `uiautomation` 2.0.29 + `comtypes`. 데스크톱 유닛 총계 1409(+49), Electron ABI 실행에서 신규 실패 0건. 당시의 24.7초/4.9초 지연 설명과 `keep_alive: 30m`·부팅 워밍업 처방은 **현재 상태가 아닌 과거 가설/완화 이력**이며, 최신 운영 결론은 위 2026-09-22 항목과 `11` GAP-LLM-04를 따른다. 직전: LLM instruction-prompt fix (`9c2b4d4`): the custom-instruction path inserted the instruction's own wording instead of the processed result and had **never worked in any shipped release** (`v0.1.0-alpha`..`v1.4.0`, introduced `fea923d` 2026-04-05, not a regression). `llm-prompts.ts` is now the SSOT for prompt resolution and placeholder substitution, shared by `VoiceModeService` / `ChainService` / `LLM.PROCESS`. AI-04/05/06/07 are demoted to `[~]` on desktop — fixed with unit tests, but **not verified in a running app** and the four related `tests/red/*.usecase.test.ts` could not execute (`better-sqlite3` ABI). New: GAP-LLM-01 (no target-language setting), GAP-LLM-02 (this fix unverified); GAP-INFRA-06 amended (the ABI masks verification, not just dev-env switching cost); GAP-I18N-01 amended (`popup.error.default` missing in 10 locales). Earlier the same day: CAP-16 (desktop key bindings rebuilt on one `@d3ro/core/keybinding` SSOT — multiple bindings per action, mouse buttons, `HOTKEY` → `KEYBINDING` IPC group), verified on Windows by a manual run, so CAP-16 and CAP-02 are `[x]` and GAP-KEY-01 is closed. Still open: GAP-KEY-02/03, GAP-QA-02, GAP-I18N-01/02, GAP-INFRA-06, GAP-LLM-01/02, GAP-INPUT-01~05; `11` §7 holds accepted design constraints (things deliberately kept, not gaps)
> Scope: entire monorepo `D:/workspace/D3ROVoice` at product version `1.5.0` (`release/product-version.json`, released 2026-09-23)
> Purpose: let any agent (or human) answer two questions in under a minute:
> 1. **What infrastructure exists?** (build, CI, services, APIs, data, packages, deploy)
> 2. **How far is each feature developed?** (per surface, with file anchors and status)

4
docs/map/02-infrastructure.md
View file

@ -60,6 +60,8 @@ Note: root `package.json` declares npm workspaces `apps/desktop`, `apps/web`, `a
| Deno | 2.8.1 | CI (`edge-functions-quality`) |
| JDK | 17 | mobile CI |
| Electron | 33.4.11 | `apps/desktop/electron-builder.yml` |
| koffi | 3.3.1 | `apps/desktop/package.json` — FFI into user32/kernel32 for foreground window title/pid/exe/bounds (`utils/win32-foreground.ts`). Chosen over `get-windows` because that package needs an install script this repo does not run; koffi ships N-API 8 prebuilds as optional deps. `asarUnpack` covers `koffi`/`@koromix`. |
| uiautomation / comtypes | 2.0.29 / 1.4.17 | `apps/desktop/sidecar/requirements.txt` (Windows only) — UI Automation snapshot behind `GET /uia/focus` (`sidecar/uia_bridge.py`). Dev install: `apps/desktop/sidecar/.venv/Scripts/python.exe -m pip install uiautomation comtypes`. |
| React Native | 0.85 | `apps/mobile-rn/package.json` |
| Turborepo | turbo.json tasks: build/typecheck/test/lint/dev | `turbo.json` |
@ -192,7 +194,7 @@ Full detail: [`09-supabase-backend.md`](./09-supabase-backend.md).
| File | Purpose |
|---|---|
| `release/product-version.json` | version `1.4.0`, `androidVersionCode`/`iosBuildNumber` `1041000`, releaseDate `2026-09-21`, desktop license keyId |
| `release/product-version.json` | version `1.5.0`, `androidVersionCode`/`iosBuildNumber` `1050000`, releaseDate `2026-09-23`, desktop license keyId |
| `release/android-release-identity.json` | package `com.d3ro.voice`, Play app ID, app-signing SHA-256, upload cert SHA-256, evidence keyId, AdMob unit IDs |
| `release/desktop-license-public.pem` | Ed25519 public key for desktop offline licenses |
| `release/mobile-release-evidence-public.pem` | Ed25519 public key for mobile release evidence |

5
docs/map/03-shared-packages.md
View file

@ -15,8 +15,9 @@ The canonical place for types and cross-surface logic. Both desktop and web/mobi
|---|---|---|
| Types | `./types` | Domain types shared across surfaces |
| Errors | `./errors` | `D3ROError`, `ErrorCode` |
| IPC channels | `./ipc-channels` | `IPC_CHANNELS` object + `IPCChannel` union. **SSOT** for every desktop IPC channel (VOICE, AUDIO, STT, TTS, LLM, KEYBINDING, CONFIG, HISTORY, DICTIONARY, WINDOW, SYSTEM, STATS, MEMO, VOICE_COMMAND, CONTEXT, CHAIN, CAPTION, FILE_TRANSCRIPTION, MEETING_SUMMARY, DICTATION_TEMPLATE, VOICE_CONVERSATION, RAG, VOICE_ACTION, MEETING_MODE, MEETING_DOC_TEMPLATE, MEETING_CHAT, LICENSE, CLOUD_SYNC, INSTRUCTION, SYSTEM_AUDIO, POPUP_RESULT, POPUP_HISTORY, POPUP_COMMAND, POPUP_CAPTION, VOICE_PARTIAL, CLIPBOARD, APP, ONLINE_AUTH, ADS, SUPPORT, PAYMENT) |
| Key bindings | `./keybinding` | **SSOT** for every global shortcut in the app: `KeyBinding` (`device`/`code`/`ctrl`/`alt`/`shift`/`meta`), `KEY_CATALOG` (10 selectable groups incl. mouse), `KEYBINDING_ACTIONS` (6 rebindable actions), `bindingKey`/`normalizeBinding`/`validateBinding`/`detectBindingConflicts`/`formatBindingSegments`/`searchKeyCatalog`/`parseBindingMap`. Persisted as `AppConfig.keyBindings`. i18n keys are exposed as plain `string` so core stays independent of `@d3ro/i18n`; consumers narrow at the boundary (`asTranslationKey`) and a contract test guards the keys — accepted constraint, `11` §7 CONSTRAINT-I18N-01. Tests: `__tests__/keybinding*.test.ts` via `vitest.config.ts` (`npm run test --workspace=@d3ro/core`), 117 cases as of 2026-09-21 |
| IPC channels | `./ipc-channels` | `IPC_CHANNELS` object + `IPCChannel` union. **SSOT** for every desktop IPC channel (VOICE, AUDIO, STT, TTS, LLM, KEYBINDING, CONFIG, HISTORY, DICTIONARY, WINDOW, SYSTEM, STATS, MEMO, VOICE_COMMAND, CONTEXT, CHAIN, CAPTION, FILE_TRANSCRIPTION, MEETING_SUMMARY, DICTATION_TEMPLATE, VOICE_CONVERSATION, RAG, VOICE_ACTION, MEETING_MODE, MEETING_DOC_TEMPLATE, MEETING_CHAT, LICENSE, CLOUD_SYNC, INSTRUCTION, SYSTEM_AUDIO, POPUP_RESULT, POPUP_HISTORY, POPUP_COMMAND, POPUP_CAPTION, VOICE_PARTIAL, CLIPBOARD, APP, ONLINE_AUTH, ADS, SUPPORT, PAYMENT, INPUT_TELEMETRY, SUGGESTION, POPUP_SUGGESTION) |
| Key bindings | `./keybinding` | **SSOT** for every global shortcut in the app: `KeyBinding` (`device`/`code`/`ctrl`/`alt`/`shift`/`meta`), `KEY_CATALOG` (10 selectable groups incl. mouse), `KEYBINDING_ACTIONS` (9 rebindable actions incl. the three suggestion actions), `bindingKey`/`normalizeBinding`/`validateBinding`/`detectBindingConflicts`/`formatBindingSegments`/`searchKeyCatalog`/`parseBindingMap`. Persisted as `AppConfig.keyBindings`. i18n keys are exposed as plain `string` so core stays independent of `@d3ro/i18n`; consumers narrow at the boundary (`asTranslationKey`) and a contract test guards the keys — accepted constraint, `11` §7 CONSTRAINT-I18N-01. Tests: `__tests__/keybinding*.test.ts` via `vitest.config.ts` (`npm run test --workspace=@d3ro/core`), 117 cases as of 2026-09-21 |
| Input intelligence | `./input-intelligence` | **SSOT** for the input-telemetry and next-sentence-suggestion domain (added 2026-09-21): key classification (`classifyKeyStroke`), text metrics (`countWords`/`countSentences`/`endsSentence`/`textBeforeCaret`), `computeTypedDelta` (UIA snapshot diff — the IME-safe way to count typed text), `decideSuggestion` + `isAppExcluded` (when to request / skip / clear), `parseSuggestionCandidates`/`sanitizeSuggestionLine` (prompt-leak and prefix-echo defence), `anchorFloatingPanel` (caret-anchored overlay placement), `mergeActivityBucket`/`summarizeActivity`/`pixelsToMeters`, `extractPhrases`/`selectPhraseHints`, `INPUT_TELEMETRY_DEFAULTS`/`SUGGESTION_DEFAULTS` |
| Constants | `./constants` | Shared constants |
| Crypto license | `./utils/crypto-license` | Ed25519 license sign/verify (used by admin issuer + desktop verifier) |
| PII | `pii-redactor`, `secure-memory` | Redaction + secure memory helpers |

45
docs/map/04-desktop-app.md
View file

@ -45,7 +45,7 @@ Singleton + `EventEmitter` pattern (`getXService()` accessors).
### LLM layer
| Service | Purpose |
|---|---|
| `LocalLLMService` | Ollama REST (models, pull w/ progress, server start, NDJSON streaming) |
| `LocalLLMService` | Ollama REST (models, pull w/ progress, server start, NDJSON streaming). The 2026-09-22 runaway guard gives each request its own `AbortController` (external caller signals are relayed and detached on completion), requires `done` before an NDJSON stream succeeds, and clears incomplete streams. Generation/stream requests are bounded to 2048 tokens / 120 s; chat is bounded to 512 / 60 s. Ollama server spawn and polling are deduplicated, and lifecycle dispose terminates owned work. |
| `PremiumLLMService` | Claude via Supabase `llm-proxy`, local fallback |
| `OnlineLLMService` | JWT-authenticated .NET backend client |
| `llm-prompts.ts` | **SSOT for prompt resolution, placeholder substitution, and argument placement.** `resolveSystemPrompt` (`:118`) maps an `LLMAction` to its base prompt and handles `custom` explicitly instead of dropping silently to `refine`. `renderInstructionPrompt` (`:78`) substitutes `{{text}}` / `{{userPrompt}}` / `{{targetLanguage}}` and **warns by name** for any placeholder left standing rather than letting it reach the model. `buildInstructionInvocation` (`:101`) decides where an instruction goes in `processText(text, action, targetLanguage, customPrompt)`: the instruction becomes the **system prompt** and the transcript the **text**, except for instructions that spell out `{{text}}`, which keep the old meaning for backward compatibility. `resolveTargetLanguage` (`:50`) is the one place translate targets are decided (still `English`, see `11` GAP-LLM-01). **All three LLM entry paths call the same functions** — `VoiceModeService` (`:838`), `ChainService` (`:196`), and the `LLM.PROCESS` IPC handler (`llm-handlers.ts:96`) — so no caller re-implements the rules |
@ -69,7 +69,7 @@ Singleton + `EventEmitter` pattern (`getXService()` accessors).
| `FileTranscriptionService` | Audio/video file → ffmpeg → 30s chunks → STT merge (events progress, complete, error, state-changed) |
| `MeetingSummaryService` | Post-caption LLM summary |
| `DictationTemplateService` | Field-by-field voice form filling |
| `VoiceConversationService` | STT→LLM→TTS loop, 10-turn memory |
| `VoiceConversationService` | STT→LLM→TTS loop, 10-turn memory. Local conversations are single-flight and pass their request signal through to local LLM chat, so a voice cancel aborts its own active request rather than only changing UI/watchdog state. |
| `TTSPlaybackService` | Platform TTS (macOS `say`, Windows SAPI), sentence queue |
| `VoiceActionService` | Voice → LLM JSON action plan → OS execution (dangerous blocked) |
@ -79,6 +79,18 @@ Singleton + `EventEmitter` pattern (`getXService()` accessors).
| `MeetingModeService` | Meeting recording: live transcript, timestamp memos, doc generation/export, diarization |
| `MeetingDocTemplateService` | Meeting-doc templates (built-ins + CRUD) |
### Input intelligence (2026-09-21; Local Flow Intelligence extension 2026-09-22)
| Service / file | Purpose |
|---|---|
| `InputTelemetryService` | Global input telemetry: keystroke/click/scroll counters, mouse travel (Manhattan sum of per-axis deltas), active time, per-hour-per-app buckets flushed every 5 s; typed-text learning via UIA snapshot diffs. It extends the existing telemetry store with Flow Radar (hourly activity density + chars + edit stability; a potential-focus score, **not** a real session), Edit Friction (chars/backspaces; edits per 100 chars), App Quality (per-app suggestion total/accepted/accept rate/average latency), Privacy Receipt (actual row counts and retention), and Smart Exclusion evidence. Raw key events, key codes and content streams are never retained; separately, opt-in learned text may be retained in `typing_samples` and `personal_phrases` under the stated receipt policy. Consent is opt-in (`inputTelemetryEnabled`, `inputLearnTypedText`). |
| `UiaContextService` | Client for the sidecar UIA bridge (`GET /uia/focus`): focused text, caret rect/offset, `IsPassword`, IME composition state and `hasSelection`. The sidecar derives `hasSelection` only by comparing the TextPattern selection range Start/End endpoints: it does not call `GetText` on that range or add a separate selected-text payload (the existing focused-field text can still include a selection). Fail-closed on password/unavailable; transient failures back off for 1.5/3/6/12/30 s and permanent failures for 5 min. |
| `SuggestionService` | Next-sentence ghost text: 600 ms debounce, then `buildSuggestionPrompt` → `LocalLLMService.streamGenerate` with a caller-owned `AbortSignal` → candidate ranking and accept (via `TextInsertService`). Presentation-active includes candidates plus `generating`, `warmingUp` and `partialText`; clear/dismiss aborts the active request, invalidates its generation token, clears TTL state, then emits `cleared`/hide so a spinner-only popup closes. A successful final candidate payload resets `generating=false` and `partialText=null`. App DNA extends phrase ranking with `appName` context and a 1.75× same-app bonus; Memory Decay uses a 30-day half-life plus frequency. Why This Suggestion exposes only `local-model`/`local-memory` provenance and continuation/related/phrase/appPhrase **counts** to the overlay, never raw evidence text. Instant Recall uses local-memory provenance without another model-budget spend only when the model is unavailable, raises a non-cancellation error, truly times out, or returns empty; it never publishes after dismiss, new typing, token/context mismatch or staleness. Its runaway limits are a 5 s minimum interval, 6 requests/min default (hard-config maximum 12), 3 candidates, 64 output tokens, 12-character regeneration growth and an 8 s timeout; app startup does not warm the model and suggestion requests use `keep_alive: 2m`. |
| `global-input-hook.ts` | Ref-counted owner of the single process-wide `uiohook` hook so `KeyBindingService` and telemetry can both attach without one stopping the other. |
| `utils/win32-foreground.ts` | Foreground window title/pid/exe/bounds via `koffi` FFI into user32/kernel32 (chosen over `get-windows`, which needs an install script this repo does not run). |
| `sidecar/uia_bridge.py` | Windows UIA snapshot (`uiautomation` 2.0.29, comtypes) on a dedicated COM-initialised thread with a 1.5 s budget; never walks the UIA tree (Chrome/VS Code tree walks take 10-30 s). Warm calls measure 0-15 ms. |
The existing `input-telemetry-handlers` and `suggestion-handlers` IPC extensions expose the receipt and the flow/suggestion summaries. Receipt reads or deletes report an IPC error when storage access fails; they do not return invented counts or claim a purge succeeded. The receipt is local-only: `input_activity`, `typing_samples`, and `suggestions` retain 30 days; `personal_phrases` has no age-based automatic expiry and is removed by individual deletion, delete-all, or consent withdrawal. Smart Exclusion never records password evidence and recommends only the current app after at least four observations, zero readable results, and a problematic ratio of at least 75%; its one-click action adds an exclusion and never auto-excludes. Shortcut Safety Audit is core `auditKeyBindingMap` applied by the Settings UI; it surfaces invalid/conflict issues while preserving the existing hold/double-press exceptions.
### Account / infra / monetization
| Service | Purpose |
|---|---|
@ -104,7 +116,7 @@ Singleton + `EventEmitter` pattern (`getXService()` accessors).
## 3. IPC layer
Registry: `src/main/ipc/index.ts` calls 29 `registerXHandlers()` in fixed order. Channel SSOT: `packages/core/src/ipc-channels.ts`.
Registry: `src/main/ipc/index.ts` calls 31 `registerXHandlers()` in fixed order. Channel SSOT: `packages/core/src/ipc-channels.ts`.
| Handler | Channel group(s) |
|---|---|
@ -118,6 +130,7 @@ Registry: `src/main/ipc/index.ts` calls 29 `registerXHandlers()` in fixed order.
| `dictionary-handlers` | DICTIONARY |
| `file-transcription-handlers` | FILE_TRANSCRIPTION |
| `history-handlers` | HISTORY + `stats:getSummary` |
| `input-telemetry-handlers` | INPUT_TELEMETRY |
| `instruction-handlers` | INSTRUCTION |
| `keybinding-handlers` | KEYBINDING |
| `license-handlers` | LICENSE |
@ -129,6 +142,7 @@ Registry: `src/main/ipc/index.ts` calls 29 `registerXHandlers()` in fixed order.
| `payment-handlers` | PAYMENT |
| `rag-handlers` | RAG |
| `stt-handlers` | STT |
| `suggestion-handlers` | SUGGESTION + POPUP_SUGGESTION |
| `support-handlers` | SUPPORT |
| `system-handlers` | SYSTEM |
| `template-handlers` | DICTATION_TEMPLATE |
@ -142,19 +156,20 @@ The **`KEYBINDING`** group replaced the old per-action `HOTKEY` group. `HOTKEY`
**`LLM.PROCESS` normalizes at the IPC boundary.** The handler runs `buildInstructionInvocation` itself when `action === 'custom'` with a `customPrompt` (`llm-handlers.ts:94-108`), so the renderer passes the **raw instruction text** and never duplicates the substitution or argument-placement rules. This is what makes `VoiceModeService`, `ChainService`, and `LLM.PROCESS` literally share one implementation. No channel or type changed for this; `LLMProcessParams` is unchanged.
Preload exposes **`window.electronAPI`** with 33 namespaces: `platform, audio, config, voice, stt, keybinding, llm (incl. premium), history, dictionary, stats, window, system, instruction, app, memo, voiceCommand, context, chain, caption, license, fileTranscription, meetingSummary, dictationTemplate, rag, voiceAction, voiceConversation, meetingMode, meetingChat, meetingDocTemplate, cloudSync, onlineAuth, ads, support, payment`. The `keybinding` bridge is 9 methods mirroring the channels above (`src/preload/index.ts:323`), replacing the 11-method `hotkey` bridge. Envelope: `IPCResult<T>` (success/error); `app.onDataChanged` is the global refresh channel.
Preload exposes **`window.electronAPI`** with 35 namespaces: `platform, audio, config, voice, stt, keybinding, llm (incl. premium), history, dictionary, stats, window, system, instruction, app, memo, voiceCommand, context, chain, caption, license, fileTranscription, meetingSummary, dictationTemplate, rag, voiceAction, voiceConversation, meetingMode, meetingChat, meetingDocTemplate, cloudSync, onlineAuth, ads, support, payment, inputTelemetry, suggestion`. The `keybinding` bridge is 9 methods mirroring the channels above (`src/preload/index.ts:323`), replacing the 11-method `hotkey` bridge. Envelope: `IPCResult<T>` (success/error); `app.onDataChanged` is the global refresh channel.
---
## 4. Windows & popups
`windows/WindowManager.ts` creates 6 windows: main (borderless, custom TitleBar; macOS `hiddenInset`), recording-tip, result-popup, history-popup, command-popup, caption-overlay. Injects popup theme CSS + i18n strings; 2-phase resize. `windows/TrayManager.ts` — tray icon + menu + double-click show.
`windows/WindowManager.ts` creates 7 windows: main (borderless, custom TitleBar; macOS `hiddenInset`), recording-tip, result-popup, history-popup, command-popup, caption-overlay, suggestion-overlay. Injects popup theme CSS + i18n strings; 2-phase resize. `windows/TrayManager.ts` — tray icon + menu + double-click show.
**Popup invariants** (each shipped broken once — do not regress):
- 팝업 HTML의 스크립트는 반드시 `<script type="module">`로 선언한다. Vite는 모듈 스크립트만 번들에 포함하므로 classic `<script src="./script.js">`는 dev에서만 로드되고 패키징 산출물에서는 파일 자체가 사라진다(오버레이가 정적 HTML로 멈춘 원인). `scripts/ci/verify-desktop-renderer-bundles.mjs`가 빌드 HTML이 참조하는 모든 로컬 asset의 존재를 검사한다.
- 렌더러 로드 전의 `webContents.send`는 조용히 버려진다. 팝업 전송은 `sendToPopupWindow`를 쓰고, 이 함수가 `did-finish-load`까지 메시지를 보관했다가 전달한다. `attachPopupLifecycle`이 로드 상태 추적·테마 주입·팝업 렌더러 진단 로그를 한 곳에서 묶는다.
- 팝업 표시는 `presentPopup`으로 통일한다(`showInactive` + topmost 재선언 + `moveTop` + `webContents.invalidate`). 한 번 `hide()`된 팝업이 두 번째 표시에서 z-order/repaint를 잃어 보이지 않던 문제를 막는다.
- `suggestion-overlay`는 비축소 UIA 선택이 생기면 `selection-active`로 즉시 clear한다. 실제 표시 중일 때만 5초 이후에도 UIA를 주기 검증하고 mouse-up 뒤 120 ms에 다시 검증한다. 입력 focus가 사라져 `available`/`editable`이 false가 되거나 선택이 생기면 요청을 abort하고 hide한다. X는 renderer panel을 먼저 즉시 숨긴 뒤 main IPC가 `BrowserWindow.hide()`를 직접 호출하고 service dismiss를 수행한다. 늦은 결과는 generation token으로 재표시할 수 없다.
Vanilla popups (`src/renderer/popups/`):
| Popup | Purpose |
@ -164,6 +179,7 @@ Vanilla popups (`src/renderer/popups/`):
| `history-popup` | Recent transcriptions; ↑↓/Enter/1-9/ESC. Opened by the `history-popup` action (default `Ctrl+Shift+V`, rebindable) |
| `command-popup` | Command selection. Opened by the `command-popup` action (default `Ctrl+Shift+C`, rebindable) |
| `caption-overlay` | Live caption overlay (font/opacity/maxLines) |
| `suggestion-overlay` | Next-sentence ghost text: caret-anchored (`anchorFloatingPanel`), non-focusable, click-through unless `suggestionOverlayInteractive`; accept/next/dismiss come from global key bindings (the window never owns focus). Its narrow layout shows a one-line local provenance/count summary only, never raw memory evidence. Presentation includes candidate, generating, warm-up and partial-text states; its X hides the panel before main-process dismissal. |
---
@ -181,21 +197,24 @@ Routing is state-based in `AppLayout.tsx` (`Route` union + `NAV_ITEMS`), no reac
| `KnowledgeBasePage` | knowledge | Local RAG: add/index docs, semantic query, reindex/remove |
| `MeetingModePage` | meeting | Meeting studio: live transcript, memos, doc generation/export, diarization |
Modals/components: `SettingsModal` (tabs General/Audio/STT/LLM/License/Cloud/About), `LicenseModal`, `LicenseTab`, `CloudSyncSection`, `OnboardingModal`, `UpgradePromptModal`, `ProBadge`, `TemplateSection`, `FileDropZone`, `OllamaGuideModal`, `CodexOAuthGuideModal`, `TitleBar`, `StatusBar`, meeting components (9), voice-conversation, payment (`CheckoutModal`, `checkout-flow.ts`), support (`SupportModal`), ads (`AdBanner`, `RewardedQuotaModal`), shared cards.
Modals/components: `SettingsModal` (tabs General/Audio/STT/LLM/Input/License/Cloud/About), `LicenseModal`, `LicenseTab`, `CloudSyncSection`, `InputInsightsPanel` (consent + suggestion policy + weekly insights + learned phrases), `InputConsentPanel` (receipt + current-app exclusion recommendation), `InputInsightsView` (flow, friction and app-quality summaries), `OnboardingModal`, `UpgradePromptModal`, `ProBadge`, `TemplateSection`, `FileDropZone`, `OllamaGuideModal`, `CodexOAuthGuideModal`, `TitleBar`, `StatusBar`, meeting components (9), voice-conversation, payment (`CheckoutModal`, `checkout-flow.ts`), support (`SupportModal`), ads (`AdBanner`, `RewardedQuotaModal`), shared cards.
Key-binding UI lives in `components/keybinding/` (`Keycap`, `KeyBindingPicker`, `KeyBindingField`, `translation-key`), embedded in the Settings **General** tab (`SettingsModal.tsx:239`) — one field per action plus a global on/off switch. The picker offers both key recording and a searchable grouped dropdown (MUI `Autocomplete` over `KEY_CATALOG`, `KeyBindingPicker.tsx:536`). It replaced `HotkeyRecordModal`. `renderer/utils/format-hotkey.ts` is now a 17-line platform adapter only; key names, modifier glyphs, and join rules come from `@d3ro/core/keybinding`.
Hooks: `useRealtimeConversation` (OpenAI Realtime WebRTC), `useLicenseState`, `useProFeature`, `useKeyBindingMap` (subscribes to `keybinding:changed`; the dashboard renders the live `dictation` binding through `BindingKeycaps`).
Hooks: `useRealtimeConversation` (OpenAI Realtime WebRTC), `useLicenseState`, `useProFeature`, `useKeyBindingMap` (subscribes to `keybinding:changed`; the dashboard renders the live `dictation` binding through `BindingKeycaps`), `useInputInsights` (telemetry + suggestion state + weekly summary + phrases; the Dashboard shows a weekly input card when collection is on).
DB schema (`src/main/db/schema.ts`, drizzle SQLite): `history`, `dictionary`, `stats`, `memo_tags`, `daily_usage`, `rag_documents`, `rag_chunks`, `meeting_sessions`, `meeting_memos`, `meeting_documents`.
DB schema (`src/main/db/schema.ts`, drizzle SQLite): `history`, `dictionary`, `stats`, `memo_tags`, `daily_usage`, `rag_documents`, `rag_chunks`, `meeting_sessions`, `meeting_memos`, `meeting_documents`, `input_activity` (hour × app counters), `typing_samples`, `personal_phrases`, `suggestions`.
---
## 6. Desktop status summary
- Core dictation/LLM/history pipeline: **implemented + tested**. The vitest case count in `apps/desktop` is **1360** after the 2026-09-21 LLM fix added 46 cases; playwright e2e is separate. Read the pass numbers together with the `better-sqlite3` ABI the tree is built for (`11` GAP-INFRA-06) — they are not comparable across configurations:
- Core dictation/LLM/history pipeline: **implemented + tested**. The vitest case count in `apps/desktop` is **1465** after input intelligence and the Local Flow Intelligence extension; playwright e2e is separate. Read the pass numbers together with the `better-sqlite3` ABI the tree is built for (`11` GAP-INFRA-06) — they are not comparable across configurations:
- **Host Node ABI** (2026-09-21, before the LLM fix): 1311 / 1314 passing. The three failures are environment-dependent rather than regressions — two need a local sidecar venv or embedding server, one pins an error message that has since changed (`11` GAP-QA-02). **This configuration has not been re-measured since the LLM fix.**
- **Electron ABI** (2026-09-21, after the LLM fix): `366 failed | 994 passed (1360)`, against a clean-tree baseline of `366 failed | 948 passed (1314)` in the same configuration — identical failure count, +46 passed, **zero new failures**. 365 of those 366 are `tests/red/*.usecase.test.ts` files dying at DB creation because of the ABI mismatch, not assertions.
- **Electron ABI** (2026-09-21, after input intelligence): `366 failed | 1042 passed (1408)`, against a clean-tree baseline of `366 failed | 994 passed (1360)` in the same configuration — identical failure count, +48 passed, **zero new failures**. 365 of those 366 are `tests/red/*.usecase.test.ts` files dying at DB creation because of the ABI mismatch, not assertions.
- **Native-ABI mismatch run** (2026-09-23, `1.5.0` release verification): `366 failed | 1099 passed (1465)`. The `better-sqlite3` module is still built for Electron ABI 130 while the host Node is ABI 131, so this is the same mismatch configuration. The failure count is unchanged (365 ABI + 1 stale sidecar error-message assertion, `11` GAP-QA-02) and the passed count rose with the new tests, so there are **zero new failures**. Full strict/full suite and Electron GUI verification remain open gates, not evidence of green.
- Input intelligence (`2026-09-21`): telemetry capture, weekly insights, next-sentence ghost text and phrase learning are implemented; real typing has verified capture → UIA snapshot → policy decisions, while overlay position/appearance, accept-insert, password blocking and weekly numbers remain manual gates (`11` GAP-INPUT-01). The 24.7 s cold / 4.9 s warm figures are a **historical latency diagnosis**, not current behaviour. **2026-09-22 operating verification:** at 19:14:12 app startup warmup plus `keep_alive: 30m` held `gemma4:e4b` resident through 19:44:12 at VRAM 3,226,342,521 bytes and context 4096. Windows GPU Engine PID sampling found no active Ollama compute then, so this was forced residency rather than infinite inference. Separately, 19:11:17–19:11:58 logs show automatic suggestions repeatedly generated under the old 900 ms / 12 per min / 5-candidate / 128-token / one-character-growth policy: a permitted burst, not proof of a single stuck request. The replacement guard removes boot warmup, uses `keep_alive: 2m`, and sets 600 ms debounce, 5 s minimum interval, 6 per min (hard maximum 12), 3 candidates, 64 tokens, 12-character growth and 8 s timeout. `LocalLLMService` adds per-request cancellation, bounded generate/stream/chat calls, required `done` frames and incomplete cleanup; voice cancellation is single-flight and reaches the request; server spawn/polling are deduplicated and disposed. Independent targeted verification passed 6 test files / 69 tests with 0 failures; changed code/tests ESLint and `git diff --check` exited 0. Raw Ollama proof: a cold bounded request hit the client hard timeout at 15.044 s then left `/api/ps` empty and `/api/version` recovered in 80 ms; explicit warmup returned HTTP 200 in 16.639 s; the subsequent `num_predict=1`, `keep_alive='2m'` request returned HTTP 200 in 553 ms with `done:true`, `eval_count:1`, `response=OK`, `done_reason:length`, and an observed `/api/ps` expiry of about 119.9 s. At 19:48:59 +09:00, with no intervening generate/unload/kill/retry, a single `/api/ps` returned HTTP 200 in 45.8 ms with `{models:[]}` and `/api/version` returned HTTP 200 in 7.3 ms with `0.32.13`: raw API expiry/unload evidence only. This does not prove an app restart, GUI overlay or real automatic-typing path, so those remain `[~]` runtime gates (`11` GAP-LLM-04, GAP-INPUT-06).
- **2026-09-23 overlay lifecycle / Windows child-process audit:** focused five test files passed 80 tests; core had 131 passing tests in an earlier independent verification; desktop typecheck/lint, Python `py_compile`, and `git diff --check` exited 0. Code and automated-test audit covers `windowsHide:true` on TTS PowerShell, VoiceAction cmd/PowerShell/general exec, audio-device and active-window PowerShell, plus all three ffmpeg paths; with existing SoX/STT/Ollama/sound-effect coverage, no Windows-capable desktop-main child-process call is known to be omitted. This is not Electron GUI runtime proof. Keep INPUT-07 and GAP-INPUT runtime gates `[~]` until an external-terminal `run-desktop.bat` restart confirms dismissal under generation/selection/focus loss and no cmd/PowerShell window recurrence for TTS, voice action, audio enumeration, screen context and file transcription.
- Cross-platform packaging: Windows NSIS (signed, `forceCodeSigning`), macOS DMG/ZIP arm64 (ad-hoc signing); auto-update via canonical Forgejo feed with update policy (`release/update-policy.json`).
- Local-first AI (SoX + faster-whisper sidecar + bundled Ollama) and cloud paths both present.
- **Local STT is packaged** (`1.3.0`): `electron-builder.yml` `extraResources` copies `sidecar-dist/sidecar` → `resources/sidecar` and `resources/ffmpeg` → `resources/ffmpeg`; `scripts/ci/verify-sidecar-bundle.mjs` gates packaging. Build locally with `npm --prefix apps/desktop run sidecar:setup && npm --prefix apps/desktop run sidecar:build`. The sidecar stays in console mode so `stdout`/`stderr` reach the app log (UTF-8, line-buffered); a packaged sidecar **must** exist or startup fails loudly instead of silently falling back to a system Python.
@ -220,6 +239,12 @@ DB schema (`src/main/db/schema.ts`, drizzle SQLite): `history`, `dictionary`, `s
| IPC registry | `src/main/ipc/index.ts` |
| IPC channel SSOT | `packages/core/src/ipc-channels.ts` |
| Key-binding contract SSOT | `packages/core/src/keybinding.ts` (catalog, actions, validation, conflicts, formatting, parsing) |
| Input intelligence domain SSOT | `packages/core/src/input-intelligence.ts` (key classification, typed delta, suggestion policy, overlay anchoring, flow/friction aggregation, phrase decay/ranking, provenance and exclusion recommendation) |
| Input intelligence services | `src/main/services/InputTelemetryService.ts`, `SuggestionService.ts`, `UiaContextService.ts`, `global-input-hook.ts` |
| Foreground window FFI | `src/main/utils/win32-foreground.ts` (koffi → user32/kernel32) |
| UIA bridge (sidecar) | `sidecar/uia_bridge.py` + `GET /uia/focus` in `sidecar/main.py` |
| Input UI | `src/renderer/components/input-insights/{InputConsentPanel,InputInsightsView}.tsx`, `src/renderer/hooks/useInputInsights.ts`, `src/renderer/popups/suggestion-overlay/` |
| Local Flow Intelligence tests | `tests/main/services/input-flow-domain.test.ts` (9), `input-flow-services.test.ts` (12); targeted suite 21/21 at the documented handoff point |
| Key-binding service / IPC / UI | `src/main/services/KeyBindingService.ts`, `src/main/ipc/keybinding-handlers.ts`, `src/renderer/components/keybinding/` |
| Preload API | `src/preload/index.ts` |
| Windows | `src/main/windows/WindowManager.ts` |

32
docs/map/10-feature-catalog.md
View file

@ -37,7 +37,7 @@ Status quick-reference: `[x]` done+verified · `[~]` partial/unverified · `[ ]`
| ID | Feature | D | W | M | B | Anchors / notes |
|---|---|---|---|---|---|---|
| AI-01 | Local LLM (Ollama) | [x] | [-] | [ ] | [-] | Desktop bundled Ollama |
| AI-01 | Local LLM (Ollama) | [~] | [-] | [ ] | [-] | Desktop bundled Ollama. 2026-09-22 guard contract: no boot warmup; suggestion `keep_alive: 2m`; request-owned cancellation; generate/stream 2048 tokens / 120 s and chat 512 / 60 s; `done` frame required with incomplete-stream cleanup; deduplicated Ollama spawn/poll plus lifecycle disposal. GAP-LLM-03 has targeted evidence (6 test files / 69 passed / 0 failed; changed code/tests ESLint and diff check exit 0). Raw Ollama confirms cold timeout cleanup, a 553 ms `num_predict=1`/`keep_alive='2m'` response, and after the about-119.9 s expiry a 19:48:59 +09:00 `/api/ps` HTTP 200 / 45.8 ms `{models:[]}` observation with no intervening generate/unload/kill/retry; `/api/version` was HTTP 200 / 7.3 ms / `0.32.13`. This is raw API expiry/unload evidence, not app-restart or GUI/runtime proof (GAP-LLM-04). |
| AI-02 | Cloud LLM (Claude/OpenAI) | [x] | [x] | [x] | [x] | Desktop `PremiumLLMService`; web/mobile via `llm-proxy`; .NET `LlmProxyService` |
| AI-03 | Auto Polish (cleanup/filler removal) | [x] | [~] | [~] | [x] | Desktop built-in; web/mobile via commands. Desktop Auto Polish is the plain `refine` action (`llm-prompts.ts:14`), not a custom instruction, so it was **not** affected by the 2026-09-21 instruction-prompt fix (AI-05); regression cases now pin `refine`/`summarize`/`grammar`/`expand` (`VoiceModeService.test.ts:435`, `llm-prompts.test.ts:153`) |
| AI-04 | Translate / summarize / rephrase | [~] | [x] | [x] | [x] | Built-in instructions. **Desktop has two paths and only one of them worked.** The plain-action path (Settings → `defaultLLMAction`, `SettingsModal.tsx:653`) reads `BASE_SYSTEM_PROMPTS` directly and was always correct. The built-in *instruction* presets (`CustomInstructionService.ts:26/35/44/53/62`) ran through the custom-instruction path and inserted the instruction's own wording instead of the result — see AI-05. Fixed in `9c2b4d4` (2026-09-21), **not verified in a running app** (`11` GAP-LLM-02). Translate still always targets English: `AppConfig` has no target-language key and neither `language` (UI locale) nor `sttLanguage` (source language) can stand in (`llm-prompts.ts:37-52`, `11` GAP-LLM-01) |
@ -155,6 +155,36 @@ Status quick-reference: `[x]` done+verified · `[~]` partial/unverified · `[ ]`
---
## INPUT — Input Intelligence (typing telemetry & next-sentence suggestions)
Everything here is **desktop-only** (Windows today) and **opt-in**: `inputTelemetryEnabled` and
`inputLearnTypedText` default to `false`. D=desktop, W=web, M=mobile, B=backend.
Status is `[~]` on desktop for one shared reason: the code and its 48 unit cases exist, but the
end-to-end behaviour has **not been verified by typing in a real app** (`11` GAP-INPUT-01).
| ID | Feature | D | W | M | B | Anchors / notes |
|---|---|---|---|---|---|---|
| INPUT-01 | Keyboard/mouse telemetry capture (opt-in) | [~] | [-] | [-] | [-] | `InputTelemetryService` — keystroke/click/scroll counters, mouse travel as the Manhattan sum of per-axis pixel deltas, active time, per-hour×app buckets flushed every 5 s. Key **contents** are never stored (ActivityWatch `aw-watcher-input` data-minimisation policy, adopted deliberately — see `11` §7). Hook ownership is ref-counted so `KeyBindingService` keeps working (`global-input-hook.ts`). |
| INPUT-02 | Foreground-app attribution | [~] | [-] | [-] | [-] | `utils/win32-foreground.ts` via `koffi` FFI (title/pid/exe/bounds), sampled at most 1×/s. `get-windows` was rejected: it needs an install script this repo does not run. |
| INPUT-03 | Weekly input insights | [~] | [-] | [-] | [-] | `INPUT_TELEMETRY.getSummary` aggregates `input_activity` into totals, daily series, top hours and top apps; rendered in Settings → Input and as a dashboard card. Daily average mouse travel is converted px → m using the display scale factor. |
| INPUT-04 | Typed-text learning (UIA, password-excluded) | [~] | [-] | [-] | [-] | Text is read from the focused field via `GET /uia/focus` (sidecar UIA bridge) and diffed longest-common-prefix/suffix, so **IME-committed Hangul/kana is counted correctly** — keycodes cannot reconstruct CJK text. UIA sends `hasSelection` only, derived by TextPattern range Start/End comparison without calling `GetText` on the selection range or adding a selected-text payload; the existing focused-field text can still include a selection. A non-collapsed selection immediately clears suggestions as `selection-active`. `IsPassword` is checked before any read (fail-closed); IME composition suppresses both stats and suggestions. |
| INPUT-05 | Personal phrase corpus (typed + voice) | [~] | [-] | [-] | [-] | Sentence-level phrases from typed text and from voice history (`HistoryService.create` feeds `recordExternalText`), ranked by frequency/recency as prompt hints; users can delete individual phrases. |
| INPUT-06 | Next-sentence suggestion (ghost text) | [~] | [-] | [-] | [-] | `SuggestionService` + `buildSuggestionPrompt` (`llm-prompts.ts` SSOT, instruction stays in the system prompt). The 2026-09-22 guard contract is 600 ms debounce, ≥5 s between requests, 6 requests/min by default (hard-config maximum 12), 3 candidates, 64 output tokens, 12-character growth before regeneration, 8 s request timeout and `keep_alive: 2m`; boot warmup is removed. Each request has its own cancellation signal. Presentation-active includes candidates, `generating`, `warmingUp` and `partialText`; clear/dismiss aborts, invalidates the generation token, clears TTL state and emits `cleared`/hide, and a final success resets `generating=false`/`partialText=null`. Focused evidence for the lifecycle and Windows-child-process change: five test files / 80 tests passed; desktop typecheck/lint, Python `py_compile`, and `git diff --check` exited 0 (core 131-test pass was independently verified earlier). Status remains `[~]`: this is not app-restart, GUI overlay, or real automatic-typing evidence. |
| INPUT-07 | Caret-anchored suggestion overlay | [~] | [-] | [-] | [-] | `suggestion-overlay` popup placed by `anchorFloatingPanel` (caret → element → cursor fallback, flip above when the caret is near the bottom, clamped to the work area). Non-focusable; click-through unless `suggestionOverlayInteractive`. While actually visible it continues periodic UIA validation after 5 s and revalidates 120 ms after mouse-up; unavailable/non-editable focus or a non-collapsed selection aborts and hides it. X first hides the renderer panel, then main IPC directly hides `BrowserWindow` and dismisses the service, so late tokened results cannot revive it. Accept/next/prev/dismiss are four global key bindings (`suggestion-accept`/`next`/`prev`/`dismiss`, default `Ctrl+Alt+→/↓/↑/←`), and the overlay has a mouse close button. Up to three candidates are shown in a scrollable list with a warm-up/generating spinner. |
| INPUT-08 | Per-app exclusions & consent controls | [~] | [-] | [-] | [-] | `inputExcludedApps` (executable names, case-insensitive) blocks both collection context and suggestions; telemetry master switch, pause, text-learning toggle and "delete collected data" all live in Settings → Input. 30-day retention prune runs on start. |
| INPUT-09 | Flow Radar | [~] | [-] | [-] | [-] | `rankFlowWindows` ranks hourly aggregate activity density, character volume and edit stability into potential-focus time windows. It is not a real-session detector or session record. |
| INPUT-10 | Edit Friction | [~] | [-] | [-] | [-] | `calculateFrictionInsight` derives friction from char/backspace quantities and reports edits per 100 chars; it does not infer sentiment or productivity. |
| INPUT-11 | App DNA | [~] | [-] | [-] | [-] | Personal phrase ranking passes `appName` context and applies a 1.75× same-app ranking bonus (`selectPhraseHints`); learned text remains opt-in. |
| INPUT-12 | App Quality | [~] | [-] | [-] | [-] | `InputTelemetryService.getSummary` aggregates per-app suggestion total, accepted count, accept rate and average latency from local suggestion history. |
| INPUT-13 | Privacy Receipt | [~] | [-] | [-] | [-] | Local-only receipt shows actual row counts. Raw individual key events/key codes/content stream are not retained; opt-in learned text can reside in `typing_samples`/`personal_phrases`. `input_activity`, `typing_samples` and `suggestions` retain 30 days; `personal_phrases` has no age-based automatic expiry and is removed by individual deletion, delete-all, or consent withdrawal. Receipt read/delete storage failures are IPC errors. |
| INPUT-14 | Smart Exclusion | [~] | [-] | [-] | [-] | Password fields add no evidence. Only the current app may receive a one-click recommendation after ≥4 observations, readable=0 and problematic ratio ≥75%; it never auto-excludes. |
| INPUT-15 | Why This Suggestion | [~] | [-] | [-] | [-] | Overlay exposes only local-model/local-memory provenance and continuation/related/phrase/appPhrase counts; raw evidence text is not shown. |
| INPUT-16 | Memory Decay | [~] | [-] | [-] | [-] | Phrase ranking combines a 30-day half-life, frequency and the App DNA 1.75× same-app bonus. |
| INPUT-17 | Instant Recall | [~] | [-] | [-] | [-] | Local-memory fallback (`provenance=local-memory`) occurs only for unavailable model, non-cancellation error, actual timeout or empty model result; it spends no additional model budget and is blocked on dismiss, new typing, token/context mismatch or staleness. |
| INPUT-18 | Shortcut Safety Audit | [~] | [-] | [-] | [-] | Settings UI applies core `auditKeyBindingMap` to surface invalid/conflict issues while hold/double-press exceptions remain unchanged. |
---
## SHELL — Platform Shell, Settings, Onboarding, Support
| ID | Feature | D | W | M | B | Anchors / notes |

11
docs/map/11-gap-backlog.md
View file

@ -72,6 +72,16 @@ Legend: `[ ]` open · `[~]` in progress · `[!]` blocked externally · `[x]` res
| GAP-INFRA-05 | Build | 패키징된 렌더러 팝업 스크립트가 번들에 없었다. 팝업 HTML이 classic `<script src="./script.js">`를 참조해 Vite가 처리하지 않았고, dev에서는 로드되지만 설치본에는 파일이 없었다. 그래서 녹음 오버레이가 0:00에서 멈추고 웨이브 바가 뜨지 않았으며 실시간 자막이 렌더되지 않았다. 로드 전 `webContents.send`가 조용히 버려지는 문제와 `hide()` 이후 재표시의 z-order/repaint 유실도 함께 있었다. | `apps/desktop/src/renderer/popups/*/index.html`, `apps/desktop/src/main/windows/WindowManager.ts`, `scripts/ci/verify-desktop-renderer-bundles.mjs` | `[x]` 2026-09-19: 팝업 5종을 `type="module"`로 전환해 Vite가 해시된 번들로 방출하도록 고쳤고, 빌드 HTML이 참조하는 모든 로컬 asset이 디스크에 있는지 검사하는 `verify-desktop-renderer-bundles.mjs`(+ self-test)를 `.forgejo`/`.github` 패키징 파이프라인에 연결했다. WindowManager는 렌더러 준비 전 IPC를 `did-finish-load`까지 보관하고, 팝업을 표시할 때마다 topmost 재선언 + 강제 repaint를 수행하며, 팝업 렌더러 콘솔/로드 실패를 main 로그로 승격한다. |
| GAP-INPUT-01 | Input intelligence | **2026-09-22 10:57 실사용 검증 성공** — 카톡 타이핑 중 제안 5개 생성(1436ms)이 로그로 확인됐고(조합 중·idle 25ms에서도), 50초 관찰에 스냅샷 67건/생성 3회/실패 0건이었다. 남은 확인: 오버레이 위치·외관, 수락 삽입, 비밀번호 차단, 주간 수치 48개 유닛 케이스는 정책/집계/프롬프트 같은 순수 함수만 덮는다. 검증되지 않은 것: (1) `Alt+Shift+←` 수락이 실제 앱에 문장을 삽입하는지, (2) 오버레이가 케어렛 옆에 붙는지(케어렛 rect Chromium 에서만 UIA 로 얻어지고 네이티브 Win32 앱은 `GetGUIThreadInfo` 경로가 아예 없다), (3) `TypeError` 없이 Electron 메인에서 `koffi` 가 로드되는지(현재 호스트 Node 에서만 확인), (4) PyInstaller 번들에 `uiautomation`/`comtypes` 가 실제로 들어가는지(`build-sidecar.mjs` 에 `--collect-all` 을 추가했지만 번들을 다시 빌드해 확인하지 않았다). 에이전트는 데스크톱 GUI 를 띄울 수 없다(`AGENTS.md` §3). | `apps/desktop/src/main/services/{InputTelemetryService,SuggestionService,UiaContextService}.ts`, `apps/desktop/src/main/utils/win32-foreground.ts`, `apps/desktop/sidecar/{uia_bridge.py,requirements.txt}`, `apps/desktop/scripts/build-sidecar.mjs`, `apps/desktop/tests/main/services/input-intelligence.test.ts`(45) + `llm-prompts.test.ts`(+4) | **2026-09-21 23:13 부분 검증 완료** (사용자가 실제로 Notepad 에 타이핑한 로그): 텔레메트리 기동, uiohook 후킹, UIA 스냅샷(Notepad — `edit=true pw=false comp=false src=value`), WindowsTerminal 의 문서형 컨트롤을 `not-editable` 로 정확히 거부, 판단 로그까지 전부 실동작 확인. 이 과정에서 결함 4건을 잡아 고쳤다: (a) 트리거 지연(1000ms)이 스냅샷 디바운스(700ms)보다 커서 **"멈춘 뒤" 게이트가 결코 열리지 않던 문제**(settle 패스 추가), (b) 켜 둔 상태로 앱을 켜면 워밍업이 한 번도 안 돌던 문제(부팅 시 워밍업 + 가용성 폴링 대기 재시도), (c) 케어렛 오프셋을 못 주는 앱에서 문서 전체를 접두로 쓰던 문제(tail 폴백), (d) 후보가 도착해야 오버레이가 떠서 "아무것도 안 나옴" 으로 보이던 문제(요청 즉시 "생성 중" 표시). **2026-09-22 09:00~09:17 추가 실측** (사용자가 KakaoTalk·WindowsTerminal·Agent Switchboard 에 타이핑한 17개 스냅샷): 편집 가능으로 판정된 것은 3건뿐이고, 그중 KakaoTalk 입력창은 `edit=true src=value` 인데도 `len=0` 이라 실제 내용을 못 읽었다 — 커스텀 렌더 앱(카톡·터미널·에이전트 UI)에서는 UIA 가 텍스트를 노출하지 않는다. 즉 "아무것도 안 나옴" 의 상당 부분은 결함이 아니라 **읽을 수 없는 앱에서의 정상 동작**이며, 이 구분이 사용자에게 보이지 않던 것이 문제였다. 그래서 설정 > 입력에 **실시간 진단 줄**(포커스 앱 · 읽기 가부 · 소스 · 글자 수 · 비밀번호/케어렛 폴백 표시)을 추가하고 12개 로케일에 문구를 넣었다. 남은 확인: 오버레이의 실제 위치·외관, `Alt+Shift+←` 수락 삽입, 비밀번호 필드 차단, 주간 수치 정확도. 함께: `sidecar:build` 후 `GET /uia/focus` 응답을 확인하고 그 결과를 `scripts/ci/verify-sidecar-bundle.mjs` 의 필수 항목에 반영한다. **검증 도구 주의**: 데스크톱 `npm run typecheck` 는 문서화된 대로 no-op 이라(GAP-INFRA-04) 이번 작업에서도 거짓 통과를 냈다 — `typecheck:strict` 로 다시 돌려 이 작업이 만든 타입 오류 7건(core 에 없는 `InputTelemetryState` 참조 5건 등)을 찾아 고쳤다. 현재 strict 기준 이 작업 파일들의 오류는 0건이다(main 13 / renderer 35 는 전부 선재). |
| GAP-INPUT-07 | Input intelligence | `[~]` **2026-09-23 focused automated evidence:** five test files / 80 tests passed; desktop typecheck/lint, Python `py_compile`, and `git diff --check` exited 0 (the core 131-test pass is earlier independent evidence). The overlay lifecycle and `windowsHide:true` child-process audit are code/automation evidence only, not Electron GUI or external-app insertion proof. | `InputTelemetryService.ts`, `SuggestionService.ts`, `KeyBindingService.ts`, `components/input-insights/{InputConsentPanel,InputInsightsView}.tsx`, `popups/suggestion-overlay/`, `tests/main/services/{input-flow-domain,input-flow-services,windows-child-process-hide}.test.ts` | From an external terminal restart with `run-desktop.bat`, verify: (1) click X during generation closes immediately and never reappears; (2) selecting text closes the overlay; (3) input-focus loss closes it; (4) TTS, voice action, audio enumeration, screen context and file transcription do not revive a cmd/PowerShell window. Retain the existing editable/read-unavailable/password, Flow Radar/Edit Friction/App Quality, receipt, Smart Exclusion, provenance/fallback, shortcut-audit and narrow-geometry checks. Automated tests do **not** replace this manual proof. |
| GAP-INPUT-02 | Input intelligence | **IME 조합 중 텍스트(preedit)는 수집되지 않는다.** 우리는 조합 중에는 통계·제안을 모두 억제하고 조합이 끝난 커밋 텍스트만 UIA 로 읽는다. 조합 문자열 자체를 읽으려면 `IUIAutomationTextEditPattern::GetActiveComposition`(또는 레거시 앱은 IME `ImmGetCompositionStringW`)이 필요하다. KeyType.Windows 도 이 부분을 명시적으로 다음 사이클로 미룬다 — 즉 업계 공통 미해결 지점이다. 조합 중 억제 자체는 MS Learn 의 IME 문서상 필수 조치이며 구현돼 있다. | `apps/desktop/sidecar/uia_bridge.py`(`_caret_rect_and_offset` 의 TextEditPattern 블록), `packages/core/src/input-intelligence.ts`(`decideSuggestion` 의 `composing` 분기) | 조합 범위를 읽어 "조합 중 미리보기"를 제안 후보로 쓸지 검토한다. 지금은 억제만 하고 있어 한국어 사용자는 조합을 끝내야 제안이 뜬다. |
| GAP-INPUT-03 | Input intelligence | **공백이 없는 언어(중국어)는 단어 수가 과소 집계된다.** `countWords` 가 공백 분리 토큰을 세므로 "今天开会讨论了三件事" 는 1단어로 계산된다. 한국어/영어/일본어(공백 사용)는 정상이다. | `packages/core/src/input-intelligence.ts`(`countWords`), 카탈로그 INPUT-03, 테스트 `input-intelligence.test.ts` | CJK 연속 구간을 문자 단위로 세는 분기를 추가하거나, 통계 라벨을 "단어" 대신 "어절"로 바꾼다. |
| GAP-INPUT-04 | Input intelligence | **UIA 브리지가 Windows 전용이다.** `uiautomation` 은 Windows UI Automation 래퍼이므로 macOS 는 AX API, Linux 는 AT-SPI 구현이 따로 필요하다. 그래서 입력 인텔리전스는 카탈로그에서 데스크톱(`D`)만 표기하고 나머지 표면은 `[-]` 다. 또한 Chromium 138 미만 앱은 접근성 트리가 켜져 있으면 입력창 텍스트를 노출하지 않는데, 그 앱들을 `--force-renderer-accessibility` 로 켜도록 강제할 방법이 없다(우리 자신은 `app.setAccessibilitySupportEnabled(true)` 로 처리했다 — `bootstrap.ts`). | `apps/desktop/sidecar/uia_bridge.py`, `apps/desktop/src/main/bootstrap.ts`(`initInputIntelligence`) | macOS AX 경로를 붙일지 결정한다. 붙이지 않으면 카탈로그에서 macOS 를 명시적 N/A 로 유지한다. |
| GAP-INPUT-05 | Input intelligence | **수락/닫기 키가 사용자 습관과 충돌할 수 있다.** 기본값을 `Alt+Shift+←/↓/↑` 로 둔 이유는 Tab·Escape·Ctrl+Space 같은 관례 키를 뺏지 않기 위해서다(우리는 키를 삼키지 않으므로 원래 동작이 함께 실행된다). 그 대가는 "Tab 으로 수락" 같은 자연스러운 조작이 아니라는 점이고, 인라인 컴플리션 도구 대부분(Tab)과 다르다. | `packages/core/src/keybinding.ts`(`suggestion-accept`/`next`/`dismiss`), `apps/desktop/src/main/bootstrap.ts`(트리거 구독) | 오버레이가 보일 때만 Tab 을 삼키는 경로(전역 후킹에서 조건부 suppress)를 검토한다. 지금은 불가능하지 않지만 포커스 없는 창에서 키를 가로채는 설계가 필요하다. |
| GAP-LLM-03 | LLM | `[x]` **2026-09-22 코드 구현 완료.** 공유 `_abortController`를 요청별 controller로 대체하고 외부 signal을 각 요청에 연결/정리한다. generate/stream은 2048 tokens / 120 s, chat은 512 / 60 s로 제한하며, stream은 `done` frame 없이는 성공 처리하지 않고 불완전 응답을 정리한다. VoiceConversation은 single-flight와 request signal을 통해 local cancel을 실제 chat 취소로 전달한다. | `apps/desktop/src/main/services/LocalLLMService.ts`, `VoiceConversationService.ts` | 독립 표적 검증: 6 test files / 69 tests passed / 0 failed. 변경 코드·테스트 ESLint exit 0, `git diff --check` exit 0. 이는 전체 strict typecheck 또는 GUI 검증이 아니다. 앱 재시작 뒤 동시 제안·음성 요청의 요청별 취소와 `done` 누락 실패 처리는 GAP-LLM-04에서 계속 확인한다. |
| GAP-LLM-04 | LLM | `[~]` **Gemma/Ollama residency·burst·unbounded chat root cause (2026-09-22).** 19:14:12 boot warmup의 `keep_alive: 30m`가 `gemma4:e4b`를 expiry 19:44:12까지 VRAM 3,226,342,521 bytes / context 4096으로 상주시켰다. Windows GPU Engine PID 표본에는 활성 Ollama compute가 없어 무한 추론이 아니라 강제 residency였다. 19:11:17–19:11:58 자동 제안 반복은 기존 900 ms / 12 per min / 5 candidates / 128 tokens / 1-char growth가 허용한 burst였다. 위험 경로는 chat의 무제한 `num_predict`(Ollama 기본 `-1`), timeout·외부 abort 부재, 공유 취소, `done` 없는 EOF 성공, voice cancel 미전파, watchdog의 비취소였다. | `apps/desktop/src/main/services/{LocalLLMService,SuggestionService,VoiceConversationService}.ts`, Ollama `/api/ps` and Windows GPU Engine observations | 구현 계약: boot warmup 제거, suggestion `keep_alive: 2m`; 600 ms debounce / 5 s interval / 6 per min (hard max 12) / 3 candidates / 64 tokens / 12-char growth / 8 s timeout; 요청별 cancellation, bounded requests, done/cleanup, voice single-flight, spawn/poll dedupe와 dispose. Raw Ollama 증거: cold bounded 요청은 client hard timeout 15.044 s에 취소된 뒤 `/api/ps` empty와 `/api/version` 80 ms 회복을 보였고, explicit warmup HTTP 200은 16.639 s, 후속 `num_predict=1` / `keep_alive='2m'` 요청은 553 ms HTTP 200 / `done:true` / `eval_count:1` / `response=OK` / `done_reason:length`였으며 `/api/ps` expiry는 약 119.9 s였다. 이어 19:48:59 +09:00에는 새 generate/unload/kill/retry 없이 충분히 지난 뒤 단일 `/api/ps`가 HTTP 200 / 45.8 ms / `{models:[]}`였고 `/api/version`은 HTTP 200 / 7.3 ms / `0.32.13`이었다. 이는 raw API 수준의 expiry 뒤 unload 확인이다. **남은 조건:** 앱 재시작·GUI·실제 자동제안 타이핑 증거는 아니므로 그 경로에서 rate limit, timeout/cancel, 2분 residency를 확인한다. |
| GAP-INPUT-06 | Input intelligence | `[~]` **제안 폭주 상한 및 runtime 확인.** 이전 24.7 s / 4.9 s, boot warmup, `keep_alive: 30m`, 48/96/128 tokens 및 2/5 candidates 기록은 역사적 가설/완화 이력이며 현재 정책이 아니다. 현 정책은 600 ms debounce, 5 s minimum interval, 6 requests/min default (hard max 12), 3 candidates, 64 output tokens, 12-character growth, 8 s timeout, `keep_alive: 2m` 및 boot warmup 제거다. | `apps/desktop/src/main/services/SuggestionService.ts`, `packages/core/src/input-intelligence.ts`, `LocalLLMService.ts` | 표적 자동 검증은 6 files / 69 passed / 0 failed이며 raw Ollama의 15.044 s cold timeout cleanup, 약 119.9 s residency expiry, 그리고 19:48:59 +09:00에 새 generate/unload/kill/retry 없이 확인한 `/api/ps` HTTP 200 / 45.8 ms / `{models:[]}` (`/api/version` HTTP 200 / 7.3 ms / `0.32.13`)도 확인됐다. 전체 strict typecheck는 green이 아니며, 이 raw API unload 증거는 앱 재시작·GUI·실제 자동제안 타이핑 검증이 아니다. 그 경로에서 rate limit, 8 s abort, 3-candidate/64-token 상한을 확인한다. |
---
## 2. Mobile checklist roll-up (from `MOBILE_APP_COMPLETION_SSOT.md` §4)
@ -188,4 +198,5 @@ Actionable checklist for the work started this session. Fields to fill are blank
| ID | 제약 | 왜 이대로 두는가 | 완화 장치 |
|---|---|---|---|
| CONSTRAINT-INPUT-01 | 입력 레메트리는 **키 내용을 저장하지 않는다.** 카운터(키 입력·클릭·스크롤)와 마우스 이동 거리, 그리고 **사용자가 명시적으로 동의한 경우에만** 읽은 텍스트(학습 DB)만 남긴다. 키코드→문자 복원은 시도하지 않는다. | ActivityWatch `aw-watcher-input` 이 같은 이유로 같은 선택을 한다 — 전역 키 내용 수집은 보안·프라이버시 파급이 크고, 신뢰를 잃으면 기능 자체가 사라진다(README 원문: "This does not track which keys you press … This is not a keylogger, and never will be"). 게다가 키코드 복원은 한/일 IME 에서 원리적으로 불가능하다(조합 결과가 텍스트다). | 수집은 옵트인(`inputTelemetryEnabled`/`inputLearnTypedText` 기본 false), 비밀번호 필드는 `IsPassword` 로 fail-closed 차단, 앱별 제외, 30일 보존 후 자동 삭제, "수집된 데이터 삭제" 버튼(동의 철회 시 즉시 삭제 + 학습 플래그 해제), 전송 없음(로컬 SQLite 전용). |
| CONSTRAINT-I18N-01 | `packages/core/src/keybinding.ts` 는 i18n 키를 평범한 `string` 으로 노출한다. 렌더러가 `asTranslationKey()`(`apps/desktop/src/renderer/components/keybinding/translation-key.ts:7`)로 경계에서 캐스팅하므로, 존재하지 않는 키를 넘겨도 컴파일러가 잡지 못한다. | core 가 로케일 패키지에 의존하지 않게 하려는 의도적 설계다. 검토한 대안 둘 다 성립하지 않는다 — (A) 키 필드를 리터럴 유니온으로 좁히는 방식은 `KEY_CATALOG` 가 `letterEntries()` 같은 함수 생성부를 포함해 불가능하고, (B) core 가 `@d3ro/i18n` 의 타입 가드를 쓰는 방식은 의존 방향을 core → i18n 으로 역전시켜 `03-shared-packages.md` §6 의 전제를 깬다. 2026-09-21 결정: 현행 유지. | `packages/core/__tests__/keybinding-i18n.test.ts` (14 케이스). core 가 참조하는 키가 12개 로케일 전부에 있는지, 값이 빈 문자열이 아닌지, core 가 렌더러 전용 `keybinding.ui.*` 를 참조하지 않는지 검사한다. 거부 사유 키는 하드코딩 목록이 아니라 실제 `validateBinding` 경로를 태워 수집하므로 새 사유가 생기면 자동으로 커버된다. |