feat(desktop): typo auto-correct and context-aware sentence prediction
Some checks failed
ci / 정본·보안·린트·타입·테스트 (push) Has been cancelled
deploy-site / deploy (push) Has been cancelled
ci / 워크스페이스 빌드 검증 (push) Has been cancelled
ci / 모바일 린트·타입·Jest (push) Has been cancelled
ci / Supabase Edge Functions + Cloudflare Worker (push) Has been cancelled
ci / .NET API 서버 테스트 (push) Has been cancelled

Typo auto-correct (INPUT-19)
- TypoCorrectionService + core typo-correction: 2-set Korean layout
  mistypes both ways, always-wrong spellings, near-misses of the user's
  frequent words, optional model check re-validated in core; applied by
  backspace N + paste and cancelled on new keys or window change.

Prediction quality (INPUT-20/21)
- gemma models are sent raw with the user's unfinished sentence pre-filled
  into the model turn, so the model truly continues the text (mid-word
  joins and spacing exact). Other models keep the instruction prompt.
- Context: text before the caret (opening lines and a fact list of names,
  dates, amounts outside the 400-char window), messages just sent in the
  same chat window (memory only, 30 min), personal-graph follow-ups and
  IDF-ranked related sentences with Korean stemming.
- Guards: first sentence only, restated-prefix stripping (also fixes the
  instruction path), repeat drop, no invented dates/names/numbers
  (one retry), plain -다 register, no emoji unless the user uses them,
  search-box keywords. Fill loop mixes instruction prompts for variety.
- Opt-in real-model probe: tests/integration/suggestion-quality.ollama.test.ts

Suggestions now actually show up
- Terminal (Claude Code): caret trust from the input line, not the whole
  screen (status line below the box blocked every keystroke); Korean
  composition counts as typing at the end; a grey placeholder filling the
  empty box at once is not user text.
- Chrome: UIA focus flicker while typing no longer resets "edited".
- KakaoTalk/RichEdit: trailing paragraph mark ignored for edit position
  and stripped so accepted text keeps its separating space.
- While the overlay is visible the suggestion shortcuts are captured so
  Ctrl+Alt+Enter no longer also types a newline; released when hidden.
- nut-js preloaded after boot (first accept waited 4.8 s).
This commit is contained in:
Yun Chan 2026-10-03 21:48:28 +09:00
parent 5ca3709b32
commit 3cac14f668
56 changed files with 4700 additions and 258 deletions

View file

@ -183,6 +183,9 @@ end-to-end behaviour has **not been verified by typing in a real app** (`11` GAP
| INPUT-16 | Memory Decay | [~] | [-] | [-] | [-] | Phrase ranking combines a 30-day half-life, frequency and the App DNA 1.75× same-app bonus. |
| INPUT-17 | Instant Recall | [~] | [-] | [-] | [-] | Local-memory fallback (`provenance=local-memory`) occurs only for unavailable model, non-cancellation error, actual timeout or empty model result; it spends no additional model budget and is blocked on dismiss, new typing, token/context mismatch or staleness. |
| INPUT-18 | Shortcut Safety Audit | [~] | [-] | [-] | [-] | Settings UI applies core `auditKeyBindingMap` to surface invalid/conflict issues while hold/double-press exceptions remain unchanged. |
| INPUT-19 | Typo auto-correct while typing | [~] | [-] | [-] | [-] | **2026-10-03.** `TypoCorrectionService` (main, singleton + EventEmitter) runs on each `typed-context` before `SuggestionService`. Only after a pause (`TYPO_MIN_IDLE_MS` 600 ms), only in the 120-char/last-line region before the caret, only in fields the user just edited (`editedSinceFocus` + `typedRecently`); never in password/selection/IME-composing/caret-unknown fields, terminals, code editors (`shouldLearnFromApp`) or user-excluded apps. Layer 1 (core `typo-correction.ts`, pure): 2-set Korean layout mistypes both ways (`dkssudgktpdy`→`안녕하세요`, `솓`→`the`; conversion must compose fully into KS X 1001 common syllables — `hangul-common-syllables.ts`), always-wrong spellings (됬→됐, ㄹ께→ㄹ게, 몇일→며칠, teh→the, dont→don't …), and 1-edit near-misses of the user's frequent English words. Layer 2: if the suggestion model is already warm, a 2.5 s / 48-token sentence check (`buildTypoCheckPrompt`, `틀린말 => 고친말`) whose output is re-validated by `validateModelCorrections` (whole-eojeol or stem match, jamo-level edit distance, no spacing/case/word swaps, no abbreviation expansion like 넘→너무, user's repeated words exempt). Apply = backspace N + paste the corrected span (`TextInsertService.replaceBeforeCaret`, serialized with other inserts); cancelled if any key arrived after the decision or the foreground window changed. Programmatic keys are excluded from input stats. Reverting a fix within 30 s puts the word in `typoIgnoredWords`. While the model check runs the suggestion is deferred (`settled`) so it is generated from the corrected text and the two requests never race on the same model. Toggle: Settings → Input (`typoAutoCorrectEnabled`, default on; `typoAutoCorrectUseModel`). Logs record counts/sources only, never the words. Evidence: core 478 tests, desktop `TypoCorrectionService.test.ts` 14; real `gemma4:e4b` probe ≈0.5–0.8 s per check, 0 false fixes on slang/proper-noun samples. Not yet verified by typing in a real external app (GAP-INPUT-08). |
| INPUT-20 | Context-aware suggestion prompt & graph retrieval | [~] | [-] | [-] | [-] | **2026-10-03.** Prompt (`buildSuggestionPrompt`) now splits the prefix with core `writing-context.ts` into earlier text / last sentence / unfinished current sentence and states the mode (finish this sentence vs. write the next one), language and Korean register (formal/polite/casual); adds the text after the caret (no overlap) and fixes dropped characters that had leaked into the old prompt ("[지금까지 텍스트]", "사용자가 언어"). Graph (`PersonalGraphService.retrieveContext`): `followUps` = what the user actually wrote after the same (or near-identical) last sentence via `follows` edges; continuations fall back from 14- to 10- to 7-char word-boundary tails; related sentences ranked by IDF-weighted term overlap (`scoreRelatedNodes`) with same-app/recency boosts plus a one-hop `follows` expansion. Korean terms are stemmed (`stemTerm`: particles and common 하다 endings) so "회의에서"/"회의는" match; stored old terms are re-stemmed at query time. Phrase hints for the prompt keep only phrases sharing terms with the current text (or same app); unfiltered phrases remain for the no-model local-memory fallback. Evidence: `personal-graph-context.test.ts` (real SQLite) 5, `llm-prompts.test.ts` +5, core `writing-context.test.ts` 8; real `gemma4:e4b` probe keeps register and follows graph hints (~0.55 s). |
| INPUT-21 | Prefill continuation + document/conversation context for suggestions | [~] | [-] | [-] | [-] | **2026-10-03.** For models whose chat template we know (gemma 4 `<\|turn>`, gemma 2/3 `<start_of_turn>`; core `chat-template.ts`), the suggestion request is sent `raw` with the user's own unfinished sentence (or last sentence) pre-filled into the model turn, so the model literally continues the text: mid-word joins (`회의실`+`에서`) and spacing come out exact (`extractContinuation` keeps joint `attach`/`separate`; trailing space is not pre-filled — it destabilised output — and an attach after a finished word is rejected). Ollama's chat-API prefill was measured and rejected (drops the leading space, closes the turn: `검토해 보시`→`부탁드립니다.`). Other models keep the instruction prompt. Context added to both paths: whole text before the caret (`TypingContext.documentBefore`, ≤6000 chars; null for document-like/terminal/password) → opening lines outside the 1,200-char window and a fact list (names+titles, attendee lines, dates, times, amounts, versions, acronyms, quoted terms — `analyzeDocumentContext`); **same-window messages the user just sent** (chat boxes empty on send — `RecentWritesBuffer`, memory only, 30 min, cleared when input observation stops, terminals excluded); search-box detection (keywords, not sentences); plain `-다` register split from casual `-어`; no emoji unless the user uses them. Output guards: first sentence/line only, stream cut as soon as it ends, restated-prefix stripping (also fixes the instruction path, where `CI는 통과했고…` was inserted after `CI는`), repeat-of-written-sentence drop, and **unsupported-fact rejection** (`findUnsupportedFacts`: dates/numbers/names+titles/lone Latin identifiers absent from document, window title, recent messages and personal hints) with one retry for the first candidate. Fill loop: temperature 1.0; in next-sentence mode every other fill uses the instruction prompt (prefill alone repeated the first candidate). Evidence: real `gemma4:e4b` probe `tests/integration/suggestion-quality.ollama.test.ts` (15 scenarios, opt-in `D3RO_SUGGESTION_PROBE=1`) — first-candidate warnings 0/15 vs 1/15 for the old path (old path invented a launch date instead of the 11월 15일 stated 1,200 chars earlier), mean ≈0.68 s both; core `suggestion-continuation.test.ts` 25, `llm-prompts.test.ts` +7. Not yet verified by typing in a real app (GAP-INPUT-09). |
---

View file

@ -98,6 +98,8 @@ Legend: `[ ]` open · `[~]` in progress · `[!]` blocked externally · `[x]` res
| GAP-LLM-03 | LLM | `[x]` **2026-09-22 코드 구현 완료.** 공유 `_abortController`를 요청별 controller로 대체하고 외부 signal을 각 요청에 연결/정리한다. generate/stream은 2048 tokens / 120 s, chat은 512 / 60 s로 제한하며, stream은 `done` frame 없이는 성공 처리하지 않고 불완전 응답을 정리한다. VoiceConversation은 single-flight와 request signal을 통해 local cancel을 실제 chat 취소로 전달한다. | `apps/desktop/src/main/services/LocalLLMService.ts`, `VoiceConversationService.ts` | 독립 표적 검증: 6 test files / 69 tests passed / 0 failed. 변경 코드·테스트 ESLint exit 0, `git diff --check` exit 0. 이는 전체 strict typecheck 또는 GUI 검증이 아니다. 앱 재시작 뒤 동시 제안·음성 요청의 요청별 취소와 `done` 누락 실패 처리는 GAP-LLM-04에서 계속 확인한다. |
| GAP-LLM-04 | LLM | `[~]` **Gemma/Ollama residency·burst·unbounded chat root cause (2026-09-22).** 19:14:12 boot warmup의 `keep_alive: 30m`가 `gemma4:e4b`를 expiry 19:44:12까지 VRAM 3,226,342,521 bytes / context 4096으로 상주시켰다. Windows GPU Engine PID 표본에는 활성 Ollama compute가 없어 무한 추론이 아니라 강제 residency였다. 19:11:17–19:11:58 자동 제안 반복은 기존 900 ms / 12 per min / 5 candidates / 128 tokens / 1-char growth가 허용한 burst였다. 위험 경로는 chat의 무제한 `num_predict`(Ollama 기본 `-1`), timeout·외부 abort 부재, 공유 취소, `done` 없는 EOF 성공, voice cancel 미전파, watchdog의 비취소였다. | `apps/desktop/src/main/services/{LocalLLMService,SuggestionService,VoiceConversationService}.ts`, Ollama `/api/ps` and Windows GPU Engine observations | 구현 계약: boot warmup 제거, suggestion `keep_alive: 2m`; 600 ms debounce / 5 s interval / 6 per min (hard max 12) / 3 candidates / 64 tokens / 12-char growth / 8 s timeout; 요청별 cancellation, bounded requests, done/cleanup, voice single-flight, spawn/poll dedupe와 dispose. Raw Ollama 증거: cold bounded 요청은 client hard timeout 15.044 s에 취소된 뒤 `/api/ps` empty와 `/api/version` 80 ms 회복을 보였고, explicit warmup HTTP 200은 16.639 s, 후속 `num_predict=1` / `keep_alive='2m'` 요청은 553 ms HTTP 200 / `done:true` / `eval_count:1` / `response=OK` / `done_reason:length`였으며 `/api/ps` expiry는 약 119.9 s였다. 이어 19:48:59 +09:00에는 새 generate/unload/kill/retry 없이 충분히 지난 뒤 단일 `/api/ps`가 HTTP 200 / 45.8 ms / `{models:[]}`였고 `/api/version`은 HTTP 200 / 7.3 ms / `0.32.13`이었다. 이는 raw API 수준의 expiry 뒤 unload 확인이다. **남은 조건:** 앱 재시작·GUI·실제 자동제안 타이핑 증거는 아니므로 그 경로에서 rate limit, timeout/cancel, 2분 residency를 확인한다. |
| GAP-INPUT-06 | Input intelligence | `[~]` **제안 폭주 상한 및 runtime 확인.** 이전 24.7 s / 4.9 s, boot warmup, `keep_alive: 30m`, 48/96/128 tokens 및 2/5 candidates 기록은 역사적 가설/완화 이력이며 현재 정책이 아니다. 현 정책은 600 ms debounce, 5 s minimum interval, 6 requests/min default (hard max 12), 3 candidates, 64 output tokens, 12-character growth, 8 s timeout, `keep_alive: 2m` 및 boot warmup 제거다. | `apps/desktop/src/main/services/SuggestionService.ts`, `packages/core/src/input-intelligence.ts`, `LocalLLMService.ts` | 표적 자동 검증은 6 files / 69 passed / 0 failed이며 raw Ollama의 15.044 s cold timeout cleanup, 약 119.9 s residency expiry, 그리고 19:48:59 +09:00에 새 generate/unload/kill/retry 없이 확인한 `/api/ps` HTTP 200 / 45.8 ms / `{models:[]}` (`/api/version` HTTP 200 / 7.3 ms / `0.32.13`)도 확인됐다. 전체 strict typecheck는 green이 아니며, 이 raw API unload 증거는 앱 재시작·GUI·실제 자동제안 타이핑 검증이 아니다. 그 경로에서 rate limit, 8 s abort, 3-candidate/64-token 상한을 확인한다. |
| GAP-INPUT-08 | Input intelligence | `[~]` **오타 자동 교정(INPUT-19)·문맥 제안(INPUT-20) 실앱 확인 (2026-10-03).** 자동 테스트와 실제 `gemma4:e4b` 프롬프트 프로브는 통과했지만, 외부 앱에서 "백스페이스 N + 붙여넣기" 교체가 정확히 동작하는지는 GUI 로 확인하지 않았다. | `apps/desktop/src/main/services/{TypoCorrectionService,SuggestionService,PersonalGraphService,TextInsertService}.ts`, `packages/core/src/{typo-correction,writing-context,personal-graph}.ts` | `run-desktop.bat` 재시작 후 메모장·카톡·크롬 입력창에서: (1) `됬어요 ` 치고 멈추면 `됐어요 ` 로 바뀌는지, (2) 한글 문장 중 `dkssudgktpdy` 가 `안녕하세요` 로 바뀌는지, (3) 교체 도중 계속 치면 교체가 취소되고 글이 섞이지 않는지, (4) 고친 단어를 원래대로 되돌리면 다시 고치지 않는지, (5) 앱마다 백스페이스 한 번이 한 글자를 지우는지(리치 에디터·웹 에디터), (6) 교정 뒤 제안이 고친 문장 기준으로 뜨는지. 문제가 있는 앱은 제외 목록 또는 앱별 차단 규칙으로 다룬다. |
| GAP-INPUT-09 | Input intelligence | `[~]` **prefill 이어 쓰기·문서/대화 문맥(INPUT-21) 실앱 확인 (2026-10-03).** 실제 `gemma4:e4b` 프로브 15개 시나리오와 단위 테스트는 통과했지만, (a) 외부 앱 UIA 가 커서 앞 문서 전체를 주는지(케어렛 미상 앱은 tail 폴백), (b) 채팅 앱이 보내기 뒤 입력칸을 실제로 비우는 스냅샷을 내는지(카톡은 UIA 텍스트 길이 0 인 경우가 있었다 — GAP-INPUT-01), (c) 단어 중간 수락(`attach`)이 실제 삽입에서 공백 없이 붙는지는 GUI 로 확인하지 않았다. 후보 다양성: 마저 쓰기 모드는 prefill 분포가 뾰족해 채우기 후보가 1~2개에서 멈출 때가 많다. | `apps/desktop/src/main/services/{SuggestionService,InputTelemetryService,LocalLLMService,llm-prompts}.ts`, `packages/core/src/{chat-template,recent-writes,suggestion-text,writing-context}.ts`, `apps/desktop/tests/integration/suggestion-quality.ollama.test.ts` | `run-desktop.bat` 재시작 후: (1) 긴 메모장 문서 끝에서 앞부분의 날짜·이름을 이어 받는지, (2) 슬랙/디스코드 같은 Chromium 채팅에서 메시지를 보낸 뒤 다음 메시지 제안이 앞 메시지를 이어 받는지(로그 `제안 판단 입력` + 프롬프트 길이), (3) `내일 3시에 회의실` 같은 단어 중간에서 수락하면 `회의실에서…` 로 붙는지, (4) 지어낸 사실로 후보를 버린 로그(`제안 후보 버림 (unsupported-facts…)`)가 과하게 잦지 않은지. 모델 교체 시 `D3RO_SUGGESTION_PROBE=1 D3RO_SUGGESTION_PROBE_MODEL=<태그>` 로 같은 프로브를 다시 돌린다. **2026-10-03 19:00 "제안 창이 잘 안 뜸" 로그 분석**: 보류 1위 WindowsTerminal(Claude Code) `caret-unknown` 721회 — 입력 상자 아래 상태줄 때문에 화면 전체 기준으로 매번 "문서 중간 편집" 판정 → `SuggestionService._terminalContext` 가 입력 줄 연속성(늘거나 줄면 커서=끝)으로 판단. 2위 Chrome `not-typing` 122회 — 타이핑 중 UIA 포커스가 입력칸↔legacy 요소를 오가며 기준선을 다시 잡음 → 같은 창·클릭 없이 치는 중의 깜빡임이면 편집한 칸 유지(`input-telemetry-focus-flicker.test.ts`). 모든 앱에서 caret 오프셋을 못 얻는 것(caret=-1)과 Kitedesk/Candy/Paca 의 legacy(읽기 불가)는 별도 과제. 19:10 재시작 뒤 6분간 첫 후보 10회 생성(터미널 caret-unknown 721→4). **수락 키 전달 문제**: uiohook 은 키를 엿듣기만 해서 Ctrl+Alt+Enter 가 Claude Code 에도 전달돼 개행이 들어갔다 → 제안 창이 떠 있는 동안만 제안 단축키를 globalShortcut(noop)으로 잡아 OS 가 삼키게 하고 창이 사라지면 해제(`KeyBindingService.setSuggestionCaptureActive`, `suggestionCaptureAccelerators`). 실앱에서 수락 시 개행이 더는 안 들어가는지, 창이 없을 때 다른 앱의 Ctrl+Alt+Enter/↑↓ 가 정상인지 확인 필요. |
| GAP-SYNC-01 | Sync | Desktop↔mobile sync was one-way and lossy. (1) `pushAll` set the cursor to *now* before `pullAll` ran, so the first pull after sign-in fetched nothing — mobile-created data never reached the desktop except via Realtime while running (which never connected, GAP-SYNC-02). (2) `_inArray` used only the first meeting id, so memos/documents of every other meeting were never pushed. (3) No deletion propagation either way. (4) Pull had no `user_id` filter — team meetings of other users landed in the personal DB and a desktop re-push cleared `team_id`. (5) One global, local-clock cursor across accounts; no pagination (1,000-row cap). (6) Desktop writes did not move `history.revision`, so mobile's optimistic concurrency silently overwrote desktop edits. (7) One case-duplicate dictionary word failed the whole batch. (8) Memo tags, custom commands, templates, favorites and history titles were not synced at all; anonymous local-mode data was never uploaded after sign-in. | `apps/desktop/src/main/services/sync/*`, `CloudSyncService.ts`, `server/supabase/migrations/20260927000034_cross_device_sync_parity.sql` | `[x]` 2026-09-27: engine rewritten (outbox + per-user server-clock keyset cursors + tombstones + per-row isolation + dictionary adoption + natural-key memo tags + client-keyed template RPCs + local-mode import). Evidence: `apps/desktop/tests/main/sync/SyncEngine.test.ts` (20) + `apps/desktop/tests/integration/cross-device-sync.supabase.test.ts` (5, local Supabase stack); full desktop suite 1499/1500 (1 = known caption env timeout); SQL checks for revision bump, tombstones, account-delete cascade. |
| GAP-SYNC-02 | Sync | **Desktop Realtime never connected.** Electron 33 main (Node 20.18) has no global `WebSocket`; supabase-js Realtime only ever logged `TIMED_OUT`/`CLOSED` (installed-app log 2026-09-16) and gave up after three retries. Also: filtered `postgres_changes` channels never deliver DELETE (Supabase limitation). | `CloudSyncService.ts`, `services/sync/realtime-transport.ts`, integration test "Realtime" case | `[x]` 2026-09-27: `ws` injected as the Realtime transport (`ws` now a declared desktop dependency, externalised in the main bundle); deletions arrive as `sync_tombstones` INSERT events; the heartbeat re-subscribes a channel that is not `joined`. Verified live against the local stack (INSERT/UPDATE + tombstone events received). |