fix(suggestion): insert accepted suggestions reliably and learn typing as whole sentences
Acceptance waited for modifiers using only the uiohook-tracked set; a missed keyup made it time out after 1.5 s and paste while Ctrl/Alt counted as held, so nothing was inserted. Trust GetAsyncKeyState instead, and suppress new suggestions while an accept is in flight. Typed deltas were saved per snapshot (every 0.4-0.8 s), storing mid-word fragments. Buffer them per field and save one diff on idle, sentence end, send or focus change; drop short pieces, URLs, trailing jamo and truncated emoji.
This commit is contained in:
parent
006909065f
commit
cdeeff4c11
7 changed files with 341 additions and 23 deletions
|
|
@ -168,9 +168,9 @@ end-to-end behaviour has **not been verified by typing in a real app** (`11` GAP
|
|||
| INPUT-01 | Keyboard/mouse telemetry capture (opt-in) | [~] | [-] | [-] | [-] | `InputTelemetryService` — keystroke/click/scroll counters, mouse travel as the Manhattan sum of per-axis pixel deltas, active time, per-hour×app buckets flushed every 5 s. Key **contents** are never stored (ActivityWatch `aw-watcher-input` data-minimisation policy, adopted deliberately — see `11` §7). Hook ownership is ref-counted so `KeyBindingService` keeps working (`global-input-hook.ts`). |
|
||||
| INPUT-02 | Foreground-app attribution | [~] | [-] | [-] | [-] | `utils/win32-foreground.ts` via `koffi` FFI (title/pid/exe/bounds), sampled at most 1×/s. `get-windows` was rejected: it needs an install script this repo does not run. |
|
||||
| INPUT-03 | Weekly input insights | [~] | [-] | [-] | [-] | `INPUT_TELEMETRY.getSummary` aggregates `input_activity` into totals, daily series, top hours and top apps; rendered in Settings → Input and as a dashboard card. Daily average mouse travel is converted px → m using the display scale factor. |
|
||||
| INPUT-04 | Typed-text learning (UIA, password-excluded) | [~] | [-] | [-] | [-] | Text is read from the focused field via `GET /uia/focus` (sidecar UIA bridge) and diffed longest-common-prefix/suffix, so **IME-committed Hangul/kana is counted correctly** — keycodes cannot reconstruct CJK text. UIA sends `hasSelection` only, derived by TextPattern range Start/End comparison without calling `GetText` on the selection range or adding a selected-text payload; the existing focused-field text can still include a selection. A non-collapsed selection immediately clears suggestions as `selection-active`. `IsPassword` is checked before any read (fail-closed); IME composition suppresses both stats and suggestions. **2026-09-24:** text equal to the control name (empty-field placeholder, e.g. "메시지 입력") is treated as empty (`withoutPlaceholderText`); a suggestion requires `editedSinceFocus` + typing within `recentTypingWindowMs` (8 s), otherwise `not-typing` — clicking into a pre-filled field no longer triggers suggestions. |
|
||||
| INPUT-04 | Typed-text learning (UIA, password-excluded) | [~] | [-] | [-] | [-] | Text is read from the focused field via `GET /uia/focus` (sidecar UIA bridge) and diffed longest-common-prefix/suffix, so **IME-committed Hangul/kana is counted correctly** — keycodes cannot reconstruct CJK text. UIA sends `hasSelection` only, derived by TextPattern range Start/End comparison without calling `GetText` on the selection range or adding a selected-text payload; the existing focused-field text can still include a selection. A non-collapsed selection immediately clears suggestions as `selection-active`. `IsPassword` is checked before any read (fail-closed); IME composition suppresses both stats and suggestions. **2026-09-24:** text equal to the control name (empty-field placeholder, e.g. "메시지 입력") is treated as empty (`withoutPlaceholderText`); a suggestion requires `editedSinceFocus` + typing within `recentTypingWindowMs` (8 s), otherwise `not-typing` — clicking into a pre-filled field no longer triggers suggestions. **2026-10-24:** typed deltas are no longer saved per snapshot (that stored mid-word fragments every 0.4–0.8 s). They are buffered per field and saved as one baseline→latest diff on 2 s idle, sentence-ending punctuation, 150-char growth, field clear (send), focus change or stop. Typed samples under 4 non-space chars, URL/query-like lines, trailing incomplete jamo and PUA/lone-surrogate chars (emoji truncated to 16 bits upstream, root cause unconfirmed) are dropped. Rows already in the DB are not cleaned up. |
|
||||
| INPUT-05 | Personal phrase corpus (typed + voice) | [~] | [-] | [-] | [-] | Sentence-level phrases from typed text and from voice history (`HistoryService.create` feeds `recordExternalText`), ranked by frequency/recency as prompt hints; users can delete individual phrases. **2026-09-24:** typed text from `LEARNING_EXCLUDED_APPS` (terminals, code editors, Agent Switchboard) is never learned, and every phrase must pass `isLearnablePhrase` (no box/block/geometric glyphs, ≥60 % letters). `_pruneUnlearnableCorpus` re-applies both rules to existing phrases/samples/edges on start and every retention cycle (first run on the author machine removed 69 phrases / 91 samples of terminal status lines and agent chats). |
|
||||
| INPUT-06 | Next-sentence suggestion (ghost text) | [~] | [-] | [-] | [-] | `SuggestionService` + `buildSuggestionPrompt` (`llm-prompts.ts` SSOT, instruction stays in the system prompt). The 2026-09-22 guard contract is 600 ms debounce, ≥5 s between requests, 6 requests/min by default (hard-config maximum 12), 3 candidates, 64 output tokens, 12-character growth before regeneration, 8 s request timeout and `keep_alive: 2m`; boot warmup is removed. Each request has its own cancellation signal. Presentation-active includes candidates, `generating`, `warmingUp` and `partialText`; clear/dismiss aborts, invalidates the generation token, clears TTL state and emits `cleared`/hide, and a final success resets `generating=false`/`partialText=null`. Focused evidence for the lifecycle and Windows-child-process change: five test files / 80 tests passed; desktop typecheck/lint, Python `py_compile`, and `git diff --check` exited 0 (core 131-test pass was independently verified earlier). Status remains `[~]`: this is not app-restart, GUI overlay, or real automatic-typing evidence. **2026-09-24 (supersedes the numbers above):** `keep_alive: 10m`, warm-up on boot, on Ollama reconnect and whenever the model is presumed cold (cold reload measured 11.4 s > 8 s timeout); a session asks for 1 candidate, then a sequential fill loop appends one unique candidate at a time up to `maxCandidatesTotal` 12 (avoid-list in the prompt, stops after 2 empty fills); only the first request consumes rate/daily budget; stale aborts refund the budget; any further typing ends the session (`matchesSessionPrefix`, IME last-syllable tolerant); prompt forbids assistant-style questions. Terminals (`TERMINAL_APPS`) never get suggestions. |
|
||||
| INPUT-06 | Next-sentence suggestion (ghost text) | [~] | [-] | [-] | [-] | `SuggestionService` + `buildSuggestionPrompt` (`llm-prompts.ts` SSOT, instruction stays in the system prompt). The 2026-09-22 guard contract is 600 ms debounce, ≥5 s between requests, 6 requests/min by default (hard-config maximum 12), 3 candidates, 64 output tokens, 12-character growth before regeneration, 8 s request timeout and `keep_alive: 2m`; boot warmup is removed. Each request has its own cancellation signal. Presentation-active includes candidates, `generating`, `warmingUp` and `partialText`; clear/dismiss aborts, invalidates the generation token, clears TTL state and emits `cleared`/hide, and a final success resets `generating=false`/`partialText=null`. Focused evidence for the lifecycle and Windows-child-process change: five test files / 80 tests passed; desktop typecheck/lint, Python `py_compile`, and `git diff --check` exited 0 (core 131-test pass was independently verified earlier). Status remains `[~]`: this is not app-restart, GUI overlay, or real automatic-typing evidence. **2026-09-24 (supersedes the numbers above):** `keep_alive: 10m`, warm-up on boot, on Ollama reconnect and whenever the model is presumed cold (cold reload measured 11.4 s > 8 s timeout); a session asks for 1 candidate, then a sequential fill loop appends one unique candidate at a time up to `maxCandidatesTotal` 12 (avoid-list in the prompt, stops after 2 empty fills); only the first request consumes rate/daily budget; stale aborts refund the budget; any further typing ends the session (`matchesSessionPrefix`, IME last-syllable tolerant); prompt forbids assistant-style questions. Terminals (`TERMINAL_APPS`) never get suggestions. **2026-10-24 accept fix:** `waitForModifiersReleased` now trusts OS `GetAsyncKeyState` (`utils/win32-key-state.ts`) over the uiohook-tracked set. A missed keyup made acceptance wait the full 1.5 s and then paste Ctrl+V while modifiers were still considered held; in the logged case the field length stayed 15 after "inserted" (6 chars). New suggestions are also suppressed while an accept is in flight, so the overlay no longer reappears during the wait. |
|
||||
| INPUT-07 | Caret-anchored suggestion overlay | [~] | [-] | [-] | [-] | `suggestion-overlay` popup placed by `anchorFloatingPanel` (caret → element → cursor fallback, flip above when the caret is near the bottom, clamped to the work area). Non-focusable; click-through unless `suggestionOverlayInteractive`. While actually visible it continues periodic UIA validation after 5 s and revalidates 120 ms after mouse-up; unavailable/non-editable focus or a non-collapsed selection aborts and hides it. X first hides the renderer panel, then main IPC directly hides `BrowserWindow` and dismisses the service, so late tokened results cannot revive it. Accept/next/prev/dismiss are four global key bindings (`suggestion-accept`/`next`/`prev`/`dismiss`, default `Ctrl+Alt+→/↓/↑/←`), and the overlay has a mouse close button. Up to three candidates are shown in a scrollable list with a warm-up/generating spinner. **2026-09-24:** one page of 3 fixed-height (2-line) numbered items, footer = range/progress + source + key guide derived from the live bindings (`buildSuggestionKeyHints`, shared modifier shown once). Keys: Ctrl+Alt+↑/↓ move (the page follows the active item; at the last item `next` waits while filling, otherwise wraps; the 1.6.0 Left/Right page actions were removed as unreachable before a page filled and colliding with Intel display rotation), Ctrl+Alt+Enter accept, plain Esc closes while visible (overlay-scoped, not a binding), Ctrl+Alt+Backspace secondary dismiss; old default bindings migrate at tuning revision 5. Streaming updates never reposition/re-present the window (`decideSuggestionOverlayAction`); placement uses `anchorKind` — caret: below the line, element: outside the element (below → above → right → left → inner bottom-right). Navigation re-arms the visible TTL. Window 460×208. |
|
||||
| INPUT-08 | Per-app exclusions & consent controls | [~] | [-] | [-] | [-] | `inputExcludedApps` (executable names, case-insensitive) blocks both collection context and suggestions; telemetry master switch, pause, text-learning toggle and "delete collected data" all live in Settings → Input. 30-day retention prune runs on start. |
|
||||
| INPUT-09 | Flow Radar | [~] | [-] | [-] | [-] | `rankFlowWindows` ranks hourly aggregate activity density, character volume and edit stability into potential-focus time windows. It is not a real-session detector or session record. |
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue