fix(desktop): prevent modifier key buffer resets, clipboard race on terminal accept, and redundant UIA settle snapshots
This commit is contained in:
parent
6d909dd0e5
commit
41901c1ec5
10 changed files with 198 additions and 23 deletions
|
|
@ -168,9 +168,9 @@ end-to-end behaviour has **not been verified by typing in a real app** (`11` GAP
|
|||
| INPUT-01 | Keyboard/mouse telemetry capture (opt-in) | [~] | [-] | [-] | [-] | `InputTelemetryService` — keystroke/click/scroll counters, mouse travel as the Manhattan sum of per-axis pixel deltas, active time, per-hour×app buckets flushed every 5 s. Key **contents** are never stored (ActivityWatch `aw-watcher-input` data-minimisation policy, adopted deliberately — see `11` §7). Hook ownership is ref-counted so `KeyBindingService` keeps working (`global-input-hook.ts`). |
|
||||
| INPUT-02 | Foreground-app attribution | [~] | [-] | [-] | [-] | `utils/win32-foreground.ts` via `koffi` FFI (title/pid/exe/bounds), sampled at most 1×/s. `get-windows` was rejected: it needs an install script this repo does not run. |
|
||||
| INPUT-03 | Weekly input insights | [~] | [-] | [-] | [-] | `INPUT_TELEMETRY.getSummary` aggregates `input_activity` into totals, daily series, top hours and top apps; rendered in Settings → Input and as a dashboard card. Daily average mouse travel is converted px → m using the display scale factor. |
|
||||
| INPUT-04 | Typed-text learning (UIA, password-excluded) | [~] | [-] | [-] | [-] | Text is read from the focused field via `GET /uia/focus` (sidecar UIA bridge) and diffed longest-common-prefix/suffix, so **IME-committed Hangul/kana is counted correctly** — keycodes cannot reconstruct CJK text. UIA sends `hasSelection` only, derived by TextPattern range Start/End comparison without calling `GetText` on the selection range or adding a selected-text payload; the existing focused-field text can still include a selection. A non-collapsed selection immediately clears suggestions as `selection-active`. `IsPassword` is checked before any read (fail-closed); IME composition suppresses both stats and suggestions. **2026-09-24:** text equal to the control name (empty-field placeholder, e.g. "메시지 입력") is treated as empty (`withoutPlaceholderText`); a suggestion requires `editedSinceFocus` + typing within `recentTypingWindowMs` (8 s), otherwise `not-typing` — clicking into a pre-filled field no longer triggers suggestions. **2026-10-29:** `_typedInCurrentFocus` and `_lastTypedHwnd` track whether a typing keystroke occurred after the latest click/focus rebase (`_rebaseFocusText`), and `_sampleForeground` forces immediate foreground sampling on that first keystroke, so typing immediately after clicking into a search box or new window sets `editedSinceFocus = true` on the very first snapshot. **2026-10-24:** typed deltas are no longer saved per snapshot (that stored mid-word fragments every 0.4–0.8 s). They are buffered per field and saved as one baseline→latest diff on 2 s idle, sentence-ending punctuation, 150-char growth, field clear (send), focus change or stop. Typed samples under 4 non-space chars, URL/query-like lines, trailing incomplete jamo and PUA/lone-surrogate chars (emoji truncated to 16 bits upstream, root cause unconfirmed) are dropped. Rows already in the DB are pruned on start by the same rule (`isLearnableTypedText` in `_pruneUnlearnableCorpus`, typed source only; dry count on the dev DB 2026-10-24: 1,235/2,664 samples, 285/1,209 phrases). Expressive jamo runs (ㅋㅋ, ㅠㅠ, ㅇㅇ) and space-less questions are kept. Mid-word fragments of 4+ chars cannot be detected and remain until the 30-day retention. **2026-10-30: UIA reads are single-flight and discarded after input/focus/lifecycle changes, stale capture times, or changed native GetLastInputInfo tokens; contexts carry inputRevision/nativeInputTick for replacement-time validation (InputTelemetryService, win32-input-activity). See GAP-INPUT-11 for load-test gate.** |
|
||||
| INPUT-04 | Typed-text learning (UIA, password-excluded) | [~] | [-] | [-] | [-] | Text is read from the focused field via `GET /uia/focus` (sidecar UIA bridge) and diffed longest-common-prefix/suffix, so **IME-committed Hangul/kana is counted correctly** — keycodes cannot reconstruct CJK text. UIA sends `hasSelection` only, derived by TextPattern range Start/End comparison without calling `GetText` on the selection range or adding a selected-text payload; the existing focused-field text can still include a selection. A non-collapsed selection immediately clears suggestions as `selection-active`. `IsPassword` is checked before any read (fail-closed); IME composition suppresses both stats and suggestions. **2026-09-24:** text equal to the control name (empty-field placeholder, e.g. "메시지 입력") is treated as empty (`withoutPlaceholderText`); a suggestion requires `editedSinceFocus` + typing within `recentTypingWindowMs` (8 s), otherwise `not-typing` — clicking into a pre-filled field no longer triggers suggestions. **2026-10-29:** `_typedInCurrentFocus` and `_lastTypedHwnd` track whether a typing keystroke occurred after the latest click/focus rebase (`_rebaseFocusText`), and `_sampleForeground` forces immediate foreground sampling on that first keystroke, so typing immediately after clicking into a search box or new window sets `editedSinceFocus = true` on the very first snapshot. **2026-10-24:** typed deltas are no longer saved per snapshot (that stored mid-word fragments every 0.4–0.8 s). They are buffered per field and saved as one baseline→latest diff on 2 s idle, sentence-ending punctuation, 150-char growth, field clear (send), focus change or stop. Typed samples under 4 non-space chars, URL/query-like lines, trailing incomplete jamo and PUA/lone-surrogate chars (emoji truncated to 16 bits upstream, root cause unconfirmed) are dropped. Rows already in the DB are pruned on start by the same rule (`isLearnableTypedText` in `_pruneUnlearnableCorpus`, typed source only; dry count on the author machine 2026-10-24: 1,235/2,664 samples, 285/1,209 phrases). Expressive jamo runs (ㅋㅋ, ㅠㅠ, ㅇㅇ) and space-less questions are kept. Mid-word fragments of 4+ chars cannot be detected and remain until the 30-day retention. **2026-10-30..31: UIA reads are single-flight and discarded after input/focus/lifecycle changes, stale capture times, or changed native GetLastInputInfo tokens; contexts carry inputRevision/nativeInputTick for replacement-time validation (`InputTelemetryService`, `win32-input-activity`). Standalone modifier keydowns/key-repeats (`Ctrl`/`Alt`/`Shift`/`Win`) and `Ctrl+Alt` suggestion navigation chords are ignored in `_handleKeyDown` without advancing `_inputRevision`/`_lastKeyAt` or scheduling UIA snapshots (`classifyKeyStroke` checks modifier VKs before modifier flags), and `_settleTimer` (780 ms) is skipped on non-typing keys and auto-cancelled when the 380 ms `_textTimer` snapshot cannot trigger a suggestion (`prefixLen < minPrefixChars`). See GAP-INPUT-11.** |
|
||||
| INPUT-05 | Personal phrase corpus (typed + voice) | [~] | [-] | [-] | [-] | Sentence-level phrases from typed text and from voice history (`HistoryService.create` feeds `recordExternalText`), ranked by frequency/recency as prompt hints; users can delete individual phrases. **2026-09-24:** typed text from `LEARNING_EXCLUDED_APPS` (terminals, code editors, Agent Switchboard) is never learned, and every phrase must pass `isLearnablePhrase` (no box/block/geometric glyphs, ≥60 % letters). `_pruneUnlearnableCorpus` re-applies both rules to existing phrases/samples/edges on start and every retention cycle (first run on the author machine removed 69 phrases / 91 samples of terminal status lines and agent chats). **2026-10-25:** Agent Switchboard moved from `LEARNING_EXCLUDED_APPS` to `APP_SCOPED_LEARNING_APPS` — its text is kept only as `typing_samples` (with app name), never in shared phrases/graph, and is used only for suggestions inside that same app (`findContinuationHints(..., { appName })` feeds `continuationHints`). It is the user's most-typed app; the 2026-09-24 exclusion was about dev instructions leaking into KakaoTalk suggestions, which scoping prevents. |
|
||||
| INPUT-06 | Next-sentence suggestion (ghost text) | [~] | [-] | [-] | [-] | `SuggestionService` + `buildSuggestionPrompt` (`llm-prompts.ts` SSOT, instruction stays in the system prompt). The 2026-09-22 guard contract is 600 ms debounce, ≥5 s between requests, 6 requests/min by default (hard-config maximum 12), 3 candidates, 64 output tokens, 12-character growth before regeneration, 8 s request timeout and `keep_alive: 2m`; boot warmup is removed. Each request has its own cancellation signal. Presentation-active includes candidates, `generating`, `warmingUp` and `partialText`; clear/dismiss aborts, invalidates the generation token, clears TTL state and emits `cleared`/hide, and a final success resets `generating=false`/`partialText=null`. Focused evidence for the lifecycle and Windows-child-process change: five test files / 80 tests passed; desktop typecheck/lint, Python `py_compile`, and `git diff --check` exited 0 (core 131-test pass was independently verified earlier). Status remains `[~]`: this is not app-restart, GUI overlay, or real automatic-typing evidence. **2026-09-24 (supersedes the numbers above):** `keep_alive: 10m`, warm-up on boot, on Ollama reconnect and whenever the model is presumed cold (cold reload measured 11.4 s > 8 s timeout); a session asks for 1 candidate, then a sequential fill loop appends one unique candidate at a time up to `maxCandidatesTotal` 12 (avoid-list in the prompt, stops after 2 empty fills); only the first request consumes rate/daily budget; stale aborts refund the budget; any further typing ends the session (`matchesSessionPrefix`, IME last-syllable tolerant); prompt forbids assistant-style questions. Terminals (`TERMINAL_APPS`) never get suggestions. **2026-10-24 accept fix:** `waitForModifiersReleased` now trusts OS `GetAsyncKeyState` (`utils/win32-key-state.ts`) over the uiohook-tracked set. A missed keyup made acceptance wait the full 1.5 s and then paste Ctrl+V while modifiers were still considered held; in the logged case the field length stayed 15 after "inserted" (6 chars). New suggestions are also suppressed while an accept is in flight, so the overlay no longer reappears during the wait. **2026-10-24~25 experiment, reverted:** an n-gram-first / confidence-gated / short-candidate rework raised offline next-word precision (3%→33%) but made suggestions rare, and the user judged the app much worse. Suggestion behaviour is back to v1.10.1. Kept: suggestions that copy on-screen text are rejected (`copiesContext`), no suggestions while typing an address/path (`isAddressInput`, skip reason `address-input`). Lessons and data: `docs/evidence/2026-10-24-suggestion-accuracy-research.md` §9; replay harness `tests/eval/`. **2026-10-25 additive upgrades (frequency unchanged):** (1) `suggestion-accept-word` (default Ctrl+Alt+Space) inserts only the first word and keeps the rest on screen (`splitFirstWord`; partial accepts are not learned as full sentences). (2) a confident personal n-gram candidate (`PersonalNgramIndex`, strict `PERSONAL_NGRAM_SETTINGS`) is shown instantly as candidate #1 while the model generates; model candidates are appended after it, and it survives an empty/failed/timed-out model run (no "no suggestions" notice then). Agent Switchboard text never enters this shared n-gram. Hotkey accept now force-releases held Ctrl/Alt/Shift after 200 ms (`prepareModifiersForPaste`). **2026-10-28 extreme personalization & n-gram upgrades (frequency unchanged):** `PersonalNgramModel` now weights repeated `personalPhrases` (`count >= 2`) and falls back to a Korean stemmed-word bigram index (`_stemBigram` via `stemTerm`, e.g. matching `"배포를 완료했습니다"` when typing `"배포가 "`), and `SuggestionService._fillOne` reranks newly generated LLM candidates by `scoreCandidatePersonalization` (overlap with personal graph `continuations`/`followUps`/`related`/`phrases` + register match) while preserving the instant n-gram slot at index 0. |
|
||||
| INPUT-06 | Next-sentence suggestion (ghost text) | [~] | [-] | [-] | [-] | `SuggestionService` + `buildSuggestionPrompt` (`llm-prompts.ts` SSOT, instruction stays in the system prompt). The 2026-09-22 guard contract is 600 ms debounce, ≥5 s between requests, 6 requests/min by default (hard-config maximum 12), 3 candidates, 64 output tokens, 12-character growth before regeneration, 8 s request timeout and `keep_alive: 2m`; boot warmup is removed. Each request has its own cancellation signal. Presentation-active includes candidates, `generating`, `warmingUp` and `partialText`; clear/dismiss aborts, invalidates the generation token, clears TTL state and emits `cleared`/hide, and a final success resets `generating=false`/`partialText=null`. Focused evidence for the lifecycle and Windows-child-process change: five test files / 80 tests passed; desktop typecheck/lint, Python `py_compile`, and `git diff --check` exited 0 (core 131-test pass was independently verified earlier). Status remains `[~]`: this is not app-restart, GUI overlay, or real automatic-typing evidence. **2026-09-24 (supersedes the numbers above):** `keep_alive: 10m`, warm-up on boot, on Ollama reconnect and whenever the model is presumed cold (cold reload measured 11.4 s > 8 s timeout); a session asks for 1 candidate, then a sequential fill loop appends one unique candidate at a time up to `maxCandidatesTotal` 12 (avoid-list in the prompt, stops after 2 empty fills); only the first request consumes rate/daily budget; stale aborts refund the budget; any further typing ends the session (`matchesSessionPrefix`, IME last-syllable tolerant); prompt forbids assistant-style questions. Terminals (`TERMINAL_APPS`) never get suggestions. **2026-10-24 accept fix:** `waitForModifiersReleased` now trusts OS `GetAsyncKeyState` (`utils/win32-key-state.ts`) over the uiohook-tracked set. A missed keyup made acceptance wait the full 1.5 s and then paste Ctrl+V while modifiers were still considered held; in the logged case the field length stayed 15 after "inserted" (6 chars). New suggestions are also suppressed while an accept is in flight, so the overlay no longer reappears during the wait. **2026-10-24~25 experiment, reverted:** an n-gram-first / confidence-gated / short-candidate rework raised offline next-word precision (3%→33%) but made suggestions rare, and the user judged the app much worse. Suggestion behaviour is back to v1.10.1. Kept: suggestions that copy on-screen text are rejected (`copiesContext`), no suggestions while typing an address/path (`isAddressInput`, skip reason `address-input`). Lessons and data: `docs/evidence/2026-10-24-suggestion-accuracy-research.md` §9; replay harness `tests/eval/`. **2026-10-25 additive upgrades (frequency unchanged):** (1) `suggestion-accept-word` (default Ctrl+Alt+Space) inserts only the first word and keeps the rest on screen (`splitFirstWord`; partial accepts are not learned as full sentences). (2) a confident personal n-gram candidate (`PersonalNgramIndex`, strict `PERSONAL_NGRAM_SETTINGS`) is shown instantly as candidate #1 while the model generates; model candidates are appended after it, and it survives an empty/failed/timed-out model run (no "no suggestions" notice then). Agent Switchboard text never enters this shared n-gram. Hotkey accept now force-releases held Ctrl/Alt/Shift after 200 ms (`prepareModifiersForPaste`). **2026-10-28 extreme personalization & n-gram upgrades (frequency unchanged):** `PersonalNgramModel` now weights repeated `personalPhrases` (`count >= 2`) and falls back to a Korean stemmed-word bigram index (`_stemBigram` via `stemTerm`, e.g. matching `"배포를 완료했습니다"` when typing `"배포가 "`), and `SuggestionService._fillOne` reranks newly generated LLM candidates by `scoreCandidatePersonalization` (overlap with personal graph `continuations`/`followUps`/`related`/`phrases` + register match) while preserving the instant n-gram slot at index 0. **2026-10-31 (`v1.15.2`) terminal accept & fill-stop fix:** when accepting a suggestion in `terminalMode === true` (`xterm-helper-textarea`, `WindowsTerminal`, `Kitedesk`, etc.), `SuggestionService._insertAccepted` uses `'keyboard'` (`sendWin32UnicodeText` / `sendWin32UnicodeReplacement` via `KEYEVENTF_UNICODE`) instead of `'clipboard'` (`Ctrl+V`) so stale OS clipboard contents (e.g., copied URLs) can never be pasted into the terminal on delayed paste processing, syncs `TerminalKeystrokeTracker` via `applyProgrammaticReplacement`, and stops `_fillRemainingCandidates` immediately when `accept()` is triggered (`suggestion-typo-correction.test.ts` 8/8 PASS). |
|
||||
| INPUT-07 | Caret-anchored suggestion overlay | [~] | [-] | [-] | [-] | `suggestion-overlay` popup placed by `anchorFloatingPanel` (caret → element → cursor fallback, flip above when the caret is near the bottom, clamped to the work area). Non-focusable; click-through unless `suggestionOverlayInteractive`. While actually visible it continues periodic UIA validation after 5 s and revalidates 120 ms after mouse-up; unavailable/non-editable focus or a non-collapsed selection aborts and hides it. X first hides the renderer panel, then main IPC directly hides `BrowserWindow` and dismisses the service, so late tokened results cannot revive it. Accept/next/prev/dismiss are four global key bindings (`suggestion-accept`/`next`/`prev`/`dismiss`, default `Ctrl+Alt+→/↓/↑/←`), and the overlay has a mouse close button. Up to three candidates are shown in a scrollable list with a warm-up/generating spinner. **2026-09-24:** one page of 3 fixed-height (2-line) numbered items, footer = range/progress + source + key guide derived from the live bindings (`buildSuggestionKeyHints`, shared modifier shown once). Keys: Ctrl+Alt+↑/↓ move (the page follows the active item; at the last item `next` waits while filling, otherwise wraps; the 1.6.0 Left/Right page actions were removed as unreachable before a page filled and colliding with Intel display rotation), Ctrl+Alt+Enter accept, plain Esc closes while visible (overlay-scoped, not a binding), Ctrl+Alt+Backspace secondary dismiss; old default bindings migrate at tuning revision 5. Streaming updates never reposition/re-present the window (`decideSuggestionOverlayAction`); placement uses `anchorKind` — caret: below the line, element: outside the element (below → above → right → left → inner bottom-right). Navigation re-arms the visible TTL. Window 460×208. |
|
||||
| INPUT-08 | Per-app exclusions & consent controls | [~] | [-] | [-] | [-] | `inputExcludedApps` (executable names, case-insensitive) blocks both collection context and suggestions; telemetry master switch, pause, text-learning toggle and "delete collected data" all live in Settings → Input. 30-day retention prune runs on start. |
|
||||
| INPUT-09 | Flow Radar | [~] | [-] | [-] | [-] | `rankFlowWindows` ranks hourly aggregate activity density, character volume and edit stability into potential-focus time windows. It is not a real-session detector or session record. |
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue