feat(desktop): built-in Han/Eng layout auto-switching, terminal/Vim support, and Unicode input
This commit is contained in:
parent
1da04b9e46
commit
a3ac44da2b
109 changed files with 8234 additions and 242 deletions
|
|
@ -83,12 +83,13 @@ Singleton + `EventEmitter` pattern (`getXService()` accessors).
|
|||
### Input intelligence (2026-09-21; Local Flow Intelligence extension 2026-09-22)
|
||||
| Service / file | Purpose |
|
||||
|---|---|
|
||||
| `InputTelemetryService` | Global input telemetry: keystroke/click/scroll counters, mouse travel (Manhattan sum of per-axis deltas), active time, per-hour-per-app buckets flushed every 5 s; typed-text learning via UIA snapshot diffs. It extends the existing telemetry store with Flow Radar (hourly activity density + chars + edit stability; a potential-focus score, **not** a real session), Edit Friction (chars/backspaces; edits per 100 chars), App Quality (per-app suggestion total/accepted/accept rate/average latency), Privacy Receipt (actual row counts and retention), and Smart Exclusion evidence. Raw key events, key codes and content streams are never retained; separately, opt-in learned text may be retained in `typing_samples` and `personal_phrases` under the stated receipt policy. Consent is opt-in (`inputTelemetryEnabled`, `inputLearnTypedText`). |
|
||||
| `InputTelemetryService` | Global input telemetry and context observer: keystroke/click/scroll counters, mouse travel (Manhattan sum of per-axis deltas), active time, per-hour-per-app buckets flushed every 5 s; typed-text learning via UIA snapshot diffs. **Since 2026-10-26**, the UIA context observer runs by default whenever `typoAutoCorrectEnabled !== false` (or `suggestionEnabled === true`), while SQLite `input_activity` persistence only runs when `inputTelemetryEnabled === true` (opt-in). **Since 2026-10-27**, includes `TerminalKeystrokeTracker` (`@d3ro/core/input-intelligence`) + `isForegroundKoreanIme` (`utils/win32-foreground.ts`) to reconstruct the active terminal/Vim line (`reconcileWithScreen` for UIA screen buffers like Windows Terminal/pwsh/cmd, `reconcileWithXtermValue` for `xterm.js` `<textarea class="xterm-helper-textarea">` controls in Agent Switchboard/VS Code/Cursor where `.value` stays populated until Enter/Ctrl+C, and `getSnapshot` for canvas/remote terminals with `len=0` like Kitedesk/WezTerm/Alacritty), preserving trailing spaces stripped by grid terminals and tracking Vim modes (`insert`, `normal`, `command`). Programmatic keys (`_programmaticKeys`) no longer advance `_lastKeyAt`, and `Enter`-triggered chat submissions that clear or shrink the input field rebase focus text immediately so cleared/toolbar text does not trigger post-send suggestions. It extends the telemetry store with Flow Radar (hourly activity density + chars + edit stability; a potential-focus score, **not** a real session), Edit Friction (chars/backspaces; edits per 100 chars), App Quality (per-app suggestion total/accepted/accept rate/average latency), Privacy Receipt (actual row counts and retention), and Smart Exclusion evidence. Raw key events, key codes and content streams are never retained; separately, opt-in learned text may be retained in `typing_samples` and `personal_phrases` under the stated receipt policy. |
|
||||
| `TypoCorrectionService` | Built-in automatic Han/Eng layout conversion and typo auto-correction (`INPUT-19`). Runs on each `typed-context`: Araso-style layout conversions (`source: 'layout'`) fire after 25 ms on word boundaries (Space/punctuation, 40 ms quick snapshot) or 135 ms at token tail (150 ms quick snapshot, even when Chromium/Discord keeps `isComposing=true`, committing IME composition first only when the trailing character is Hangul/jamo), replacing 2-set Han/Eng layout errors, mixed English+Korean words (`githubdp`→`github에`, `PDFvkdlf`→`PDF파일`, `ㅖㅇㄹ파일`→`PDF파일`), lingering Shift on double consonants (`djEJs`→`어떤`), reverse Hangul-to-English (`ㅎ샤ㅗㅕㅠ에`→`github에`, `ㅑㄹ ㅑ 쇼ㅔㄷ ㅗ디ㅣㅐ 소두 ㅑㅅ 초뭏ㄷㄴ ㅑㅜ 10 ㄴㄷ채ㅜㅇㄴ`→`if I type hello then it changes in 10 seconds`), and partially-converted tokens (`안녕gktpdy`→`안녕하세요`). **In terminals and Vim (`terminalMode: true`)**, layout-only conversion stays active while spelling/model checks are suppressed: converts CLI commands (`퍄ㅡ `→`vim `, `ㅜ퍄ㅡ `→`nvim `, `ㅎ샤 `→`git `, `ㅣㄴ `→`ls `, `채ㅜㄴㅅ `→`const `), Vim Insert-mode typing (stripping leading `i/a/o/I/A/O/s/S` mode-entry keys when present in raw keystroke buffers), and Vim Command-line ex-commands (`:ㅈㅂ`→`:wq`, `:ㅈㅂ!`→`:wq!`, `:ㅈ`→`:w`, `:ㅂ`→`:q`, `:ㅂ!`→`:q!`, `:ㅌ`→`:x`, `:ㅍ네`→`:vsp`, `:네`→`:sp`, `:ㅜㅐㅗ`→`:noh`, preserving `:`), while blocking all conversions in Vim Normal mode (`vimMode === 'normal'`). On Windows, `replaceBeforeCaret` uses `SendInput(KEYEVENTF_UNICODE)` (`utils/win32-unicode-input.ts`) instead of `Ctrl+V` so it never touches the user's clipboard or collides with `ClipboardSvcGroup` `OpenClipboard` locks. Deterministic/model spelling fixes run after a 600 ms pause. When `SuggestionService` is enabled, `handleTypingContext(context, { layoutOnly: true })` still auto-converts layout mistypes immediately in-place while `propose()` surfaces spelling/model fixes inside the suggestion popup (`INPUT-22`). |
|
||||
| `UiaContextService` | Client for the sidecar UIA bridge (`GET /uia/focus`): focused text, caret rect/offset, `IsPassword`, IME composition state and `hasSelection`. The sidecar derives `hasSelection` only by comparing the TextPattern selection range Start/End endpoints: it does not call `GetText` on that range or add a separate selected-text payload (the existing focused-field text can still include a selection). Fail-closed on password/unavailable; transient failures back off for 1.5/3/6/12/30 s and permanent failures for 5 min. |
|
||||
| `SuggestionService` | Next-sentence ghost text: 600 ms debounce, then `buildSuggestionPrompt` → `LocalLLMService.streamGenerate` with a caller-owned `AbortSignal` → candidate ranking and accept (via `TextInsertService`). Presentation-active includes candidates plus `generating`, `warmingUp` and `partialText`; clear/dismiss aborts the active request, invalidates its generation token, clears TTL state, then emits `cleared`/hide so a spinner-only popup closes. A successful final candidate payload resets `generating=false` and `partialText=null`. App DNA extends phrase ranking with `appName` context and a 1.75× same-app bonus; Memory Decay uses a 30-day half-life plus frequency. Why This Suggestion exposes only `local-model`/`local-memory` provenance and continuation/related/phrase/appPhrase **counts** to the overlay, never raw evidence text. Instant Recall uses local-memory provenance without another model-budget spend only when the model is unavailable, raises a non-cancellation error, truly times out, or returns empty; it never publishes after dismiss, new typing, token/context mismatch or staleness. Its runaway limits are a 5 s minimum interval, 6 requests/min default (hard-config maximum 12), 3 candidates, 64 output tokens, 12-character regeneration growth and an 8 s timeout; app startup does not warm the model and suggestion requests use `keep_alive: 2m`. |
|
||||
| `global-input-hook.ts` | Ref-counted owner of the single process-wide `uiohook` hook so `KeyBindingService` and telemetry can both attach without one stopping the other. |
|
||||
| `utils/win32-foreground.ts` | Foreground window title/pid/exe/bounds via `koffi` FFI into user32/kernel32 (chosen over `get-windows`, which needs an install script this repo does not run). |
|
||||
| `sidecar/uia_bridge.py` | Windows UIA snapshot (`uiautomation` 2.0.29, comtypes) on a dedicated COM-initialised thread with a 1.5 s budget; never walks the UIA tree (Chrome/VS Code tree walks take 10-30 s). Warm calls measure 0-15 ms. |
|
||||
| `utils/win32-foreground.ts` | Foreground window title/pid/exe/bounds and active thread Korean IME state (`isForegroundKoreanIme` via `GetGUIThreadInfo` + `ImmGetDefaultIMEWnd` + `SendMessageTimeoutW(WM_IME_CONTROL)`) via `koffi` FFI into user32/kernel32/imm32 (chosen over `get-windows`, which needs an install script this repo does not run). |
|
||||
| `sidecar/uia_bridge.py` | Windows UIA snapshot (`uiautomation` 2.0.29, comtypes) on a dedicated COM-initialised thread with a 1.5 s budget; never walks the UIA tree (Chrome/VS Code tree walks take 10-30 s). Uses `GetVisibleRanges()` / tail slicing `[-MAX_TEXT_CHARS:]` for large terminal scrollback buffers and `ExpandToEnclosingUnit(..., waitTime=0)` to avoid `uiautomation.py`'s default 500 ms sleep on Windows Terminal. Warm calls measure 0-15 ms. |
|
||||
|
||||
The existing `input-telemetry-handlers` and `suggestion-handlers` IPC extensions expose the receipt and the flow/suggestion summaries. Receipt reads or deletes report an IPC error when storage access fails; they do not return invented counts or claim a purge succeeded. The receipt is local-only: `input_activity`, `typing_samples`, and `suggestions` retain 30 days; `personal_phrases` has no age-based automatic expiry and is removed by individual deletion, delete-all, or consent withdrawal. Smart Exclusion never records password evidence and recommends only the current app after at least four observations, zero readable results, and a problematic ratio of at least 75%; its one-click action adds an exclusion and never auto-excludes. Shortcut Safety Audit is core `auditKeyBindingMap` applied by the Settings UI; it surfaces invalid/conflict issues while preserving the existing hold/double-press exceptions.
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue