feat(commands): multi-select ordered command pipeline with step reordering and single-pass composition (AI-05, AI-07)
This commit is contained in:
parent
f49acf0105
commit
10af20a0ad
157 changed files with 29455 additions and 913 deletions
|
|
@ -84,7 +84,7 @@ Singleton + `EventEmitter` pattern (`getXService()` accessors).
|
|||
| Service / file | Purpose |
|
||||
|---|---|
|
||||
| `InputTelemetryService` | Global input telemetry and context observer: keystroke/click/scroll counters, mouse travel (Manhattan sum of per-axis deltas), active time, per-hour-per-app buckets flushed every 5 s; typed-text learning via UIA snapshot diffs. **Since 2026-10-26**, the UIA context observer runs by default whenever `typoAutoCorrectEnabled !== false` (or `suggestionEnabled === true`), while SQLite `input_activity` persistence only runs when `inputTelemetryEnabled === true` (opt-in). **Since 2026-10-27**, includes `TerminalKeystrokeTracker` (`@d3ro/core/input-intelligence`) + `isForegroundKoreanIme` (`utils/win32-foreground.ts`) to reconstruct the active terminal/Vim line (`reconcileWithScreen` for UIA screen buffers like Windows Terminal/pwsh/cmd, `reconcileWithXtermValue` for `xterm.js` `<textarea class="xterm-helper-textarea">` controls in Agent Switchboard/VS Code/Cursor where `.value` stays populated until Enter/Ctrl+C, and `getSnapshot` for canvas/remote terminals with `len=0` like Kitedesk/WezTerm/Alacritty), preserving trailing spaces stripped by grid terminals and tracking Vim modes (`insert`, `normal`, `command`). Programmatic keys (`_programmaticKeys`) no longer advance `_lastKeyAt`, and `Enter`-triggered chat submissions that clear or shrink the input field rebase focus text immediately so cleared/toolbar text does not trigger post-send suggestions. **Since 2026-10-29**, tracks `_typedInCurrentFocus` and `_lastTypedHwnd` and forces immediate foreground sampling on the first keystroke after a mouse click (`_rebaseFocusText`) or window switch, so typing `pc` (`ㅔㅊ`) immediately after clicking into a search box or new field sets `editedSinceFocus = true` on the very first snapshot. It extends the telemetry store with Flow Radar (hourly activity density + chars + edit stability; a potential-focus score, **not** a real session), Edit Friction (chars/backspaces; edits per 100 chars), App Quality (per-app suggestion total/accepted/accept rate/average latency), Privacy Receipt (actual row counts and retention), and Smart Exclusion evidence. Raw key events, key codes and content streams are never retained; separately, opt-in learned text may be retained in `typing_samples` and `personal_phrases` under the stated receipt policy. |
|
||||
| `TypoCorrectionService` | Built-in automatic Han/Eng layout conversion and typo auto-correction (`INPUT-19`). Runs on each `typed-context`: Araso-style layout conversions (`source: 'layout'`) fire after 25 ms on word boundaries (Space/punctuation, 40 ms quick snapshot) or 135 ms at token tail (150 ms quick snapshot, even when Chromium/Discord keeps `isComposing=true`, committing IME composition first only when the trailing character is Hangul/jamo), replacing 2-set Han/Eng layout errors, mixed English+Korean words (`githubdp`→`github에`, `PDFvkdlf`→`PDF파일`, `ㅖㅇㄹ파일`→`PDF파일`), lingering Shift on double consonants (`djEJs`→`어떤`), reverse Hangul-to-English (`ㅎ샤ㅗㅕㅠ에`→`github에`, `ㅑㄹ ㅑ 쇼ㅔㄷ ㅗ디ㅣㅐ 소두 ㅑㅅ 초뭏ㄷㄴ ㅑㅜ 10 ㄴㄷ채ㅜㅇㄴ`→`if I type hello then it changes in 10 seconds`), lookahead English clauses (`새 ㅠㄷ `→`to be `, `새 애 this `→`to do this `), and partially-converted tokens (`안녕gktpdy`→`안녕하세요`). **In terminals and Vim (`terminalMode: true`)**, layout-only conversion stays active while spelling/model checks are suppressed: converts CLI commands (`퍄ㅡ `→`vim `, `ㅜ퍄ㅡ `→`nvim `, `ㅎ샤 `→`git `, `ㅣㄴ `→`ls `, `채ㅜㄴㅅ `→`const `), Vim Insert-mode typing (stripping leading `i/a/o/I/A/O/s/S` mode-entry keys when present in raw keystroke buffers), and Vim Command-line ex-commands (`:ㅈㅂ`→`:wq`, `:ㅈㅂ!`→`:wq!`, `:ㅈ`→`:w`, `:ㅂ`→`:q`, `:ㅂ!`→`:q!`, `:ㅌ`→`:x`, `:ㅍ네`→`:vsp`, `:네`→`:sp`, `:ㅜㅐㅗ`→`:noh`, preserving `:`), while blocking all conversions in Vim Normal mode (`vimMode === 'normal'`). **Since 2026-10-28**, on Windows, `TextInsertService.replaceBeforeCaret` (`method === 'keyboard'`) executes an atomic single-syscall `sendWin32UnicodeReplacement` (`utils/win32-unicode-input.ts`) packing `(optional VK_SPACE) + N × VK_BACK + M × KEYEVENTF_UNICODE` into one `SendInput` call (`<0.5 ms`, zero JS sleep, zero clipboard access), and `TypoCorrectionService` delta-rebases replacement plans (`rebaseTypoReplacementPlan`) when the user appends characters while an async snapshot or LLM check is in flight. Both Stage 1 (deterministic layout/spelling) and Stage 2 (tail-anchored `trailingWordOnly: true` Gemma check) run seamlessly even when `SuggestionService` is enabled. |
|
||||
| `TypoCorrectionService` | Built-in automatic Han/Eng layout conversion and typo auto-correction (`INPUT-19`). Runs on each `typed-context`: Araso-style layout conversions (`source: 'layout'`) fire after 25 ms on word boundaries (Space/punctuation, 40 ms quick snapshot) or 135 ms at token tail (150 ms quick snapshot, even when Chromium/Discord keeps `isComposing=true`, committing IME composition first only when the trailing character is Hangul/jamo), replacing 2-set Han/Eng layout errors, mixed English+Korean words (`githubdp`→`github에`, `PDFvkdlf`→`PDF파일`, `ㅖㅇㄹ파일`→`PDF파일`), lingering Shift on double consonants (`djEJs`→`어떤`), reverse Hangul-to-English (`ㅎ샤ㅗㅕㅠ에`→`github에`, `ㅑㄹ ㅑ 쇼ㅔㄷ ㅗ디ㅣㅐ 소두 ㅑㅅ 초뭏ㄷㄴ ㅑㅜ 10 ㄴㄷ채ㅜㅇㄴ`→`if I type hello then it changes in 10 seconds`), lookahead English clauses (`새 ㅠㄷ `→`to be `, `새 애 this `→`to do this `), and partially-converted tokens (`안녕gktpdy`→`안녕하세요`). **In terminals and Vim (`terminalMode: true`)**, layout-only conversion stays active while spelling/model checks are suppressed: converts CLI commands (`퍄ㅡ `→`vim `, `ㅜ퍄ㅡ `→`nvim `, `ㅎ샤 `→`git `, `ㅣㄴ `→`ls `, `채ㅜㄴㅅ `→`const `), Vim Insert-mode typing (stripping leading `i/a/o/I/A/O/s/S` mode-entry keys when present in raw keystroke buffers), and Vim Command-line ex-commands (`:ㅈㅂ`→`:wq`, `:ㅈㅂ!`→`:wq!`, `:ㅈ`→`:w`, `:ㅂ`→`:q`, `:ㅂ!`→`:q!`, `:ㅌ`→`:x`, `:ㅍ네`→`:vsp`, `:네`→`:sp`, `:ㅜㅐㅗ`→`:noh`, preserving `:`), while blocking all conversions in Vim Normal mode (`vimMode === 'normal'`). **Since 2026-10-28**, on Windows, `TextInsertService.replaceBeforeCaret` (`method === 'keyboard'`) executes an atomic single-syscall `sendWin32UnicodeReplacement` (`utils/win32-unicode-input.ts`) packing `(optional VK_SPACE) + N × VK_BACK + M × KEYEVENTF_UNICODE` into one `SendInput` call (zero JS sleep, zero clipboard access), and `TypoCorrectionService` delta-rebases replacement plans (`rebaseTypoReplacementPlan`) when the user appends characters while an async snapshot or LLM check is in flight. Both Stage 1 (deterministic layout/spelling) and Stage 2 (tail-anchored `trailingWordOnly: true` Gemma check) run seamlessly even when `SuggestionService` is enabled. |
|
||||
| `UiaContextService` | Client for the sidecar UIA bridge (`GET /uia/focus`): focused text, caret rect/offset, `IsPassword`, IME composition state and `hasSelection`. The sidecar derives `hasSelection` only by comparing the TextPattern selection range Start/End endpoints: it does not call `GetText` on that range or add a separate selected-text payload (the existing focused-field text can still include a selection). Fail-closed on password/unavailable; transient failures back off for 1.5/3/6/12/30 s and permanent failures for 5 min. |
|
||||
| `SuggestionService` | Next-sentence ghost text: 600 ms debounce, then `buildSuggestionPrompt` → `LocalLLMService.streamGenerate` with a caller-owned `AbortSignal` → candidate ranking and accept (via `TextInsertService`). Presentation-active includes candidates plus `generating`, `warmingUp` and `partialText`; clear/dismiss aborts the active request, invalidates its generation token, clears TTL state, then emits `cleared`/hide so a spinner-only popup closes. A successful final candidate payload resets `generating=false` and `partialText=null`. App DNA extends phrase ranking with `appName` context and a 1.75× same-app bonus; Memory Decay uses a 30-day half-life plus frequency. Why This Suggestion exposes only `local-model`/`local-memory` provenance and continuation/related/phrase/appPhrase **counts** to the overlay, never raw evidence text. Instant Recall uses local-memory provenance without another model-budget spend only when the model is unavailable, raises a non-cancellation error, truly times out, or returns empty; it never publishes after dismiss, new typing, token/context mismatch or staleness. Its runaway limits are a 5 s minimum interval, 6 requests/min default (hard-config maximum 12), 3 candidates, 64 output tokens, 12-character regeneration growth and an 8 s timeout; app startup does not warm the model and suggestion requests use `keep_alive: 2m`. |
|
||||
| `global-input-hook.ts` | Ref-counted owner of the single process-wide `uiohook` hook so `KeyBindingService` and telemetry can both attach without one stopping the other. |
|
||||
|
|
@ -213,6 +213,13 @@ DB schema (`src/main/db/schema.ts`, drizzle SQLite): `history`, `dictionary`, `s
|
|||
|
||||
## 6. Desktop status summary
|
||||
|
||||
- **2026-10-30 high-load typing protection (INPUT-04/19, GAP-INPUT-11):** UIA snapshots are deduplicated and discarded on input/focus/native activity changes before updating text baselines. `TypoCorrectionService` passes a replacement-time guard through `bootstrap` to `TextInsertService`; the guard checks monotonic input revision, native `GetLastInputInfo` token, and foreground after queue waits. Synthetic-key suppression starts only at actual injection. Automatic Windows corrections require a single native input batch; partial/unknown `SendInput` results fail without replaying deletion. A successful batch enqueue does not prove that the target app/IME processed the intended text. The native token is session-scoped and only compared for equality (same-tick input remains limited by Windows timestamp resolution).
|
||||
- **Local inference worker headroom:** `utils/inference-cpu-budget.ts` and `sidecar/cpu_budget.py` use `max(1, min(4, available CPUs - 2))` for loopback Ollama generate/stream/chat and Whisper primary/auxiliary/CPU fallback. Remote Ollama defaults are unchanged. This bounds inference workers, not model-load IO, GPU usage, total CPU percentage, or concurrent engines; CPU-only inference may slow down. Existing per-letter terminal IME probing can still synchronously wait up to 30 ms and needs separate profiling before changing its correctness-sensitive behavior.
|
||||
- **Verification for this change:** 14 focused desktop test files / 160 tests and Python CPU-budget + device-fallback tests / 23 tests pass; touched TypeScript ESLint and desktop production bundle build pass. Compiler-API comparison against tracked HEAD sources gives renderer strict diagnostics 0→0 and main strict diagnostics 16→14, with no introduced diagnostics. Remaining errors are in support handlers, KeyBindingService, PersonalGraphService and core typo-correction; full strict typecheck remains failing. Windows GetLastInputInfo read-only native probe succeeds. No interactive high-load typing run or packaged-sidecar rebuild is claimed.
|
||||
|
||||
- **2026-10-30 deterministic source A/B:** `node scripts/verification/typing-race-proof.mjs` overlays only `InputTelemetryService`, `TypoCorrectionService` and `TextInsertService` from HEAD `f49acf0` without modifying the checkout; dependencies, test doubles and tests remain current in both runs. This is not a complete historical-release A/B. HEAD passed **1/17** (16 failures); current sources passed **17/17**. Three delayed-queue cases independently isolate timestamp, monotonic revision and native-tick guards: the old services corrupted the next word to `오늘 잘 됬어요 안녕됐어요 ` while the fix preserved `오늘 잘 됬어요 안녕하세요 `. The other 13 failures test snapshot invalidation/metadata/coalescing contracts, not 13 additional text-loss reproductions. The unchanged-input control still corrects in both runs. Source hashes remained unchanged during the run. Evidence: [`report.json`](../../test-results/typing-load/race-proof-2026-10-30T00-50-32-284Z/report.json). This proves the service-level race with an in-memory text sink and delayed UIA mocks; it does **not** reproduce the exact user symptom in Windows IME or under actual CPU/model load.
|
||||
- **Interactive verification infrastructure (prepared, not yet run):** [`scripts/verification/README.md`](../../scripts/verification/README.md) defines the matrix, evidence requirements and Windows contracts. `typing-fixture` builds a dedicated WinForms text sink with scan-code injection, mandatory Hangul/WM_IME_COMPOSITION calibration, 60/30/15 ms pacing, optional bounded CPU phases, focus cancellation and JSON result capture. `typing-runtime/build.ps1` builds `test-results/typing-load/runtime/D3ROTypingRuntime.exe`; this interactive launcher refuses to start while the installed `D3RO Voice` process is running. The runtime loads the actual product bundle with a separate local-only profile (automatic/model correction, suggestions, learning and telemetry off initially), blocks protocol/OS startup registration, and records product package version plus bundle SHA-256 in `runtime.json`. Syntax checking and launcher compilation passed; the runtime has not started. Auto-correction on/off, a positive correction control, actual model cold/warm overlap, CPU observations, Chromium/terminal surfaces and physical hardware typing remain GAP-INPUT-11 gates.
|
||||
|
||||
- Core dictation/LLM/history pipeline: **implemented + tested**. The vitest case count in `apps/desktop` is **1465** after input intelligence and the Local Flow Intelligence extension; playwright e2e is separate. Read the pass numbers together with the `better-sqlite3` ABI the tree is built for (`11` GAP-INFRA-06) — they are not comparable across configurations:
|
||||
- **Host Node ABI** (2026-09-21, before the LLM fix): 1311 / 1314 passing. The three failures are environment-dependent rather than regressions — two need a local sidecar venv or embedding server, one pins an error message that has since changed (`11` GAP-QA-02). **This configuration has not been re-measured since the LLM fix.**
|
||||
- **Electron ABI** (2026-09-21, after input intelligence): `366 failed | 1042 passed (1408)`, against a clean-tree baseline of `366 failed | 994 passed (1360)` in the same configuration — identical failure count, +48 passed, **zero new failures**. 365 of those 366 are `tests/red/*.usecase.test.ts` files dying at DB creation because of the ABI mismatch, not assertions.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue