feat(desktop): screen text capture (OCR) overlay with selectable text
A rebindable screen-ocr action (default Ctrl+Alt+O) freezes the monitor under the cursor and covers it with a focusable overlay. Text is selected by flow drag, rectangle drag, double/triple click, Shift+click or Ctrl+A, then copied, saved as PNG or searchable PDF, translated, summarized, read aloud, opened as a link or searched. - Selection, text assembly and the PDF document are pure functions in @d3ro/core/screen-ocr, shared by the main process and the overlay. - Recognition uses Windows.Media.Ocr through powershell.exe -EncodedCommand with 2x upscaling; the capture reaches disk only as a temp PNG that is removed right after recognition. - Capture uses GDI BitBlt for the target monitor (39 ms) because desktopCapturer captured all three monitors and took 2.9 s when DXGI duplication timed out. - The accelerator is reserved only where the feature is available, so macOS does not swallow the key for a feature it cannot run (fails closed with a message). Map: CAP-17 [~], GAP-OCR-01 (live hotkey journey not yet driven in the running app), GAP-OCR-02 (macOS engine, mixed-language accuracy).
This commit is contained in:
parent
6791e29f27
commit
03cd8bf890
41 changed files with 4561 additions and 17 deletions
|
|
@ -30,7 +30,7 @@ Singleton + `EventEmitter` pattern (`getXService()` accessors).
|
|||
| `VoiceModeService` | Orchestrator: 9-state `RecognitionState` + 4-state `AudioState`, dual-condition flush, action queue, and secure audio buffer zeroing (`secureWipeAudioChunks` / `secureZeroBuffer`) on cancel/cleanup (2026-11-13 `W5-01`). Events: session-started/completed/cancelled, transcription-update, audio-level, recognition/audio-state-changed, premium-llm-fallback, error |
|
||||
| `AudioCaptureService` | Mic PCM16 16kHz mono (bundled SoX on Windows, node-record-lpcm16 elsewhere) with `secureWipeAudioChunks` on stop/dispose. Spawns hidden (`windowsHide`); a missing SoX fails with the exact fix command |
|
||||
| `LocalSTTService` | faster-whisper Python sidecar manager (state machine, dual-flush, model download/cancel, background warm-up, live partial transcription, `sidecarBaseUrl` getter for `POST /diarize`, and **2026-11-13** `releasePrimaryModel()` / 15-min `IDLE_MODEL_UNLOAD_MS` timer via `POST /unload` `{ slot: "primary" }` so switching to cloud STT or going idle frees the 1.6–3.1 GB Whisper model while keeping the sidecar HTTP server alive for `/uia/focus`). Connects over IPv4 loopback (`getSidecarBaseUrl`) and fails fast with an actionable message when the bundled engine or virtualenv is missing |
|
||||
| `KeyBindingService` | uiohook-napi global hooking for **keyboard and mouse**, driven by the `@d3ro/core/keybinding` contract: 6 rebindable actions (dictation, hands-free, command, caption, history-popup, command-popup), several bindings per action, structural reserved-combo checks. Events: `triggered` (in-process payload carries `actionId`, `type` (`pressed`/`released`), `isDoublePress`, `holdMode`, `timestamp`; the renderer-facing `keybinding:triggered` event is the narrower `KeyBindingTriggeredEvent`, `keybinding.ts:1092`), `changed`, `error`. `globalShortcut` is used only to mute the macOS system beep, and only for accelerators it registered itself. Mouse events cannot be suppressed by uiohook, so a bound button also performs its native action |
|
||||
| `KeyBindingService` | uiohook-napi global hooking for **keyboard and mouse**, driven by the `@d3ro/core/keybinding` contract: rebindable actions (dictation, hands-free, command, caption, history-popup, command-popup, screen-ocr, and the suggestion-* set), several bindings per action, structural reserved-combo checks. Events: `triggered` (in-process payload carries `actionId`, `type` (`pressed`/`released`), `isDoublePress`, `holdMode`, `timestamp`; the renderer-facing `keybinding:triggered` event is the narrower `KeyBindingTriggeredEvent`, `keybinding.ts:1092`), `changed`, `error`. `globalShortcut` is used only to mute the macOS system beep, and only for accelerators it registered itself. Mouse events cannot be suppressed by uiohook, so a bound button also performs its native action |
|
||||
| `TextInsertService` | Clipboard save→set→Ctrl+V→restore (`NutKeyboardPort`: uses native `koffi` Win32 `SendInput` via `utils/win32-unicode-input.ts` on Windows without loading `@nut-tree-fork/nut-js`; `nut-js` used on macOS/Linux) |
|
||||
| `SoundEffectService` | Preloaded WAV feedback (start/stop/error/cancel/chime) |
|
||||
|
||||
|
|
@ -62,6 +62,7 @@ Singleton + `EventEmitter` pattern (`getXService()` accessors).
|
|||
| `VoiceCommandService` | Keyword → command rule matching |
|
||||
| `ChainService` | Multi-step LLM pipelines (LLMChain). Each step resolves its instruction through `llm-prompts.ts` (`ChainService.ts:196`); before that, chain steps sent placeholders through unsubstituted |
|
||||
| `ScreenContextService` | Active-window + selected-text context (on Windows uses `sendWin32CopyChord()` via `koffi` `SendInput` without loading `nut-js`) |
|
||||
| `ScreenOcrService` | Screen text capture (CAP-17): owns one capture session — capture → overlay → async recognition → overlay requests (copy text/image, save PNG/searchable PDF, translate, summarize, read aloud, open link/search, language switch) → close. All OS access is behind ports (`services/screen-ocr/electron-adapters.ts`); engine port `OcrEnginePort` has one adapter, `windows-ocr-engine.ts` (Windows.Media.Ocr). Assembled in `src/main/screen-ocr-wiring.ts` (key binding, overlay IPC with sender check, `Escape` capture while open except during a save dialog). Events: `opened`, `closed`. |
|
||||
|
||||
### Phase 10+ features
|
||||
| Service | Purpose |
|
||||
|
|
@ -168,7 +169,7 @@ Preload exposes **`window.electronAPI`** with 37 namespaces: `platform, audio, c
|
|||
|
||||
## 4. Windows & popups
|
||||
|
||||
`windows/WindowManager.ts` manages 7 windows: main (borderless, custom TitleBar; macOS `hiddenInset`) created at boot, and 6 vanilla popups (`recording-tip`, `result-popup`, `history-popup`, `command-popup`, `caption-overlay`, `suggestion-overlay`) created **lazily on first demand** (`peekHistoryPopupWindow` / `peekCommandPopupWindow` inspect existing windows without instantiating hidden Chromium renderers at startup). Injects popup theme CSS + i18n strings via `@d3ro/i18n/main`; 2-phase resize. `windows/TrayManager.ts` — tray icon + menu + double-click show.
|
||||
`windows/WindowManager.ts` manages 7 windows: main (borderless, custom TitleBar; macOS `hiddenInset`) created at boot, and 6 vanilla popups (`recording-tip`, `result-popup`, `history-popup`, `command-popup`, `caption-overlay`, `suggestion-overlay`); an 8th window, the `screen-ocr` overlay, lives in `windows/screen-ocr-overlay.ts` and reuses the same `popup-window-factory` (spec flags `focusable: true`, `transparent: false`) created **lazily on first demand** (`peekHistoryPopupWindow` / `peekCommandPopupWindow` inspect existing windows without instantiating hidden Chromium renderers at startup). Injects popup theme CSS + i18n strings via `@d3ro/i18n/main`; 2-phase resize. `windows/TrayManager.ts` — tray icon + menu + double-click show.
|
||||
|
||||
**Popup invariants** (each shipped broken once — do not regress):
|
||||
|
||||
|
|
@ -186,6 +187,7 @@ Vanilla popups (`src/renderer/popups/`):
|
|||
| `command-popup` | Command selection. Opened by the `command-popup` action (default `Ctrl+Shift+C`, rebindable) |
|
||||
| `caption-overlay` | Live caption overlay (font/opacity/maxLines) |
|
||||
| `suggestion-overlay` | Next-sentence ghost text: caret-anchored (`anchorFloatingPanel`), non-focusable, click-through unless `suggestionOverlayInteractive`; accept/next/dismiss come from global key bindings (the window never owns focus). Its narrow layout shows a one-line local provenance/count summary only, never raw memory evidence. Presentation includes candidate, generating, warm-up and partial-text states; its X hides the panel before main-process dismissal. |
|
||||
| `screen-ocr` | Screen text capture overlay (CAP-17): the only **focusable, opaque** popup; covers one monitor with the frozen capture at `screen-saver` level. 2-phase show — main sends `screenOcr:session`, the popup paints the image and answers `screenOcr:ready`, then the window is shown. Imports `@d3ro/core/screen-ocr` for selection math (same code as main). Channels `POPUP_SCREEN_OCR` (`session`, `recognized`, `request`/`response`, `ready`, `dismiss`). |
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue