feat(desktop): screen text capture (OCR) overlay with selectable text
A rebindable screen-ocr action (default Ctrl+Alt+O) freezes the monitor under the cursor and covers it with a focusable overlay. Text is selected by flow drag, rectangle drag, double/triple click, Shift+click or Ctrl+A, then copied, saved as PNG or searchable PDF, translated, summarized, read aloud, opened as a link or searched. - Selection, text assembly and the PDF document are pure functions in @d3ro/core/screen-ocr, shared by the main process and the overlay. - Recognition uses Windows.Media.Ocr through powershell.exe -EncodedCommand with 2x upscaling; the capture reaches disk only as a temp PNG that is removed right after recognition. - Capture uses GDI BitBlt for the target monitor (39 ms) because desktopCapturer captured all three monitors and took 2.9 s when DXGI duplication timed out. - The accelerator is reserved only where the feature is available, so macOS does not swallow the key for a feature it cannot run (fails closed with a message). Map: CAP-17 [~], GAP-OCR-01 (live hotkey journey not yet driven in the running app), GAP-OCR-02 (macOS engine, mixed-language accuracy).
This commit is contained in:
parent
6791e29f27
commit
03cd8bf890
41 changed files with 4561 additions and 17 deletions
|
|
@ -30,6 +30,7 @@ Status quick-reference: `[x]` done+verified · `[~]` partial/unverified · `[ ]`
|
|||
| CAP-14 | Recording persistence / crash recovery | [x] | [ ] | [x] | [-] | Desktop WAV persist; mobile durable queue + process-kill WAV recovery |
|
||||
| CAP-15 | Android foreground recording service | [-] | [-] | [x] | [-] | Mobile API 34 FGS + persistent notification (SSOT R-005 GREEN) |
|
||||
| CAP-16 | Rebindable global key bindings (keyboard + mouse) | [x] | [-] | [-] | [-] | Contract SSOT `packages/core/src/keybinding.ts`: `KEY_CATALOG` (10 groups), `KEYBINDING_ACTIONS` (6 actions), `validateBinding`, `detectBindingConflicts`. Multiple bindings per action persist as one `AppConfig.keyBindings` map (`packages/core/src/types.ts`), replacing the four singular `*Shortcut` fields; `ConfigService` migrates legacy values once. `KeyBindingService` hooks keyboard **and** mouse via uiohook — MB1 is not bindable, MB2/MB3 need a modifier, MB4/MB5 are free. **Verified 2026-09-21 on Windows by a manual run** and **2026-10-30 unit-tested** in `KeyBindingService.test.ts` (11 tests, GAP-KEY-01 `[x]`), with dedicated `command` action toggle/hold + forced voice-command matching in `VoiceModeService.ts` / `voice-intent.ts` (`voice-intent.test.ts` 19 tests, GAP-KEY-03 `[x]`). **Still open:** macOS/Linux mouse behavior (`11` GAP-KEY-02). W/M `[-]`: no OS-level global binding surface exists there. B `[-]`: device-local setting. |
|
||||
| CAP-17 | Screen text capture (OCR overlay) | [~] | [-] | [-] | [-] | Desktop/Windows. The rebindable `screen-ocr` action (default `Ctrl+Alt+O`, CAP-16) freezes the monitor under the cursor and covers it with the `screen-ocr` popup; text is selected by flow drag (start on a word), rectangle drag (start on empty space or in Area mode), double-click word, triple-click line, Shift+click extend, `Ctrl+A`. Actions: copy (multi-line / one line), copy image, save PNG, **searchable PDF** (image above a real text layer), translate and summarize through `LlmGateway`, read aloud (`TTSPlaybackService`), open link / web search, recognition language switch (`AppConfig.screenOcrLanguage`). Pure selection/text/PDF logic `packages/core/src/screen-ocr.ts`; service `apps/desktop/src/main/services/ScreenOcrService.ts`; engine `services/screen-ocr/windows-ocr-engine.ts` (Windows.Media.Ocr via `powershell.exe -EncodedCommand`, 2x upscale); capture `utils/win32-screen-capture.ts` (GDI BitBlt, 39 ms vs 2.9 s for `desktopCapturer` on a 3-monitor PC). The capture touches disk only as one temp PNG that is deleted right after recognition. **Verified 2026-11-20:** core 34 + desktop 41 tests, real-engine probe (Korean+English screen, 0.64 s), headless overlay run (flow/rect selection, copy, translate, PDF request, release on close) and a printed PDF whose text layer is searchable. **Not yet verified:** the live hotkey journey in the running app (focus, mixed-DPI multi-monitor) — `GAP-OCR-01`. macOS has no engine and fails closed with a message — `GAP-OCR-02`. |
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue