feat(desktop): screen text capture (OCR) overlay with selectable text

A rebindable screen-ocr action (default Ctrl+Alt+O) freezes the monitor under the cursor and
covers it with a focusable overlay. Text is selected by flow drag, rectangle drag, double/triple
click, Shift+click or Ctrl+A, then copied, saved as PNG or searchable PDF, translated,
summarized, read aloud, opened as a link or searched.

- Selection, text assembly and the PDF document are pure functions in @d3ro/core/screen-ocr,
  shared by the main process and the overlay.
- Recognition uses Windows.Media.Ocr through powershell.exe -EncodedCommand with 2x upscaling;
  the capture reaches disk only as a temp PNG that is removed right after recognition.
- Capture uses GDI BitBlt for the target monitor (39 ms) because desktopCapturer captured all
  three monitors and took 2.9 s when DXGI duplication timed out.
- The accelerator is reserved only where the feature is available, so macOS does not swallow
  the key for a feature it cannot run (fails closed with a message).

Map: CAP-17 [~], GAP-OCR-01 (live hotkey journey not yet driven in the running app),
GAP-OCR-02 (macOS engine, mixed-language accuracy).
This commit is contained in:
Yun Chan 2026-11-20 15:16:37 +09:00
parent 6791e29f27
commit 03cd8bf890
41 changed files with 4561 additions and 17 deletions

View file

@ -13,6 +13,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Plugin system for custom pipelines
## [1.18.0] - 2026-11-20
> Screen text capture (OCR) for Windows, plus the Electron 43 toolchain upgrade and the 2026-11-17 backlog batch.
### Added
- **Screen text capture (OCR).** Press `Ctrl+Alt+O` (rebindable under Settings → key bindings) to freeze the monitor under the cursor and select the text on it:
- drag from a word to select along the text flow, drag from empty space (or switch to Area mode) to select with a rectangle, double-click a word, triple-click a line, `Shift`+click to extend, `Ctrl+A` for everything;
- copy the text (keeping line breaks or as one line), copy or save the selected area as an image, and save it as a **searchable PDF** whose text can be found and copied;
- translate or summarize the selection with the configured AI model, have it read aloud, open a link found in it, or search the web for it;
- switch the recognition language among the Windows OCR languages installed on the PC.
- Mobile: share a history entry as a Markdown file or as its original audio.
### Changed
- Desktop runtime upgraded from Electron 33 to Electron 43 (with Vite 8, Vitest 5 and Node 24 for the build); clipboard reads and writes are awaited end to end.
- Screen capture for OCR reads only the target monitor through GDI (about 40 ms) instead of capturing every monitor.
### Fixed
- Scheduled push delivery no longer fails with HTTP 401: the drain sends the `apikey` header that current Supabase service keys require.
### Notes
- Screen text capture uses the OCR engine built into Windows and is not available on macOS yet. With only the Korean OCR language installed, small English UI text can be misread; installing the English language pack in Windows Settings adds it to the language list.
- The capture stays on the PC. It is written to disk only as a temporary file for recognition and deleted right after; only the text you select is sent to the AI model when you choose translate or summarize.
## [1.17.1] - 2026-11-14
> Supersedes unpublished `1.17.0` with the full-length ASAR Pickle header smoke verification fix and ships the full `1.17.0` multi-surface release.