feat(desktop): screen text capture (OCR) overlay with selectable text
A rebindable screen-ocr action (default Ctrl+Alt+O) freezes the monitor under the cursor and covers it with a focusable overlay. Text is selected by flow drag, rectangle drag, double/triple click, Shift+click or Ctrl+A, then copied, saved as PNG or searchable PDF, translated, summarized, read aloud, opened as a link or searched. - Selection, text assembly and the PDF document are pure functions in @d3ro/core/screen-ocr, shared by the main process and the overlay. - Recognition uses Windows.Media.Ocr through powershell.exe -EncodedCommand with 2x upscaling; the capture reaches disk only as a temp PNG that is removed right after recognition. - Capture uses GDI BitBlt for the target monitor (39 ms) because desktopCapturer captured all three monitors and took 2.9 s when DXGI duplication timed out. - The accelerator is reserved only where the feature is available, so macOS does not swallow the key for a feature it cannot run (fails closed with a message). Map: CAP-17 [~], GAP-OCR-01 (live hotkey journey not yet driven in the running app), GAP-OCR-02 (macOS engine, mixed-language accuracy).
This commit is contained in:
parent
6791e29f27
commit
03cd8bf890
41 changed files with 4561 additions and 17 deletions
|
|
@ -30,7 +30,7 @@ Singleton + `EventEmitter` pattern (`getXService()` accessors).
|
|||
| `VoiceModeService` | Orchestrator: 9-state `RecognitionState` + 4-state `AudioState`, dual-condition flush, action queue, and secure audio buffer zeroing (`secureWipeAudioChunks` / `secureZeroBuffer`) on cancel/cleanup (2026-11-13 `W5-01`). Events: session-started/completed/cancelled, transcription-update, audio-level, recognition/audio-state-changed, premium-llm-fallback, error |
|
||||
| `AudioCaptureService` | Mic PCM16 16kHz mono (bundled SoX on Windows, node-record-lpcm16 elsewhere) with `secureWipeAudioChunks` on stop/dispose. Spawns hidden (`windowsHide`); a missing SoX fails with the exact fix command |
|
||||
| `LocalSTTService` | faster-whisper Python sidecar manager (state machine, dual-flush, model download/cancel, background warm-up, live partial transcription, `sidecarBaseUrl` getter for `POST /diarize`, and **2026-11-13** `releasePrimaryModel()` / 15-min `IDLE_MODEL_UNLOAD_MS` timer via `POST /unload` `{ slot: "primary" }` so switching to cloud STT or going idle frees the 1.6–3.1 GB Whisper model while keeping the sidecar HTTP server alive for `/uia/focus`). Connects over IPv4 loopback (`getSidecarBaseUrl`) and fails fast with an actionable message when the bundled engine or virtualenv is missing |
|
||||
| `KeyBindingService` | uiohook-napi global hooking for **keyboard and mouse**, driven by the `@d3ro/core/keybinding` contract: 6 rebindable actions (dictation, hands-free, command, caption, history-popup, command-popup), several bindings per action, structural reserved-combo checks. Events: `triggered` (in-process payload carries `actionId`, `type` (`pressed`/`released`), `isDoublePress`, `holdMode`, `timestamp`; the renderer-facing `keybinding:triggered` event is the narrower `KeyBindingTriggeredEvent`, `keybinding.ts:1092`), `changed`, `error`. `globalShortcut` is used only to mute the macOS system beep, and only for accelerators it registered itself. Mouse events cannot be suppressed by uiohook, so a bound button also performs its native action |
|
||||
| `KeyBindingService` | uiohook-napi global hooking for **keyboard and mouse**, driven by the `@d3ro/core/keybinding` contract: rebindable actions (dictation, hands-free, command, caption, history-popup, command-popup, screen-ocr, and the suggestion-* set), several bindings per action, structural reserved-combo checks. Events: `triggered` (in-process payload carries `actionId`, `type` (`pressed`/`released`), `isDoublePress`, `holdMode`, `timestamp`; the renderer-facing `keybinding:triggered` event is the narrower `KeyBindingTriggeredEvent`, `keybinding.ts:1092`), `changed`, `error`. `globalShortcut` is used only to mute the macOS system beep, and only for accelerators it registered itself. Mouse events cannot be suppressed by uiohook, so a bound button also performs its native action |
|
||||
| `TextInsertService` | Clipboard save→set→Ctrl+V→restore (`NutKeyboardPort`: uses native `koffi` Win32 `SendInput` via `utils/win32-unicode-input.ts` on Windows without loading `@nut-tree-fork/nut-js`; `nut-js` used on macOS/Linux) |
|
||||
| `SoundEffectService` | Preloaded WAV feedback (start/stop/error/cancel/chime) |
|
||||
|
||||
|
|
@ -62,6 +62,7 @@ Singleton + `EventEmitter` pattern (`getXService()` accessors).
|
|||
| `VoiceCommandService` | Keyword → command rule matching |
|
||||
| `ChainService` | Multi-step LLM pipelines (LLMChain). Each step resolves its instruction through `llm-prompts.ts` (`ChainService.ts:196`); before that, chain steps sent placeholders through unsubstituted |
|
||||
| `ScreenContextService` | Active-window + selected-text context (on Windows uses `sendWin32CopyChord()` via `koffi` `SendInput` without loading `nut-js`) |
|
||||
| `ScreenOcrService` | Screen text capture (CAP-17): owns one capture session — capture → overlay → async recognition → overlay requests (copy text/image, save PNG/searchable PDF, translate, summarize, read aloud, open link/search, language switch) → close. All OS access is behind ports (`services/screen-ocr/electron-adapters.ts`); engine port `OcrEnginePort` has one adapter, `windows-ocr-engine.ts` (Windows.Media.Ocr). Assembled in `src/main/screen-ocr-wiring.ts` (key binding, overlay IPC with sender check, `Escape` capture while open except during a save dialog). Events: `opened`, `closed`. |
|
||||
|
||||
### Phase 10+ features
|
||||
| Service | Purpose |
|
||||
|
|
@ -168,7 +169,7 @@ Preload exposes **`window.electronAPI`** with 37 namespaces: `platform, audio, c
|
|||
|
||||
## 4. Windows & popups
|
||||
|
||||
`windows/WindowManager.ts` manages 7 windows: main (borderless, custom TitleBar; macOS `hiddenInset`) created at boot, and 6 vanilla popups (`recording-tip`, `result-popup`, `history-popup`, `command-popup`, `caption-overlay`, `suggestion-overlay`) created **lazily on first demand** (`peekHistoryPopupWindow` / `peekCommandPopupWindow` inspect existing windows without instantiating hidden Chromium renderers at startup). Injects popup theme CSS + i18n strings via `@d3ro/i18n/main`; 2-phase resize. `windows/TrayManager.ts` — tray icon + menu + double-click show.
|
||||
`windows/WindowManager.ts` manages 7 windows: main (borderless, custom TitleBar; macOS `hiddenInset`) created at boot, and 6 vanilla popups (`recording-tip`, `result-popup`, `history-popup`, `command-popup`, `caption-overlay`, `suggestion-overlay`); an 8th window, the `screen-ocr` overlay, lives in `windows/screen-ocr-overlay.ts` and reuses the same `popup-window-factory` (spec flags `focusable: true`, `transparent: false`) created **lazily on first demand** (`peekHistoryPopupWindow` / `peekCommandPopupWindow` inspect existing windows without instantiating hidden Chromium renderers at startup). Injects popup theme CSS + i18n strings via `@d3ro/i18n/main`; 2-phase resize. `windows/TrayManager.ts` — tray icon + menu + double-click show.
|
||||
|
||||
**Popup invariants** (each shipped broken once — do not regress):
|
||||
|
||||
|
|
@ -186,6 +187,7 @@ Vanilla popups (`src/renderer/popups/`):
|
|||
| `command-popup` | Command selection. Opened by the `command-popup` action (default `Ctrl+Shift+C`, rebindable) |
|
||||
| `caption-overlay` | Live caption overlay (font/opacity/maxLines) |
|
||||
| `suggestion-overlay` | Next-sentence ghost text: caret-anchored (`anchorFloatingPanel`), non-focusable, click-through unless `suggestionOverlayInteractive`; accept/next/dismiss come from global key bindings (the window never owns focus). Its narrow layout shows a one-line local provenance/count summary only, never raw memory evidence. Presentation includes candidate, generating, warm-up and partial-text states; its X hides the panel before main-process dismissal. |
|
||||
| `screen-ocr` | Screen text capture overlay (CAP-17): the only **focusable, opaque** popup; covers one monitor with the frozen capture at `screen-saver` level. 2-phase show — main sends `screenOcr:session`, the popup paints the image and answers `screenOcr:ready`, then the window is shown. Imports `@d3ro/core/screen-ocr` for selection math (same code as main). Channels `POPUP_SCREEN_OCR` (`session`, `recognized`, `request`/`response`, `ready`, `dismiss`). |
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -30,6 +30,7 @@ Status quick-reference: `[x]` done+verified · `[~]` partial/unverified · `[ ]`
|
|||
| CAP-14 | Recording persistence / crash recovery | [x] | [ ] | [x] | [-] | Desktop WAV persist; mobile durable queue + process-kill WAV recovery |
|
||||
| CAP-15 | Android foreground recording service | [-] | [-] | [x] | [-] | Mobile API 34 FGS + persistent notification (SSOT R-005 GREEN) |
|
||||
| CAP-16 | Rebindable global key bindings (keyboard + mouse) | [x] | [-] | [-] | [-] | Contract SSOT `packages/core/src/keybinding.ts`: `KEY_CATALOG` (10 groups), `KEYBINDING_ACTIONS` (6 actions), `validateBinding`, `detectBindingConflicts`. Multiple bindings per action persist as one `AppConfig.keyBindings` map (`packages/core/src/types.ts`), replacing the four singular `*Shortcut` fields; `ConfigService` migrates legacy values once. `KeyBindingService` hooks keyboard **and** mouse via uiohook — MB1 is not bindable, MB2/MB3 need a modifier, MB4/MB5 are free. **Verified 2026-09-21 on Windows by a manual run** and **2026-10-30 unit-tested** in `KeyBindingService.test.ts` (11 tests, GAP-KEY-01 `[x]`), with dedicated `command` action toggle/hold + forced voice-command matching in `VoiceModeService.ts` / `voice-intent.ts` (`voice-intent.test.ts` 19 tests, GAP-KEY-03 `[x]`). **Still open:** macOS/Linux mouse behavior (`11` GAP-KEY-02). W/M `[-]`: no OS-level global binding surface exists there. B `[-]`: device-local setting. |
|
||||
| CAP-17 | Screen text capture (OCR overlay) | [~] | [-] | [-] | [-] | Desktop/Windows. The rebindable `screen-ocr` action (default `Ctrl+Alt+O`, CAP-16) freezes the monitor under the cursor and covers it with the `screen-ocr` popup; text is selected by flow drag (start on a word), rectangle drag (start on empty space or in Area mode), double-click word, triple-click line, Shift+click extend, `Ctrl+A`. Actions: copy (multi-line / one line), copy image, save PNG, **searchable PDF** (image above a real text layer), translate and summarize through `LlmGateway`, read aloud (`TTSPlaybackService`), open link / web search, recognition language switch (`AppConfig.screenOcrLanguage`). Pure selection/text/PDF logic `packages/core/src/screen-ocr.ts`; service `apps/desktop/src/main/services/ScreenOcrService.ts`; engine `services/screen-ocr/windows-ocr-engine.ts` (Windows.Media.Ocr via `powershell.exe -EncodedCommand`, 2x upscale); capture `utils/win32-screen-capture.ts` (GDI BitBlt, 39 ms vs 2.9 s for `desktopCapturer` on a 3-monitor PC). The capture touches disk only as one temp PNG that is deleted right after recognition. **Verified 2026-11-20:** core 34 + desktop 41 tests, real-engine probe (Korean+English screen, 0.64 s), headless overlay run (flow/rect selection, copy, translate, PDF request, release on close) and a printed PDF whose text layer is searchable. **Not yet verified:** the live hotkey journey in the running app (focus, mixed-DPI multi-monitor) — `GAP-OCR-01`. macOS has no engine and fails closed with a message — `GAP-OCR-02`. |
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -21,6 +21,8 @@ Legend: `[ ]` open · `[~]` in progress · `[!]` blocked externally · `[x]` res
|
|||
|
||||
| ID | Area | Gap | Evidence | Suggested next step |
|
||||
|---|---|---|---|---|
|
||||
| GAP-OCR-01 | Desktop | Screen text capture (CAP-17) has not been driven end-to-end in the running app: global hotkey → overlay takes focus → drag select → copy/PDF on a real desktop, including a mixed-DPI multi-monitor setup. Unit, real-engine and headless-overlay checks pass. | `apps/desktop/src/main/screen-ocr-wiring.ts`, `apps/desktop/src/main/windows/screen-ocr-overlay.ts`, `apps/desktop/tests/main/services/screen-ocr-service.test.ts` | Run the installed build: press `Ctrl+Alt+O` on each monitor, confirm the frozen image lines up with the real screen, Esc/Ctrl+C work without a first click, and a saved PDF is searchable. Then flip CAP-17 to `[x]`. |
|
||||
| GAP-OCR-02 | Desktop | Screen text capture is Windows-only and uses a single recognizer language per pass. macOS has no engine (fails closed with a message); with only the Korean Windows OCR pack installed, small English UI text is sometimes misread. | `apps/desktop/src/main/services/screen-ocr/windows-ocr-engine.ts` (`OcrEnginePort`) | Add a macOS Vision adapter behind `OcrEnginePort`; consider a second pass with another installed language for mixed-language screens, or a bundled ONNX recognizer downloaded at runtime. |
|
||||
| GAP-QA-01 | Quality | Extreme Red Team: headful end-to-end bug hunting across real desktop Electron, Web Next.js, and CI pipelines. | `red_team_log.md`, `apps/desktop/tests/e2e`, `apps/web/e2e/smoke.spec.ts` | `[x]` 2026-09-15: 18 scenarios executed, 14 defects caught and 100% resolved (infinite chunking loop DEF-008, IPC signature mismatch DEF-004, markdown editor typing rollback DEF-006, Web RSC Link serialization DEF-012, secret scanner lookahead DEF-013, etc.). All 18 scenarios GREEN with zero regressions. **2026-11-04 rework:** the red-team cycle specs were one-off bug hunts (machine screenshot paths, fixed sleeps, conditional assertions) and were replaced by 5 behavior-scoped protected journeys + a shared Electron fixture; 4 stale/duplicate feature specs, the orphaned root `tests/e2e`, and 7 ad-hoc Playwright runner scripts were deleted. Duplicates merged. New `npm run test:hygiene` gate blocks the patterns that made them meaningless. Log: `docs/tdd-red/06-repo-application.md`. |
|
||||
| GAP-REL-01 | Release | Official release publication to Forgejo and active public download center deployment. | `scripts/ci/publish-forgejo-release.mjs`, `apps/web/src/app/download/page.tsx`, `site/src/sections/Download.tsx`, `apps/web/e2e/smoke.spec.ts` | `[~]` 2026-09-15: v1.1.0 release assets (`D3RO-Voice-Setup-1.1.0-x64.exe`, `.blockmap`, `latest.yml`, `update-policy.json`) published to canonical Forgejo registry and release hub. 2026-09-16: the published 1.1.0 installer carries no Authenticode signature, so it does not satisfy the release policy; product version moved to `1.2.0` and publication must come from CI with the signing gate GREEN. Download centers in `apps/web` (`/download`) and `site` (`#download`) link the canonical Forgejo feed. 2026-09-19: `1.3.7` (overlay fix) is published to the canonical updater feed (`latest.yml` = 1.3.7, 90.6MiB); the CI signing gate that blocks tag-driven publication is still unresolved, so this went out through the local updater path (GAP-REL-06). 2026-09-23: `1.5.0` published the same way (updater feed `latest.yml` = 1.5.0, installer 91.2MiB / 95,612,108 bytes, sha512 matches the local build and the feed serves it anonymously); the runtime and portable channels were republished first. **2026-11-14 (`v1.17.1`):** Desktop `1.17.1` (`6c6875b`, `D3RO-Voice-Setup-1.17.1-x64.exe` `95,294,301` bytes + Ed25519 signed `latest.yml`/`.sig`), `runtime-1.17.1`/`runtime-latest`, `portable-1.17.1`/`portable-latest`, 및 Landing `site/` (`https://d3ro.chanpaca.net/release-identity.json` = `1.17.1`/`6c6875b`)가 프로덕션에 게시됐다(단 Authenticode SmartScreen 평판 `GAP-REL-02`/`06` 및 Release #339 ~91MB 첨부 `GAP-REL-16` 예외는 유지). |
|
||||
| GAP-REL-08 | Release | 다운로드 센터가 **존재하지 않는 설치 파일**을 가리켰다. `apps/web/src/lib/desktop-release.ts`와 `site/src/release.ts`의 `DESKTOP_VERSION`이 `1.2.0`에 멈춰 있어 설치 URL이 `D3RO-Voice-Setup-1.2.0-x64.exe`였고, 그 경로는 피드에서 404다(실측: 1.2.0=404, 1.3.7=206). `version:sync`가 이 두 표면을 덮지 않아 계속 어긋났다. | `scripts/ci/sync-version.mjs`, `apps/web/src/lib/desktop-release.ts`, `site/src/release.ts` | `[x]` 2026-09-19: 두 다운로드 계약 파일을 `sync-version.mjs` 대상에 추가해 버전·릴리스일이 SSOT에서 자동 반영되도록 하고, 현재 값(1.3.7 / 2026-09-19)으로 정정했다. `version:check`·typecheck·site 빌드 GREEN. |
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue