feat(desktop): screen text capture (OCR) overlay with selectable text

A rebindable screen-ocr action (default Ctrl+Alt+O) freezes the monitor under the cursor and
covers it with a focusable overlay. Text is selected by flow drag, rectangle drag, double/triple
click, Shift+click or Ctrl+A, then copied, saved as PNG or searchable PDF, translated,
summarized, read aloud, opened as a link or searched.

- Selection, text assembly and the PDF document are pure functions in @d3ro/core/screen-ocr,
  shared by the main process and the overlay.
- Recognition uses Windows.Media.Ocr through powershell.exe -EncodedCommand with 2x upscaling;
  the capture reaches disk only as a temp PNG that is removed right after recognition.
- Capture uses GDI BitBlt for the target monitor (39 ms) because desktopCapturer captured all
  three monitors and took 2.9 s when DXGI duplication timed out.
- The accelerator is reserved only where the feature is available, so macOS does not swallow
  the key for a feature it cannot run (fails closed with a message).

Map: CAP-17 [~], GAP-OCR-01 (live hotkey journey not yet driven in the running app),
GAP-OCR-02 (macOS engine, mixed-language accuracy).
This commit is contained in:
Yun Chan 2026-11-20 15:16:37 +09:00
parent 6791e29f27
commit 03cd8bf890
41 changed files with 4561 additions and 17 deletions

View file

@ -21,6 +21,8 @@ Legend: `[ ]` open · `[~]` in progress · `[!]` blocked externally · `[x]` res
| ID | Area | Gap | Evidence | Suggested next step |
|---|---|---|---|---|
| GAP-OCR-01 | Desktop | Screen text capture (CAP-17) has not been driven end-to-end in the running app: global hotkey → overlay takes focus → drag select → copy/PDF on a real desktop, including a mixed-DPI multi-monitor setup. Unit, real-engine and headless-overlay checks pass. | `apps/desktop/src/main/screen-ocr-wiring.ts`, `apps/desktop/src/main/windows/screen-ocr-overlay.ts`, `apps/desktop/tests/main/services/screen-ocr-service.test.ts` | Run the installed build: press `Ctrl+Alt+O` on each monitor, confirm the frozen image lines up with the real screen, Esc/Ctrl+C work without a first click, and a saved PDF is searchable. Then flip CAP-17 to `[x]`. |
| GAP-OCR-02 | Desktop | Screen text capture is Windows-only and uses a single recognizer language per pass. macOS has no engine (fails closed with a message); with only the Korean Windows OCR pack installed, small English UI text is sometimes misread. | `apps/desktop/src/main/services/screen-ocr/windows-ocr-engine.ts` (`OcrEnginePort`) | Add a macOS Vision adapter behind `OcrEnginePort`; consider a second pass with another installed language for mixed-language screens, or a bundled ONNX recognizer downloaded at runtime. |
| GAP-QA-01 | Quality | Extreme Red Team: headful end-to-end bug hunting across real desktop Electron, Web Next.js, and CI pipelines. | `red_team_log.md`, `apps/desktop/tests/e2e`, `apps/web/e2e/smoke.spec.ts` | `[x]` 2026-09-15: 18 scenarios executed, 14 defects caught and 100% resolved (infinite chunking loop DEF-008, IPC signature mismatch DEF-004, markdown editor typing rollback DEF-006, Web RSC Link serialization DEF-012, secret scanner lookahead DEF-013, etc.). All 18 scenarios GREEN with zero regressions. **2026-11-04 rework:** the red-team cycle specs were one-off bug hunts (machine screenshot paths, fixed sleeps, conditional assertions) and were replaced by 5 behavior-scoped protected journeys + a shared Electron fixture; 4 stale/duplicate feature specs, the orphaned root `tests/e2e`, and 7 ad-hoc Playwright runner scripts were deleted. Duplicates merged. New `npm run test:hygiene` gate blocks the patterns that made them meaningless. Log: `docs/tdd-red/06-repo-application.md`. |
| GAP-REL-01 | Release | Official release publication to Forgejo and active public download center deployment. | `scripts/ci/publish-forgejo-release.mjs`, `apps/web/src/app/download/page.tsx`, `site/src/sections/Download.tsx`, `apps/web/e2e/smoke.spec.ts` | `[~]` 2026-09-15: v1.1.0 release assets (`D3RO-Voice-Setup-1.1.0-x64.exe`, `.blockmap`, `latest.yml`, `update-policy.json`) published to canonical Forgejo registry and release hub. 2026-09-16: the published 1.1.0 installer carries no Authenticode signature, so it does not satisfy the release policy; product version moved to `1.2.0` and publication must come from CI with the signing gate GREEN. Download centers in `apps/web` (`/download`) and `site` (`#download`) link the canonical Forgejo feed. 2026-09-19: `1.3.7` (overlay fix) is published to the canonical updater feed (`latest.yml` = 1.3.7, 90.6MiB); the CI signing gate that blocks tag-driven publication is still unresolved, so this went out through the local updater path (GAP-REL-06). 2026-09-23: `1.5.0` published the same way (updater feed `latest.yml` = 1.5.0, installer 91.2MiB / 95,612,108 bytes, sha512 matches the local build and the feed serves it anonymously); the runtime and portable channels were republished first. **2026-11-14 (`v1.17.1`):** Desktop `1.17.1` (`6c6875b`, `D3RO-Voice-Setup-1.17.1-x64.exe` `95,294,301` bytes + Ed25519 signed `latest.yml`/`.sig`), `runtime-1.17.1`/`runtime-latest`, `portable-1.17.1`/`portable-latest`, 및 Landing `site/` (`https://d3ro.chanpaca.net/release-identity.json` = `1.17.1`/`6c6875b`)가 프로덕션에 게시됐다(단 Authenticode SmartScreen 평판 `GAP-REL-02`/`06` 및 Release #339 ~91MB 첨부 `GAP-REL-16` 예외는 유지). |
| GAP-REL-08 | Release | 다운로드 센터가 **존재하지 않는 설치 파일**을 가리켰다. `apps/web/src/lib/desktop-release.ts`와 `site/src/release.ts`의 `DESKTOP_VERSION`이 `1.2.0`에 멈춰 있어 설치 URL이 `D3RO-Voice-Setup-1.2.0-x64.exe`였고, 그 경로는 피드에서 404다(실측: 1.2.0=404, 1.3.7=206). `version:sync`가 이 두 표면을 덮지 않아 계속 어긋났다. | `scripts/ci/sync-version.mjs`, `apps/web/src/lib/desktop-release.ts`, `site/src/release.ts` | `[x]` 2026-09-19: 두 다운로드 계약 파일을 `sync-version.mjs` 대상에 추가해 버전·릴리스일이 SSOT에서 자동 반영되도록 하고, 현재 값(1.3.7 / 2026-09-19)으로 정정했다. `version:check`·typecheck·site 빌드 GREEN. |