fix(desktop): add precise local OCR reading
This commit is contained in:
parent
59cdddbecb
commit
25fd3ac004
30 changed files with 626 additions and 15 deletions
|
|
@ -30,7 +30,7 @@ Status quick-reference: `[x]` done+verified · `[~]` partial/unverified · `[ ]`
|
|||
| CAP-14 | Recording persistence / crash recovery | [x] | [ ] | [x] | [-] | Desktop WAV persist; mobile durable queue + process-kill WAV recovery |
|
||||
| CAP-15 | Android foreground recording service | [-] | [-] | [x] | [-] | Mobile API 34 FGS + persistent notification (SSOT R-005 GREEN) |
|
||||
| CAP-16 | Rebindable global key bindings (keyboard + mouse) | [x] | [-] | [-] | [-] | Contract SSOT `packages/core/src/keybinding.ts`: `KEY_CATALOG` (10 groups), `KEYBINDING_ACTIONS` (6 actions), `validateBinding`, `detectBindingConflicts`. Multiple bindings per action persist as one `AppConfig.keyBindings` map (`packages/core/src/types.ts`), replacing the four singular `*Shortcut` fields; `ConfigService` migrates legacy values once. `KeyBindingService` hooks keyboard **and** mouse via uiohook — MB1 is not bindable, MB2/MB3 need a modifier, MB4/MB5 are free. **Verified 2026-09-21 on Windows by a manual run** and **2026-10-30 unit-tested** in `KeyBindingService.test.ts` (11 tests, GAP-KEY-01 `[x]`), with dedicated `command` action toggle/hold + forced voice-command matching in `VoiceModeService.ts` / `voice-intent.ts` (`voice-intent.test.ts` 19 tests, GAP-KEY-03 `[x]`). **Still open:** macOS/Linux mouse behavior (`11` GAP-KEY-02). W/M `[-]`: no OS-level global binding surface exists there. B `[-]`: device-local setting. |
|
||||
| CAP-17 | Screen text capture (OCR overlay) | [~] | [-] | [-] | [-] | Desktop/Windows. The rebindable `screen-ocr` action (default `Ctrl+Alt+O`, CAP-16) freezes the monitor under the cursor and covers it with the `screen-ocr` popup; text is selected by flow drag (start on a word), rectangle drag (start on empty space or in Area mode), double-click word, triple-click line, Shift+click extend, `Ctrl+A`. Actions: copy (multi-line / one line), copy image, save PNG, **searchable PDF** (image above a real text layer), translate and summarize through `LlmGateway`, read aloud (`TTSPlaybackService`), open link / web search, recognition language switch (`AppConfig.screenOcrLanguage`). Pure selection/text/PDF logic `packages/core/src/screen-ocr.ts`; service `apps/desktop/src/main/services/ScreenOcrService.ts`; engine `services/screen-ocr/windows-ocr-engine.ts` (Windows.Media.Ocr via `powershell.exe -EncodedCommand`, 2x upscale); capture `utils/win32-screen-capture.ts` (GDI BitBlt, 39 ms vs 2.9 s for `desktopCapturer` on a 3-monitor PC). The capture touches disk only as one temp PNG that is deleted right after recognition. **Verified 2026-11-20:** core 34 + desktop 41 tests, real-engine probe (Korean+English screen, 0.64 s), headless overlay run (flow/rect selection, copy, translate, PDF request, release on close) and a printed PDF whose text layer is searchable. **Not yet verified:** the live hotkey journey in the running app (focus, mixed-DPI multi-monitor) — `GAP-OCR-01`. macOS has no engine and fails closed with a message — `GAP-OCR-02`. |
|
||||
| CAP-17 | Screen text capture (OCR overlay) | [~] | [-] | [-] | [-] | Desktop/Windows. The rebindable `screen-ocr` action (default `Ctrl+Alt+O`, CAP-16) freezes the monitor under the cursor and covers it with the `screen-ocr` popup; text is selected by flow drag (start on a word), rectangle drag (start on empty space or in Area mode), double-click word, triple-click line, Shift+click extend, `Ctrl+A`. Actions: copy (multi-line / one line), copy image, save PNG, **searchable PDF** (image above a real text layer), translate and summarize through `LlmGateway`, read aloud (`TTSPlaybackService`), open link / web search, recognition language switch (`AppConfig.screenOcrLanguage`). Pure selection/text/PDF logic `packages/core/src/screen-ocr.ts`; service `apps/desktop/src/main/services/ScreenOcrService.ts`; engine `services/screen-ocr/windows-ocr-engine.ts` (Windows.Media.Ocr via `powershell.exe -EncodedCommand`, 2x upscale); capture `utils/win32-screen-capture.ts` (GDI BitBlt, 39 ms vs 2.9 s for `desktopCapturer` on a 3-monitor PC). The capture touches disk only as one temp PNG that is deleted right after recognition. **Verified 2026-11-20:** core 34 + desktop 39 tests, real-engine probe (Korean+English screen, 0.64 s), headless overlay run (flow/rect selection, copy, translate, PDF request, release on close) and a printed PDF whose text layer is searchable. **2026-11-21:** installed 1.18.4 ran the real OCR overlay. Pixel-aware joining has 37 core tests (full core 666); GLM precise mode has 51 focused tests. A local GLM-OCR precise action now reads only the selected crop, downloads the model on explicit request, keeps the original on failure, and rejects remote Ollama URLs and truncated/repeated output. Real model comparison: Windows 32.29% overall CER, PP-OCRv5 14.35%, PaddleOCR-VL crop 2.99%, GLM crop 5.08% (GLM has rare Korean/format errors but exact code content apart from generated fences and was 3.74s vs Paddle 20.89s mean). The real service preserved all 1,215 non-whitespace characters and 403 symbols in the code fixture, with three whitespace-only differences and no temp files left. Built-overlay selection→crop→result→copy passes. **Not yet verified:** installed-app precise button, mixed-DPI multi-monitor focus/copy/PDF, rare Korean/whitespace parity — `GAP-OCR-01` and `GAP-OCR-02`. macOS has no engine and fails closed with a message — `GAP-OCR-02`. |
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -234,7 +234,7 @@ end-to-end behaviour has **not been verified by typing in a real app** (`11` GAP
|
|||
| INFRA-12 | Cloudflare edge + tunnel | [x] | `server/cloudflare-worker`, Cloudflare Tunnel `kd-nas` |
|
||||
| INFRA-13 | Site deploy (Cloudflare Pages) | [x] | `.forgejo/workflows/deploy-site.yml`; the unused GitHub Pages workflow was removed in Wave 3 |
|
||||
| INFRA-17 | Test hygiene & TDD v2.0 gates | [x] | `scripts/ci/check-test-hygiene.mjs` (`test:hygiene`), `check-protected-tests.mjs` (`check:protected-tests`), `check-test-quality.mjs` (`test:quality`), `check-i18n-contract.mjs` (`check:i18n`), `check-ds-contract.mjs` (`check:ds-contract`), `run-affected-tests.mjs` (`test:affected`), `verify-desktop-installer-smoke.mjs` (`release:installer:smoke`), and `@d3ro/core/ports` (`Clock`/`Random` deterministic ports) — all equipped with `--self-test` and wired into `verify:all`, `.forgejo/workflows/ci.yml`, `release-desktop.mjs`, and `.forgejo/workflows/release.yml`. Applied policy: `docs/tdd-red/06-repo-application.md`, `07-revamp-plan.md`. |
|
||||
| INFRA-15 | Update & release system | [~] | Canonical Forgejo feed + channels/policy, publisher `scripts/ci/publish-forgejo-release.mjs`; `npm run release:metadata:test`. 2026-11-20 `1.18.4`(`f77c86d`) Windows full-only 설치본 `104,691,365 bytes`를 게시하고 공개 피드·signed manifest·Release HTTP 206을 확인했다. 과거 release 재귀 포함과 source map·미사용 플랫폼 binary를 차단한다. 실제 1.17.1 설치본의 업데이트 제안·설치 GUI는 `11` GAP-REL-17의 남은 게이트다. |
|
||||
| INFRA-15 | Update & release system | [x] | Canonical Forgejo feed + channels/policy, publisher `scripts/ci/publish-forgejo-release.mjs`; `npm run release:metadata:test`. `1.18.4`(`f77c86d`) Windows full-only 설치본 `104,691,365 bytes`를 게시했다. 2026-11-21 실제 1.17.1 설치본에서 제안 발견·다운로드·서명 검증·재시작 설치·기존 DB 열기를 확인했고 ProductVersion `1.18.4.0`과 실제 화면 OCR 사용을 검증했다. 과거 release 재귀 포함과 source map·미사용 플랫폼 binary를 차단한다. |
|
||||
| INFRA-16 | Desktop STT engine packaging | [x] | `apps/desktop/scripts/setup-sidecar.mjs` + `build-sidecar.mjs`, `electron-builder.yml` `extraResources` (`sidecar-dist/sidecar` → `resources/sidecar`, `resources/ffmpeg`), and `scripts/ci/verify-sidecar-bundle.mjs` run in `package-windows`/`package-macos` before electron-builder. Verified on the real bundle: `sidecar.exe` + `_internal` including `faster_whisper/assets/silero_vad_v6.onnx`, plus a packaged-engine transcription round-trip on GPU. |
|
||||
| INFRA-17 | 서명 없는 배포 채널 (portable + Scoop) | [x] | `scripts/ci/build-portable.mjs` (95MiB 7z 분할 볼륨 + Scoop 매니페스트), `scripts/ci/publish-portable-release.mjs`, `scripts/local/install-d3ro-voice.ps1`, `bucket/` 버킷, `.forgejo/workflows/portable.yml`; 7z 분할 볼륨(Scoop, 162MiB) + zip 분할 부품(수동 설치, 243MiB, 7-Zip 불필요); updater feed와 분리. 2026-09-18 `portable-1.3.1` 게시 + 실제 설치 검증. |
|
||||
| INFRA-18 | 로컬 런타임 온디맨드 설치 | [x] | `RuntimeProvisioner`(부품 다운로드 + SHA-256 검증 + tar 해제, `%APPDATA%/d3ro-voice/runtime`), `POST runtime:ensure` / `runtime:progress` IPC, 설정 > STT 상태/내려받기 UI. 설치본에서 엔진/ffmpeg를 분리해 189MB → 90.6MiB, 업데이트 피드 게시 복구. 검증은 전부 디스크에 기록된 파일 기준이며(부품 크기·해시 → 결합본 크기·해시), 부품 다운로드는 최대 3회 재시도한다. 2026-09-18 실제 feed 통합 검증(엔진 94.4MiB/18초, ffmpeg 21.7MiB/5초). |
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue