fix(desktop): add precise local OCR reading

This commit is contained in:
Yun Chan 2026-11-21 10:15:49 +09:00
parent 59cdddbecb
commit 25fd3ac004
30 changed files with 626 additions and 15 deletions

View file

@ -62,7 +62,7 @@ Singleton + `EventEmitter` pattern (`getXService()` accessors).
| `VoiceCommandService` | Keyword → command rule matching |
| `ChainService` | Multi-step LLM pipelines (LLMChain). Each step resolves its instruction through `llm-prompts.ts` (`ChainService.ts:196`); before that, chain steps sent placeholders through unsubstituted |
| `ScreenContextService` | Active-window + selected-text context (on Windows uses `sendWin32CopyChord()` via `koffi` `SendInput` without loading `nut-js`) |
| `ScreenOcrService` | Screen text capture (CAP-17): owns one capture session — capture → overlay → async recognition → overlay requests (copy text/image, save PNG/searchable PDF, translate, summarize, read aloud, open link/search, language switch) → close. All OS access is behind ports (`services/screen-ocr/electron-adapters.ts`); engine port `OcrEnginePort` has one adapter, `windows-ocr-engine.ts` (Windows.Media.Ocr). Assembled in `src/main/screen-ocr-wiring.ts` (key binding, overlay IPC with sender check, `Escape` capture while open except during a save dialog). Events: `opened`, `closed`. |
| `ScreenOcrService` | Screen text capture (CAP-17): owns one capture session — capture → overlay → async recognition → overlay requests (copy text/image, save PNG/searchable PDF, **local GLM-OCR precise reading**, translate, summarize, read aloud, open link/search, language switch) → close. All OS access is behind ports (`services/screen-ocr/electron-adapters.ts`); base engine port `OcrEnginePort` has one adapter, `windows-ocr-engine.ts` (Windows.Media.Ocr). `selectionText` uses word boxes to join tightly spaced URL/email/punctuation tokens without collapsing ordinary word spacing; it never guesses replacement glyphs. The separate `PrecisionOcrEnginePort` (`screen-ocr/ollama-precision-engine.ts`) sends only a selected crop to native `/api/generate` on a **loopback** Ollama server (remote URLs rejected), downloads `glm-ocr:latest` only after the explicit precise action, keeps it for 2m, bounds output tokens, rejects empty/truncated output, and preserves the original on failure. `precision-output.ts` removes only a single generated outer code fence when neither the body nor source contains fences. Assembled in `src/main/screen-ocr-wiring.ts` (key binding, overlay IPC with sender check, `Escape` capture while open except during a save dialog). Events: `opened`, `closed`. |
### Phase 10+ features
| Service | Purpose |
@ -105,7 +105,7 @@ The existing `input-telemetry-handlers` and `suggestion-handlers` IPC extensions
| `CloudSyncService` | Supabase auth + lifecycle: per-user DB switching, local-mode import on first sign-in, Realtime (`ws` transport) + 5-min heartbeat, device check-in, debounced flush/pull triggers. The sync itself lives in `services/sync/` |
| `services/sync/` | `SyncEngine` (backfill once per user DB → push outbox → pull by server-clock keyset cursor → tombstones → memo-tag reconcile), `sync-outbox` (`sync_outbox`/`sync_state` tables), `sync-adapters` (history, dictionary, meetings, meeting memos/documents, custom commands, templates), `memo-tag-sync`, `supabase-sync-remote`, `device-registration`, `realtime-transport`, `settings-sync` (mobile `user_settings` ↔ language/theme/defaultLLMAction/activeInstructionId), `audio-sync` (recording upload to the `audio` bucket, remote cleanup on delete), knowledge adapter (source-text chunks), `transcript-sync` (meeting segments from the desktop transcript), `builtin-instruction-sync` (preset prompt edits ↔ server preset rows). Services record changes with `getCloudSyncService().pushOne/pushDelete(entity, id)`; pulled rows are applied without re-queuing |
| `CloudSTTService` | Thin cloud STT wrapper over `D3ROCloudDriver` |
| `UpdateService` | electron-updater (canonical Forgejo feed, channels, mandatory/full-vs-delta policy, staged rollout). Native consent/restart dialogs resolve `update.dialog.*` through a fresh `getPopupTranslator()` at each prompt, so a runtime locale change is reflected without rebuilding the prompter; unit tests cover required/optional paths and a ko→en switch on the same prompter. `update-dialog-preview.ts` exposes only the explicit `--preview-update-dialogs` manual-QA entry described above. The current-source package was built and its native ABI verified; packaged dialog appearance/interaction remains unverified until that executable is run on the interactive desktop. |
| `UpdateService` | electron-updater (canonical Forgejo feed, channels, mandatory/full-vs-delta policy, staged rollout). Native consent/restart dialogs resolve `update.dialog.*` through a fresh `getPopupTranslator()` at each prompt, so a runtime locale change is reflected without rebuilding the prompter; unit tests cover required/optional paths and a ko→en switch on the same prompter. `update-dialog-preview.ts` exposes only the explicit `--preview-update-dialogs` manual-QA entry described above. The installed 1.17.1 app found, downloaded, signature-verified and installed 1.18.4 on 2026-11-21, then opened its existing DB and ran the real OCR overlay. The optional consent/restart journey is verified; the four explicit preview-dialog variants are not an exhaustive visual test. |
| `AutoLaunchService` | OS login-item auto-start |
| `LoggerService` | electron-log wrapper + category loggers |
| upgrade / billing | No in-app checkout. `license-handlers.ts` `LICENSE.OPEN_BILLING` opens `billingUrl({ tier })` (web Payple); tier returns via `license:tierChanged`. Stripe `payment-handlers.ts`·`CheckoutModal`·`payment:*` IPC removed 2026-09-26 |