fix(desktop): add precise local OCR reading

This commit is contained in:
Yun Chan 2026-11-21 10:15:49 +09:00
parent 59cdddbecb
commit 25fd3ac004
30 changed files with 626 additions and 15 deletions

View file

@ -22,7 +22,7 @@ Legend: `[ ]` open · `[~]` in progress · `[!]` blocked externally · `[x]` res
| ID | Area | Gap | Evidence | Suggested next step |
|---|---|---|---|---|
| GAP-OCR-01 | Desktop | Screen text capture (CAP-17) has not been driven end-to-end in the running app: global hotkey → overlay takes focus → drag select → copy/PDF on a real desktop, including a mixed-DPI multi-monitor setup. Unit, real-engine and headless-overlay checks pass. | `apps/desktop/src/main/screen-ocr-wiring.ts`, `apps/desktop/src/main/windows/screen-ocr-overlay.ts`, `apps/desktop/tests/main/services/screen-ocr-service.test.ts` | Run the installed build: press `Ctrl+Alt+O` on each monitor, confirm the frozen image lines up with the real screen, Esc/Ctrl+C work without a first click, and a saved PDF is searchable. Then flip CAP-17 to `[x]`. |
| GAP-OCR-02 | Desktop | Screen text capture is Windows-only and uses a single recognizer language per pass. macOS has no engine (fails closed with a message); with only the Korean Windows OCR pack installed, small English UI text is sometimes misread. | `apps/desktop/src/main/services/screen-ocr/windows-ocr-engine.ts` (`OcrEnginePort`) | Add a macOS Vision adapter behind `OcrEnginePort`; consider a second pass with another installed language for mixed-language screens, or a bundled ONNX recognizer downloaded at runtime. |
| GAP-OCR-02 | Desktop | The default screen OCR remains Windows-only; macOS has no engine (fails closed). Korean Windows OCR misread symbols and English/code, while generative alternatives can still alter rare Hangul, punctuation and whitespace. The installed-app precise-reading journey and mixed real-screen evaluation are not verified. | `apps/desktop/src/main/services/screen-ocr/{windows-ocr-engine,ollama-precision-engine,precision-output}.ts`, `apps/desktop/src/main/services/ScreenOcrService.ts`, `packages/core/src/screen-ocr.ts` | `[~]` 2026-11-21: GLM-OCR precise reading is implemented for selected crops with loopback-only privacy, explicit model download, bounded output, original-text fallback and 51 focused tests. Official model/GGUF comparisons included Windows, PP-OCRv5 and PaddleOCR-VL 1.6; GLM was selected for code fidelity and crop latency, not the best overall prose CER. The real service preserved all 1,215 non-whitespace characters and 403 symbols in a code fixture but had three whitespace differences. **Next:** verify the precise button in an installed build and test held-out real screens; add macOS Vision and evaluate rare Hangul/whitespace separately. Do not claim universal exact OCR or ship Paddle/PyTorch dependencies based on synthetic scores alone. |
| GAP-QA-01 | Quality | Extreme Red Team: headful end-to-end bug hunting across real desktop Electron, Web Next.js, and CI pipelines. | `red_team_log.md`, `apps/desktop/tests/e2e`, `apps/web/e2e/smoke.spec.ts` | `[x]` 2026-09-15: 18 scenarios executed, 14 defects caught and 100% resolved (infinite chunking loop DEF-008, IPC signature mismatch DEF-004, markdown editor typing rollback DEF-006, Web RSC Link serialization DEF-012, secret scanner lookahead DEF-013, etc.). All 18 scenarios GREEN with zero regressions. **2026-11-04 rework:** the red-team cycle specs were one-off bug hunts (machine screenshot paths, fixed sleeps, conditional assertions) and were replaced by 5 behavior-scoped protected journeys + a shared Electron fixture; 4 stale/duplicate feature specs, the orphaned root `tests/e2e`, and 7 ad-hoc Playwright runner scripts were deleted. Duplicates merged. New `npm run test:hygiene` gate blocks the patterns that made them meaningless. Log: `docs/tdd-red/06-repo-application.md`. |
| GAP-REL-01 | Release | Official release publication to Forgejo and active public download center deployment. | `scripts/ci/publish-forgejo-release.mjs`, `apps/web/src/app/download/page.tsx`, `site/src/sections/Download.tsx`, `apps/web/e2e/smoke.spec.ts` | `[~]` 2026-09-15: v1.1.0 release assets (`D3RO-Voice-Setup-1.1.0-x64.exe`, `.blockmap`, `latest.yml`, `update-policy.json`) published to canonical Forgejo registry and release hub. 2026-09-16: the published 1.1.0 installer carries no Authenticode signature, so it does not satisfy the release policy; product version moved to `1.2.0` and publication must come from CI with the signing gate GREEN. Download centers in `apps/web` (`/download`) and `site` (`#download`) link the canonical Forgejo feed. 2026-09-19: `1.3.7` (overlay fix) is published to the canonical updater feed (`latest.yml` = 1.3.7, 90.6MiB); the CI signing gate that blocks tag-driven publication is still unresolved, so this went out through the local updater path (GAP-REL-06). 2026-09-23: `1.5.0` published the same way (updater feed `latest.yml` = 1.5.0, installer 91.2MiB / 95,612,108 bytes, sha512 matches the local build and the feed serves it anonymously); the runtime and portable channels were republished first. **2026-11-14 (`v1.17.1`):** Desktop `1.17.1` (`6c6875b`, `D3RO-Voice-Setup-1.17.1-x64.exe` `95,294,301` bytes + Ed25519 signed `latest.yml`/`.sig`), `runtime-1.17.1`/`runtime-latest`, `portable-1.17.1`/`portable-latest`, 및 Landing `site/` (`https://d3ro.chanpaca.net/release-identity.json` = `1.17.1`/`6c6875b`)가 프로덕션에 게시됐다(단 Authenticode SmartScreen 평판 `GAP-REL-02`/`06` 및 Release #339 ~91MB 첨부 `GAP-REL-16` 예외는 유지). |
| GAP-REL-08 | Release | 다운로드 센터가 **존재하지 않는 설치 파일**을 가리켰다. `apps/web/src/lib/desktop-release.ts`와 `site/src/release.ts`의 `DESKTOP_VERSION`이 `1.2.0`에 멈춰 있어 설치 URL이 `D3RO-Voice-Setup-1.2.0-x64.exe`였고, 그 경로는 피드에서 404다(실측: 1.2.0=404, 1.3.7=206). `version:sync`가 이 두 표면을 덮지 않아 계속 어긋났다. | `scripts/ci/sync-version.mjs`, `apps/web/src/lib/desktop-release.ts`, `site/src/release.ts` | `[x]` 2026-09-19: 두 다운로드 계약 파일을 `sync-version.mjs` 대상에 추가해 버전·릴리스일이 SSOT에서 자동 반영되도록 하고, 현재 값(1.3.7 / 2026-09-19)으로 정정했다. `version:check`·typecheck·site 빌드 GREEN. |
@ -72,7 +72,7 @@ Legend: `[ ]` open · `[~]` in progress · `[!]` blocked externally · `[x]` res
| GAP-MTG-03 | Meetings | Existing recordings could become plain transcription history, but not a meeting with linked audio and minutes. | desktop `MeetingModeService`, `meeting/imported-recording.ts`, `meeting-mode-handlers.ts`; mobile `MeetingsScreen` → `RecordScreen`; MTG-13 | `[~]` 2026-11-02: installed Windows 1.16.0 passed native picker→timestamped transcript→local summary/action item with a real WAV (`2026-11-02-verification.md`). Desktop GUI gate is complete; authenticated Android picker→upload→transcript→minutes on Fold6 remains. |
| GAP-MTG-04 | Meetings | Learned typing vocabulary never reached meeting transcription: STT used a hardcoded Korean hint, refine had no glossary, refined lines stopped inside `CaptionService`, and the stored transcript was raw segments. | `packages/core/src/transcript-correction.ts`, `apps/desktop/src/main/services/TranscriptGlossaryService.ts`, `CaptionService.ts`, `meeting/MeetingRecordingSession.ts`, `MeetingModeService.ts`, `FileTranscriptionService.ts`, `VoiceModeService.ts`; MTG-14 | `[~]` 2026-11-05: one glossary SSOT wired into caption, meeting, file transcription and dictation; refined lines now propagate to the meeting transcript and renderer (`meetingMode:segmentUpdated`). **Direct measurement against the real local database** (`users/_local/d3ro.db`: 1,952 learned phrases / 2,522 typing samples / 3 meeting transcripts) exposed that the first cut also rewrote ordinary Korean (286 of 3,005 words: 하면→라면, 이제→이게, 매일→메일, 정보→정도), and v1.16.3 restricts deterministic correction to dictionary pronunciations plus typed Latin identifiers — re-measured **0 of 3,005**. Real-engine check with the local sidecar (`large-v3-turbo`): a Korean TTS fixture containing the user term transcribes as 하람로그에 with no hint and haramlog에 with the learned hint; a dictionary pronunciation entry maps 디에로 → D3RO. Hangul-layout artifacts (dkssudgktpdy) are excluded from the hint. Evidence: `packages/core/__tests__/transcript-correction.test.ts` 16/16, `apps/desktop/tests/main/services/transcript-glossary.test.ts` 10/10, core 589/589, desktop offline suite, workspace typecheck, touched-file ESLint, `test:hygiene`. **Remaining:** a live meeting with real names/jargon by voice, and a packaged-install check of the new IPC event. |
| GAP-REL-16 | Release | Forgejo release-page asset attachment for the ~91 MB installer timed out through Cloudflare (HTTP 524), although package registry upload and signed manifest succeeded. | `scripts/ci/publish-forgejo-release.mjs`, `scripts/ci/lib/immutable-package-guard.mjs` | `[x]` 2026-11-20: `v1.18.4`(`f77c86d`)의 `104,691,365-byte` 설치본·`latest.yml`·`.sig` Release 첨부가 모두 성공하고 공개 Release 설치본 Range GET도 HTTP 206이다. 과거 1.16.2~1.17.1의 524/ECONNRESET 이력과 달리 현재 경로는 완료했다. 증거: `docs/evidence/2026-11-17-backlog-operations.md`. |
| GAP-REL-17 | Release | 설치된 1.17.1은 자동 업데이트를 정상 확인하지만 공개 피드도 1.17.1이라 1.18.x 제안이 나오지 않았다. 미게시 1.18.1·1.18.2 설치본이 각각 120,821,996·120,821,907 bytes(115.2 MiB)라 100 MiB 업로드 게이트에서 중단됐고, 버전 태그와 원격 피드가 어긋났다. | `apps/desktop/electron-builder.yml`, `scripts/ci/{build-desktop-installer,release-desktop,publish-forgejo-release}.mjs`, `docs/evidence/2026-11-17-backlog-operations.md` | `[~]` 2026-11-20: `1.18.4`(`f77c86d`)는 `!release/**`·명시적 `out/**/*`, source map·미사용 플랫폼 바이너리 제외로 `104,691,365 bytes`가 됐다. Windows는 `.blockmap` 대신 전체 설치본을 받는다(`fullInstallVersionGap=0`). 빌드·native DB·smoke(3 valid/10 negative)·공개 피드·signed manifest·Release 자산 모두 GREEN. `.env` Forgejo token 401은 남아 있고 이번 게시는 유효한 기존 Git credential을 프로세스 메모리에서만 사용했다. **남은 것:** 1.17.1 실제 앱의 설정→정보→업데이트 제안·다운로드·재시작 GUI 확인. |
| GAP-REL-17 | Release | 설치된 1.17.1은 자동 업데이트를 정상 확인하지만 공개 피드도 1.17.1이라 1.18.x 제안이 나오지 않았다. 미게시 1.18.1·1.18.2 설치본이 각각 120,821,996·120,821,907 bytes(115.2 MiB)라 100 MiB 업로드 게이트에서 중단됐고, 버전 태그와 원격 피드가 어긋났다. | `apps/desktop/electron-builder.yml`, `scripts/ci/{build-desktop-installer,release-desktop,publish-forgejo-release}.mjs`, `docs/evidence/2026-11-17-backlog-operations.md` | `[x]` 2026-11-21: `1.18.4`(`f77c86d`, `104,691,365 bytes`)의 제안 발견·전체 설치본 다운로드·Ed25519 서명 검증·`quitAndInstall`·재시작·기존 DB 열기를 실제 설치 앱 로그로 확인했다. ProductVersion `1.18.4.0`, 이후 최신 버전 확인 및 실제 화면 OCR 사용도 검증했다. Windows full-only와 packaging exclusions는 유지한다. `.env` Forgejo token 401은 별도 교체 대상이며 이번 게시는 유효한 기존 Git credential을 프로세스 메모리에서만 사용했다. |
| GAP-INFRA-01 | Build | `apps/mobile-rn` is outside npm workspaces, so root `typecheck`/`test`/`lint` skip it. `typecheck:mobile`/`lint:mobile`/`test:mobile`/`verify:all` root scripts added 2026-09-13 (`package.json`), but membership/CI integration is still open. | `package.json`, `.forgejo/workflows/ci.yml`, `.gitlab-ci.yml` | `[x]` 2026-10-30: React Native Metro/Gradle 네이티브 오토링킹 격리를 위해 `apps/mobile-rn`은 독립 `node_modules`로 유지하고, 루트 `verify:all` 및 `.forgejo/workflows/ci.yml`·`.gitlab-ci.yml`의 `mobile-quality` 잡이 모바일 `lint`·`typecheck`·`test:ci`를 필수 게이트로 실행하도록 정본화했다. **2026-11-04:** root `test:mobile`도 watch-capable 기본 Jest가 아니라 mobile `test:ci`(`jest --runInBand`)를 호출하도록 고정했다. |
| GAP-MOB-01 | Legacy | `apps/mobile` Expo skeleton duplicated auth/data code and got version-sync edits. | SSOT G-002 | `[x]` 2026-09-13: deleted `apps/mobile`, removed from `sync-version.mjs` and `package-lock.json`; `version:check` GREEN. |
| GAP-INFRA-02 | CI | GitLab admin NAS deploy job is disabled; admin deploy is manual/GHCR. | `.gitlab-ci.yml`, `docs/deployment/nas-deploy-runbook.md` | `[x]` 2026-10-30: 임시 CI DinD 데몬만 바꾸던 구 잡 대신 `docker-compose.nas.yml` 기반의 정본 배포·헬스체크·롤백 절차를 `docs/deployment/nas-deploy-runbook.md`로 문서화하고 `.gitlab-ci.yml` 및 `02-infrastructure.md`에 연결했다. |