From 03cd8bf8903eb12706c5fe34ef0e2546bd2390b4 Mon Sep 17 00:00:00 2001 From: Yun Chan Date: Fri, 20 Nov 2026 15:16:37 +0900 Subject: [PATCH] feat(desktop): screen text capture (OCR) overlay with selectable text A rebindable screen-ocr action (default Ctrl+Alt+O) freezes the monitor under the cursor and covers it with a focusable overlay. Text is selected by flow drag, rectangle drag, double/triple click, Shift+click or Ctrl+A, then copied, saved as PNG or searchable PDF, translated, summarized, read aloud, opened as a link or searched. - Selection, text assembly and the PDF document are pure functions in @d3ro/core/screen-ocr, shared by the main process and the overlay. - Recognition uses Windows.Media.Ocr through powershell.exe -EncodedCommand with 2x upscaling; the capture reaches disk only as a temp PNG that is removed right after recognition. - Capture uses GDI BitBlt for the target monitor (39 ms) because desktopCapturer captured all three monitors and took 2.9 s when DXGI duplication timed out. - The accelerator is reserved only where the feature is available, so macOS does not swallow the key for a feature it cannot run (fails closed with a message). Map: CAP-17 [~], GAP-OCR-01 (live hotkey journey not yet driven in the running app), GAP-OCR-02 (macOS engine, mixed-language accuracy). --- CHANGELOG.md | 24 + apps/desktop/electron.vite.config.ts | 3 +- apps/desktop/src/main/bootstrap.ts | 8 + apps/desktop/src/main/screen-ocr-wiring.ts | 169 +++++ .../src/main/services/ConfigService.ts | 1 + .../src/main/services/ScreenOcrService.ts | 555 ++++++++++++++ .../services/keybinding-accelerator-policy.ts | 9 + .../services/screen-ocr/electron-adapters.ts | 246 +++++++ .../services/screen-ocr/windows-ocr-engine.ts | 267 +++++++ .../src/main/utils/win32-screen-capture.ts | 171 +++++ .../desktop/src/main/windows/WindowManager.ts | 2 + .../src/main/windows/popup-window-factory.ts | 14 +- .../src/main/windows/screen-ocr-overlay.ts | 69 ++ .../src/renderer/popups/screen-ocr/index.html | 62 ++ .../src/renderer/popups/screen-ocr/script.js | 678 ++++++++++++++++++ .../src/renderer/popups/screen-ocr/style.css | 311 ++++++++ .../services/keybinding-redteam-r2-16.test.ts | 12 + .../main/services/screen-ocr-service.test.ts | 561 +++++++++++++++ .../main/windows/popup-window-options.test.ts | 45 ++ docs/map/04-desktop-app.md | 6 +- docs/map/10-feature-catalog.md | 1 + docs/map/11-gap-backlog.md | 2 + memory/project_status.md | 12 + packages/core/__tests__/screen-ocr.test.ts | 279 +++++++ packages/core/src/errors.ts | 10 + packages/core/src/ipc-channels.ts | 16 + packages/core/src/keybinding.ts | 11 + packages/core/src/screen-ocr.ts | 516 +++++++++++++ packages/core/src/types.ts | 2 + packages/i18n/src/locales/de.json | 43 +- packages/i18n/src/locales/en.json | 43 +- packages/i18n/src/locales/es.json | 43 +- packages/i18n/src/locales/fr.json | 43 +- packages/i18n/src/locales/ja.json | 43 +- packages/i18n/src/locales/ko.json | 43 +- packages/i18n/src/locales/pt.json | 43 +- packages/i18n/src/locales/ru.json | 43 +- packages/i18n/src/locales/th.json | 43 +- packages/i18n/src/locales/vi.json | 43 +- packages/i18n/src/locales/zh-TW.json | 43 +- packages/i18n/src/locales/zh.json | 43 +- 41 files changed, 4561 insertions(+), 17 deletions(-) create mode 100644 apps/desktop/src/main/screen-ocr-wiring.ts create mode 100644 apps/desktop/src/main/services/ScreenOcrService.ts create mode 100644 apps/desktop/src/main/services/screen-ocr/electron-adapters.ts create mode 100644 apps/desktop/src/main/services/screen-ocr/windows-ocr-engine.ts create mode 100644 apps/desktop/src/main/utils/win32-screen-capture.ts create mode 100644 apps/desktop/src/main/windows/screen-ocr-overlay.ts create mode 100644 apps/desktop/src/renderer/popups/screen-ocr/index.html create mode 100644 apps/desktop/src/renderer/popups/screen-ocr/script.js create mode 100644 apps/desktop/src/renderer/popups/screen-ocr/style.css create mode 100644 apps/desktop/tests/main/services/screen-ocr-service.test.ts create mode 100644 apps/desktop/tests/main/windows/popup-window-options.test.ts create mode 100644 packages/core/__tests__/screen-ocr.test.ts create mode 100644 packages/core/src/screen-ocr.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index f85ebe41..61989dcc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Plugin system for custom pipelines +## [1.18.0] - 2026-11-20 + +> Screen text capture (OCR) for Windows, plus the Electron 43 toolchain upgrade and the 2026-11-17 backlog batch. + +### Added +- **Screen text capture (OCR).** Press `Ctrl+Alt+O` (rebindable under Settings → key bindings) to freeze the monitor under the cursor and select the text on it: + - drag from a word to select along the text flow, drag from empty space (or switch to Area mode) to select with a rectangle, double-click a word, triple-click a line, `Shift`+click to extend, `Ctrl+A` for everything; + - copy the text (keeping line breaks or as one line), copy or save the selected area as an image, and save it as a **searchable PDF** whose text can be found and copied; + - translate or summarize the selection with the configured AI model, have it read aloud, open a link found in it, or search the web for it; + - switch the recognition language among the Windows OCR languages installed on the PC. +- Mobile: share a history entry as a Markdown file or as its original audio. + +### Changed +- Desktop runtime upgraded from Electron 33 to Electron 43 (with Vite 8, Vitest 5 and Node 24 for the build); clipboard reads and writes are awaited end to end. +- Screen capture for OCR reads only the target monitor through GDI (about 40 ms) instead of capturing every monitor. + +### Fixed +- Scheduled push delivery no longer fails with HTTP 401: the drain sends the `apikey` header that current Supabase service keys require. + +### Notes +- Screen text capture uses the OCR engine built into Windows and is not available on macOS yet. With only the Korean OCR language installed, small English UI text can be misread; installing the English language pack in Windows Settings adds it to the language list. +- The capture stays on the PC. It is written to disk only as a temporary file for recognition and deleted right after; only the text you select is sent to the AI model when you choose translate or summarize. + + ## [1.17.1] - 2026-11-14 > Supersedes unpublished `1.17.0` with the full-length ASAR Pickle header smoke verification fix and ships the full `1.17.0` multi-surface release. diff --git a/apps/desktop/electron.vite.config.ts b/apps/desktop/electron.vite.config.ts index dee49331..e3094497 100644 --- a/apps/desktop/electron.vite.config.ts +++ b/apps/desktop/electron.vite.config.ts @@ -88,7 +88,8 @@ export default defineConfig({ 'popups/suggestion-overlay': resolve( __dirname, 'src/renderer/popups/suggestion-overlay/index.html' - ) + ), + 'popups/screen-ocr': resolve(__dirname, 'src/renderer/popups/screen-ocr/index.html') } } }, diff --git a/apps/desktop/src/main/bootstrap.ts b/apps/desktop/src/main/bootstrap.ts index 550dbc3c..5917d964 100644 --- a/apps/desktop/src/main/bootstrap.ts +++ b/apps/desktop/src/main/bootstrap.ts @@ -52,6 +52,7 @@ import { RecordingTipPresenter, createWindowRecordingTipView } from './windows/r import { SuggestionOverlayPresenter, createWindowSuggestionOverlayView } from './windows/suggestion-overlay-presenter' import { ConfigEffects } from './config-effects' import { PopupNavigationController } from './windows/popup-navigation' +import { initScreenOcr } from './screen-ocr-wiring' import { setupLifecycle } from './lifecycle' import { getShutdownRegistry } from './shutdown-registry' import { shouldDismissOnEscape } from './suggestion-overlay-policy' @@ -94,6 +95,7 @@ export async function bootstrap(): Promise { { name: 'auto-launch', critical: false, fn: initAutoLaunch }, { name: 'popup-preload', critical: false, fn: initPopupWindows }, { name: 'key-bindings', critical: false, fn: initKeyBindings }, + { name: 'screen-ocr', critical: false, fn: initScreenOcrStep }, { name: 'input-intelligence', critical: false, fn: initInputIntelligence }, { name: 'voice-mode', critical: false, fn: initVoiceMode }, { name: 'stt-warmup', critical: false, fn: initSTTWarmup }, @@ -196,6 +198,12 @@ async function initIpcHandlers(): Promise { registerAllIpcHandlers() } +/** 화면 OCR — 단축키로 화면을 얼려 덮고 글자를 고른다 (조립은 screen-ocr-wiring). */ +async function initScreenOcrStep(): Promise { + const dispose = initScreenOcr() + getShutdownRegistry().register({ name: 'screen-ocr', dispose }) +} + async function initKeyBindings(): Promise { const keyBindings = getKeyBindingService() keyBindings.loadFromConfig() diff --git a/apps/desktop/src/main/screen-ocr-wiring.ts b/apps/desktop/src/main/screen-ocr-wiring.ts new file mode 100644 index 00000000..4d76638d --- /dev/null +++ b/apps/desktop/src/main/screen-ocr-wiring.ts @@ -0,0 +1,169 @@ +// src/main/screen-ocr-wiring.ts +// +// 화면 OCR 조립 — 서비스에 Electron 어댑터를 꽂고, 단축키 · 오버레이 IPC · Esc 를 잇는다. +// 정책은 ScreenOcrService 에 있다. 여기는 얇은 어댑터만 둔다. + +import { globalShortcut, ipcMain, type IpcMainEvent } from 'electron' +import { IPC_CHANNELS } from '@d3ro/core/ipc-channels' +import { D3ROError, ErrorCode } from '@d3ro/core/errors' +import { getLogger } from './services/LoggerService' +import { getKeyBindingService } from './services/KeyBindingService' +import { initScreenOcrService, type ScreenOcrRequest } from './services/ScreenOcrService' +import { createOverlayPort, createScreenOcrDeps } from './services/screen-ocr/electron-adapters' +import { getPopupTranslator, showResultPopup } from './windows/WindowManager' +import { peekScreenOcrOverlayWindow } from './windows/screen-ocr-overlay' + +const logger = getLogger('ScreenOcrWiring') + +/** 오버레이(Vanilla JS)가 쓰는 문구 — 키 이름은 popups/screen-ocr/script.js 와 짝이다. */ +const OVERLAY_STRING_KEYS = [ + 'modeText', + 'modeRect', + 'copy', + 'copyLine', + 'copyImage', + 'saveImage', + 'savePdf', + 'translate', + 'summarize', + 'speak', + 'stopSpeak', + 'openLink', + 'search', + 'selectAll', + 'close', + 'language', + 'languageAuto', + 'statusRecognizing', + 'statusReady', + 'statusEmpty', + 'statusSelected', + 'hint', + 'toastCopied', + 'toastImageCopied', + 'toastSaved', + 'working', + 'resultTranslate', + 'resultSummary', + 'errorRecognition', + 'errorUnavailable', + 'errorExport', + 'errorLlm', + 'errorTts', + 'errorLink', + 'errorInvalid', + 'errorEmpty' +] as const + +export function screenOcrOverlayStrings(): Record { + const t = getPopupTranslator() + const out: Record = {} + for (const key of OVERLAY_STRING_KEYS) out[key] = t(`screenOcr.${key}`) + return out +} + +function isRecord(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value) +} + +/** 화면 OCR 을 조립한다. 돌려주는 함수는 종료 시 정리한다. */ +export function initScreenOcr(): () => void { + const overlay = createOverlayPort() + + // 오버레이가 포커스를 못 얻은 경우(Windows 전경 잠금)에도 Esc 로 닫히게 열려 있는 동안만 잡는다. + // 저장 대화상자가 떠 있는 동안은 푼다 — 대화상자의 Esc(취소)가 오버레이까지 닫지 않게. + const ESCAPE = 'Escape' + let escapeHeld = false + let overlayOpen = false + let dialogDepth = 0 + const syncEscape = (): void => { + const wanted = overlayOpen && dialogDepth === 0 + if (wanted === escapeHeld) return + if (wanted) { + try { + escapeHeld = globalShortcut.register(ESCAPE, () => service.close()) + } catch (error) { + logger.warn(`Escape capture failed: ${error instanceof Error ? error.message : String(error)}`) + } + } else { + escapeHeld = false + globalShortcut.unregister(ESCAPE) + } + } + + const service = initScreenOcrService( + createScreenOcrDeps( + overlay, + (kind) => getPopupTranslator()(kind === 'pdf' ? 'screenOcr.dialog.savePdf' : 'screenOcr.dialog.saveImage'), + screenOcrOverlayStrings, + { + enter: () => { + dialogDepth += 1 + syncEscape() + }, + leave: () => { + dialogDepth = Math.max(0, dialogDepth - 1) + syncEscape() + } + } + ) + ) + + const start = (): void => { + service.toggle().catch((error: unknown) => { + logger.error(`Screen OCR start failed: ${error instanceof Error ? error.message : String(error)}`) + // 쓸 수 없는 환경 · 캡처 실패를 조용히 삼키지 않는다 — 왜 아무 일도 안 일어났는지 알린다. + const unavailable = error instanceof D3ROError && error.code === ErrorCode.ScreenOcrUnavailable + const t = getPopupTranslator() + showResultPopup(t(unavailable ? 'screenOcr.errorUnavailable' : 'screenOcr.errorCapture')) + }) + } + + const onTriggered = (payload: { type: string; actionId: string }): void => { + if (payload.type === 'pressed' && payload.actionId === 'screen-ocr') start() + } + getKeyBindingService().on('triggered', onTriggered) + + service.on('opened', () => { + overlayOpen = true + syncEscape() + }) + service.on('closed', () => { + overlayOpen = false + syncEscape() + }) + + /** 오버레이 창이 보낸 것만 받는다. */ + const fromOverlay = (event: IpcMainEvent): boolean => { + const win = peekScreenOcrOverlayWindow() + return win !== null && event.sender === win.webContents + } + + const onReady = (event: IpcMainEvent, data: unknown): void => { + if (!fromOverlay(event) || !isRecord(data) || typeof data.sessionId !== 'string') return + overlay.notifyReady(data.sessionId) + } + const onDismiss = (event: IpcMainEvent): void => { + if (fromOverlay(event)) service.close() + } + const onRequest = (event: IpcMainEvent, data: unknown): void => { + if (!fromOverlay(event) || !isRecord(data)) return + const { sessionId, requestId, request } = data + if (typeof sessionId !== 'string' || typeof requestId !== 'number' || !isRecord(request)) return + if (typeof request.type !== 'string') return + // 모양 검증은 서비스가 한다(글 · 영역 · 언어) — 여기서는 봉투만 확인한다. + void service.handleRequest(sessionId, requestId, request as unknown as ScreenOcrRequest) + } + + ipcMain.on(IPC_CHANNELS.POPUP_SCREEN_OCR.READY, onReady) + ipcMain.on(IPC_CHANNELS.POPUP_SCREEN_OCR.DISMISS, onDismiss) + ipcMain.on(IPC_CHANNELS.POPUP_SCREEN_OCR.REQUEST, onRequest) + + return () => { + getKeyBindingService().off('triggered', onTriggered) + ipcMain.removeListener(IPC_CHANNELS.POPUP_SCREEN_OCR.READY, onReady) + ipcMain.removeListener(IPC_CHANNELS.POPUP_SCREEN_OCR.DISMISS, onDismiss) + ipcMain.removeListener(IPC_CHANNELS.POPUP_SCREEN_OCR.REQUEST, onRequest) + service.dispose() + } +} diff --git a/apps/desktop/src/main/services/ConfigService.ts b/apps/desktop/src/main/services/ConfigService.ts index 3e11b441..62530c50 100644 --- a/apps/desktop/src/main/services/ConfigService.ts +++ b/apps/desktop/src/main/services/ConfigService.ts @@ -182,6 +182,7 @@ const CONFIG_DEFAULTS: AppConfig = { agentModeEnabled: false, handsFreeEnabled: false, screenContextEnabled: false, + screenOcrLanguage: 'auto', hfToken: '', diarizationEnabled: false, supabaseUrl: '', diff --git a/apps/desktop/src/main/services/ScreenOcrService.ts b/apps/desktop/src/main/services/ScreenOcrService.ts new file mode 100644 index 00000000..e0ef3155 --- /dev/null +++ b/apps/desktop/src/main/services/ScreenOcrService.ts @@ -0,0 +1,555 @@ +// src/main/services/ScreenOcrService.ts +// +// 화면 OCR — 단축키로 지금 화면을 얼려 덮고, 그 위의 글자를 끌어서 고르게 한다. +// +// 이 서비스가 소유하는 것은 **캡처 한 번의 수명주기와 그 위의 동작 정책**이다: +// 캡처 → 오버레이 표시 → (비동기) 글자 인식 → 오버레이의 요청 처리 → 닫기(정리) +// 선택 계산 · 글 만들기 · PDF 문서 모양은 @d3ro/core/screen-ocr 가 정본이고, 화면 캡처 · OCR 엔진 · +// 창 · 파일 · 클립보드 · LLM · TTS 는 전부 포트로 받는다(Electron 어댑터는 screen-ocr/electron-adapters.ts). +// +// 사생활: 캡처한 그림은 인식에 쓰는 임시 파일 하나로만 디스크에 닿고, 인식이 끝나면 곧바로 지운다. +// 닫으면 메모리의 그림도 놓는다. 어디에도 보내지 않는다(번역·요약은 사용자가 고른 글자만 LLM 으로 간다). + +import { EventEmitter } from 'events' +import { randomUUID } from 'crypto' +import { D3ROError, ErrorCode } from '@d3ro/core/errors' +import { + buildSearchablePdfHtml, + chooseOcrScale, + clampRect, + pickTranslationTarget, + wordsRelativeTo, + type OcrRect, + type OcrResult +} from '@d3ro/core/screen-ocr' +import { getLogger } from './LoggerService' +import type { OcrEnginePort } from './screen-ocr/windows-ocr-engine' + +const logger = getLogger('ScreenOcrService') + +/** LLM 으로 보내는 글의 상한 — 화면 한 장을 통째로 골라도 프롬프트가 터지지 않게. */ +export const SCREEN_OCR_MAX_LLM_CHARS = 12_000 +const LLM_TIMEOUT_MS = 90_000 + +// ============================================================ +// 포트 +// ============================================================ + +/** 캡처한 화면 한 장. */ +export interface ScreenShot { + /** 전체 그림 (PNG) */ + png: Buffer + /** 그림 크기 (물리 픽셀) */ + width: number + height: number + /** 그 화면의 배율 (1 · 1.25 · 1.5 · 2 …) */ + scaleFactor: number + /** 그 화면의 자리 (DIP) — 오버레이 창이 덮을 영역 */ + displayBounds: OcrRect + /** 영역을 잘라 PNG 로 (물리 픽셀 좌표) */ + crop(rect: OcrRect): Buffer +} + +export interface ScreenOcrOverlayPort { + /** 오버레이를 화면에 덮고 새 캡처를 알린다 */ + show(bounds: OcrRect, payload: ScreenOcrSessionPayload): void + sendRecognized(payload: ScreenOcrRecognizedPayload): void + respond(payload: ScreenOcrResponse): void + hide(): void +} + +export interface ScreenOcrFilesPort { + /** 임시 파일로 쓰고 경로를 돌려준다 */ + writeTemp(name: string, data: Buffer | string): Promise + remove(path: string): Promise + /** 저장 위치를 묻는다. 취소하면 null */ + askSavePath(kind: 'pdf' | 'png', defaultName: string): Promise + write(path: string, data: Buffer): Promise +} + +export interface ScreenOcrDeps { + capture: () => Promise + engine: OcrEnginePort + overlay: ScreenOcrOverlayPort + files: ScreenOcrFilesPort + /** HTML 파일 → PDF */ + renderPdf: (htmlPath: string) => Promise + /** Electron 43 부터 글 쓰기가 비동기다 — 끝까지 기다려 실패를 성공으로 꾸미지 않는다 */ + clipboard: { writeText(text: string): Promise; writeImage(png: Buffer): Promise } + llm: { + translate(text: string, targetLanguage: string, signal: AbortSignal, timeoutMs: number): Promise + summarize(text: string, signal: AbortSignal, timeoutMs: number): Promise + } + tts: { speak(text: string): Promise; stop(): void } + openExternal: (url: string) => Promise + config: { + getOcrLanguage(): string + setOcrLanguage(language: string): void + getAppLocale(): string + getTranslationTarget(): string + } + /** 오버레이 문구 (앱 언어) */ + strings: () => Record + now?: () => number +} + +// ============================================================ +// 오버레이와 주고받는 값 +// ============================================================ + +export interface ScreenOcrSessionPayload { + sessionId: string + /** 전체 그림 (PNG) */ + image: Uint8Array + width: number + height: number + /** 고를 수 있는 인식 언어 (설치된 것). 아직 못 읽었으면 비어 있고, 인식 결과와 함께 다시 온다. */ + languages: string[] + /** 지금 고른 인식 언어 ('auto' 포함) */ + language: string + _i18n: Record +} + +export type ScreenOcrRecognizedPayload = + | { sessionId: string; ok: true; result: OcrResult; languages: string[] } + | { sessionId: string; ok: false; error: ScreenOcrErrorKind; languages: string[] } + +/** 오버레이가 문구로 옮기는 실패 종류. */ +export type ScreenOcrErrorKind = + | 'recognition' + | 'unavailable' + | 'export' + | 'llm' + | 'tts' + | 'link' + | 'invalid' + | 'empty' + +export type ScreenOcrRequest = + | { type: 'copy-text'; text: string } + | { type: 'copy-image'; rect: OcrRect | null } + | { type: 'save-image'; rect: OcrRect | null } + | { type: 'save-pdf'; rect: OcrRect | null } + | { type: 'translate'; text: string } + | { type: 'summarize'; text: string } + | { type: 'speak'; text: string } + | { type: 'stop-speak' } + | { type: 'open-link'; url: string } + | { type: 'search-web'; text: string } + | { type: 'set-language'; language: string } + +export type ScreenOcrResponse = + | { requestId: number; ok: true; text?: string; cancelled?: boolean } + | { requestId: number; ok: false; error: ScreenOcrErrorKind } + +interface Session { + id: string + shot: ScreenShot + result: OcrResult | null + /** 닫히면 진행 중인 인식 · LLM 호출을 끊는다 */ + abort: AbortController + /** 인식을 다시 돌릴 때마다 올린다 — 늦게 끝난 옛 인식 결과를 버린다 */ + recognitionRound: number + /** 이 캡처에서 읽어 주기를 시작했는지 — 닫을 때 그 읽기만 멈춘다(다른 기능의 재생은 건드리지 않는다) */ + spoke: boolean +} + +interface ScreenOcrServiceEvents { + opened: (payload: { sessionId: string }) => void + closed: (payload: { sessionId: string }) => void +} + +function isRect(value: unknown): value is OcrRect { + if (typeof value !== 'object' || value === null) return false + const r = value as Record + return [r.x, r.y, r.width, r.height].every((n) => typeof n === 'number' && Number.isFinite(n)) +} + +function fileStamp(now: number): string { + const d = new Date(now) + const pad = (n: number): string => String(n).padStart(2, '0') + return `${d.getFullYear()}${pad(d.getMonth() + 1)}${pad(d.getDate())}-${pad(d.getHours())}${pad(d.getMinutes())}${pad(d.getSeconds())}` +} + +// ============================================================ +// ScreenOcrService +// ============================================================ + +export class ScreenOcrService extends EventEmitter { + private _session: Session | null = null + /** 캡처가 진행 중인 동안 단축키를 다시 눌러도 겹쳐 시작하지 않는다 */ + private _starting = false + private _languages: string[] | null = null + + constructor(private readonly _deps: ScreenOcrDeps) { + super() + } + + get isOpen(): boolean { + return this._session !== null + } + + /** 이 환경에서 화면 OCR 을 쓸 수 있는지 */ + isSupported(): boolean { + return this._deps.engine.isSupported() + } + + /** 단축키 — 열려 있으면 닫고, 닫혀 있으면 새로 캡처한다. */ + async toggle(): Promise { + if (this._session) { + this.close() + return + } + await this.start() + } + + /** + * 화면을 캡처해 오버레이를 띄우고 글자 인식을 시작한다. + * 인식은 기다리지 않는다 — 얼린 화면은 곧바로 보이고 글자는 준비되는 대로 고를 수 있게 된다. + */ + async start(): Promise { + if (this._starting || this._session) return + if (!this.isSupported()) { + throw new D3ROError(ErrorCode.ScreenOcrUnavailable, 'Screen OCR is not available on this platform') + } + this._starting = true + try { + let shot: ScreenShot + try { + shot = await this._deps.capture() + } catch (error) { + throw new D3ROError( + ErrorCode.ScreenOcrCaptureFailed, + `Screen capture failed: ${error instanceof Error ? error.message : String(error)}` + ) + } + + const session: Session = { + id: randomUUID(), + shot, + result: null, + abort: new AbortController(), + recognitionRound: 0, + spoke: false + } + this._session = session + + this._deps.overlay.show(shot.displayBounds, { + sessionId: session.id, + image: shot.png, + width: shot.width, + height: shot.height, + languages: this._languages ?? [], + language: this._deps.config.getOcrLanguage(), + _i18n: this._deps.strings() + }) + this.emit('opened', { sessionId: session.id }) + logger.info(`Screen OCR opened: ${shot.width}x${shot.height} @${shot.scaleFactor}x`) + + void this._recognize(session) + } finally { + this._starting = false + } + } + + /** 오버레이를 닫고 캡처를 놓는다. 진행 중인 인식 · LLM 호출 · 읽기는 끊는다. */ + close(): void { + const session = this._session + if (!session) return + this._session = null + session.abort.abort() + if (session.spoke) this._deps.tts.stop() + this._deps.overlay.hide() + this.emit('closed', { sessionId: session.id }) + logger.info('Screen OCR closed') + } + + /** 오버레이의 요청 하나를 처리하고 응답을 보낸다. 끝난 캡처에 대한 요청은 거절한다. */ + async handleRequest(sessionId: string, requestId: number, request: ScreenOcrRequest): Promise { + const session = this._session + if (!session || session.id !== sessionId) { + this._deps.overlay.respond({ requestId, ok: false, error: 'invalid' }) + return + } + try { + const response = await this._perform(session, request) + // 처리하는 사이 닫혔으면 응답할 곳이 없다. + if (this._session !== session) return + this._deps.overlay.respond({ requestId, ...response }) + } catch (error) { + if (this._session !== session) return + const kind = this._errorKind(request, error) + logger.warn( + `Screen OCR request "${request.type}" failed (${kind}): ${error instanceof Error ? error.message : String(error)}` + ) + this._deps.overlay.respond({ requestId, ok: false, error: kind }) + } + } + + dispose(): void { + this.close() + this.removeAllListeners() + } + + // ── 인식 ────────────────────────────────────────────── + + private async _availableLanguages(): Promise { + if (this._languages) return this._languages + try { + this._languages = await this._deps.engine.listLanguages() + } catch (error) { + // 목록을 못 읽어도 인식 자체는 시도한다(자동 언어). 다음 캡처 때 다시 읽는다. + logger.warn(`OCR language list failed: ${error instanceof Error ? error.message : String(error)}`) + return [] + } + return this._languages + } + + private async _recognize(session: Session): Promise { + const round = ++session.recognitionRound + const isCurrent = (): boolean => this._session === session && session.recognitionRound === round + let tempPath: string | null = null + try { + // 언어 목록은 인식과 나란히 읽는다 — 첫 캡처에서 오버레이가 목록을 기다리지 않게. + const languagesPending = this._availableLanguages() + tempPath = await this._deps.files.writeTemp(`${session.id}.png`, session.shot.png) + const result = await this._deps.engine.recognize({ + imagePath: tempPath, + language: this._deps.config.getOcrLanguage(), + scale: chooseOcrScale( + session.shot.width, + session.shot.height, + session.shot.scaleFactor, + this._deps.engine.maxDimension + ), + signal: session.abort.signal + }) + const languages = await languagesPending + if (!isCurrent()) return + session.result = result + this._deps.overlay.sendRecognized({ sessionId: session.id, ok: true, result, languages }) + logger.info(`Screen OCR recognized ${result.words.length} word(s) in ${result.lines.length} line(s) [${result.language}]`) + } catch (error) { + if (!isCurrent()) return + const unavailable = error instanceof D3ROError && error.code === ErrorCode.ScreenOcrUnavailable + logger.error(`Screen OCR recognition failed: ${error instanceof Error ? error.message : String(error)}`) + this._deps.overlay.sendRecognized({ + sessionId: session.id, + ok: false, + error: unavailable ? 'unavailable' : 'recognition', + languages: this._languages ?? [] + }) + } finally { + if (tempPath) { + await this._deps.files.remove(tempPath).catch((error: unknown) => { + logger.warn(`Screen OCR temp file cleanup failed: ${error instanceof Error ? error.message : String(error)}`) + }) + } + } + } + + // ── 요청 처리 ───────────────────────────────────────── + + private async _perform( + session: Session, + request: ScreenOcrRequest + ): Promise<{ ok: true; text?: string; cancelled?: boolean } | { ok: false; error: ScreenOcrErrorKind }> { + switch (request.type) { + case 'copy-text': { + const text = this._requireText(request.text) + await this._deps.clipboard.writeText(text) + return { ok: true } + } + case 'copy-image': { + await this._deps.clipboard.writeImage(this._imageOf(session, request.rect)) + return { ok: true } + } + case 'save-image': { + const png = this._imageOf(session, request.rect) + const path = await this._deps.files.askSavePath('png', `screen-${fileStamp(this._now())}.png`) + if (!path) return { ok: true, cancelled: true } + await this._deps.files.write(path, png) + return { ok: true } + } + case 'save-pdf': + return this._savePdf(session, request.rect) + case 'translate': { + const text = this._requireText(request.text).slice(0, SCREEN_OCR_MAX_LLM_CHARS) + const target = pickTranslationTarget( + text, + this._deps.config.getAppLocale(), + this._deps.config.getTranslationTarget() + ) + const translated = await this._deps.llm.translate(text, target, session.abort.signal, LLM_TIMEOUT_MS) + return this._llmResult(translated) + } + case 'summarize': { + const text = this._requireText(request.text).slice(0, SCREEN_OCR_MAX_LLM_CHARS) + const summary = await this._deps.llm.summarize(text, session.abort.signal, LLM_TIMEOUT_MS) + return this._llmResult(summary) + } + case 'speak': { + const text = this._requireText(request.text) + if (session.spoke) this._deps.tts.stop() + session.spoke = true + // 읽기는 끝날 때까지 기다리지 않는다 — 시작만 확인하고, 읽는 중 실패는 로그로 남긴다. + void this._deps.tts.speak(text).catch((error: unknown) => { + logger.warn(`Screen OCR read-aloud failed: ${error instanceof Error ? error.message : String(error)}`) + }) + return { ok: true } + } + case 'stop-speak': + this._deps.tts.stop() + return { ok: true } + case 'open-link': { + const opened = await this._deps.openExternal(String(request.url)) + return opened ? { ok: true } : { ok: false, error: 'link' } + } + case 'search-web': { + const query = this._requireText(request.text).replace(/\s+/gu, ' ').slice(0, 300) + const opened = await this._deps.openExternal( + `https://www.google.com/search?q=${encodeURIComponent(query)}` + ) + return opened ? { ok: true } : { ok: false, error: 'link' } + } + case 'set-language': { + const language = String(request.language) + const known = language === 'auto' || (await this._availableLanguages()).includes(language) + if (!known) return { ok: false, error: 'invalid' } + this._deps.config.setOcrLanguage(language) + session.result = null + void this._recognize(session) + return { ok: true } + } + default: + return { ok: false, error: 'invalid' } + } + } + + private async _savePdf( + session: Session, + rect: OcrRect | null + ): Promise<{ ok: true; cancelled?: boolean }> { + const area = this._areaOf(session, rect) + const png = area ? session.shot.crop(area) : session.shot.png + const width = area ? area.width : session.shot.width + const height = area ? area.height : session.shot.height + // 인식이 아직 안 끝났거나 실패했어도 그림만 든 PDF 는 만든다 — 글자 층은 있는 만큼만 넣는다. + const words = session.result + ? wordsRelativeTo(session.result, area ?? { x: 0, y: 0, width, height }) + : [] + + const path = await this._deps.files.askSavePath('pdf', `screen-${fileStamp(this._now())}.pdf`) + if (!path) return { ok: true, cancelled: true } + + const html = buildSearchablePdfHtml({ + imageDataUrl: `data:image/png;base64,${png.toString('base64')}`, + width, + height, + words, + language: session.result?.language ?? '' + }) + const htmlPath = await this._deps.files.writeTemp(`${session.id}-${randomUUID()}.html`, html) + try { + const pdf = await this._deps.renderPdf(htmlPath) + await this._deps.files.write(path, pdf) + } finally { + await this._deps.files.remove(htmlPath).catch(() => { + /* 임시 파일 정리 실패는 저장 결과를 바꾸지 않는다 */ + }) + } + return { ok: true } + } + + private _llmResult(text: string): { ok: true; text: string } | { ok: false; error: ScreenOcrErrorKind } { + const trimmed = text.trim() + // 빈 응답을 성공으로 꾸미지 않는다. + return trimmed.length === 0 ? { ok: false, error: 'llm' } : { ok: true, text: trimmed } + } + + private _requireText(value: unknown): string { + const text = typeof value === 'string' ? value.trim() : '' + if (text.length === 0) { + throw new D3ROError(ErrorCode.ScreenOcrInvalidRequest, 'No text selected') + } + return text + } + + /** 요청한 영역(없으면 화면 전체)을 그림 안으로 자른다. */ + private _areaOf(session: Session, rect: OcrRect | null): OcrRect | null { + if (rect === null || rect === undefined) return null + const clamped = isRect(rect) ? clampRect(rect, { width: session.shot.width, height: session.shot.height }) : null + if (!clamped) { + throw new D3ROError(ErrorCode.ScreenOcrInvalidRequest, 'Selected area is outside the capture') + } + return clamped + } + + private _imageOf(session: Session, rect: OcrRect | null): Buffer { + const area = this._areaOf(session, rect) + return area ? session.shot.crop(area) : session.shot.png + } + + private _errorKind(request: ScreenOcrRequest, error: unknown): ScreenOcrErrorKind { + if (error instanceof D3ROError && error.code === ErrorCode.ScreenOcrInvalidRequest) { + return request.type === 'copy-image' || request.type === 'save-image' || request.type === 'save-pdf' + ? 'invalid' + : 'empty' + } + switch (request.type) { + case 'translate': + case 'summarize': + return 'llm' + case 'speak': + case 'stop-speak': + return 'tts' + case 'open-link': + case 'search-web': + return 'link' + case 'set-language': + return 'recognition' + default: + return 'export' + } + } + + private _now(): number { + return this._deps.now ? this._deps.now() : Date.now() + } + + // ── EventEmitter 타입 오버라이드 ─────────────────────── + + override on(event: K, listener: ScreenOcrServiceEvents[K]): this { + return super.on(event, listener) + } + + override emit( + event: K, + ...args: Parameters + ): boolean { + return super.emit(event, ...args) + } +} + +// ── 싱글톤 ──────────────────────────────────────────── + +let instance: ScreenOcrService | null = null + +/** 앱 조립부(bootstrap)가 Electron 어댑터를 넣어 한 번 만든다. */ +export function initScreenOcrService(deps: ScreenOcrDeps): ScreenOcrService { + instance?.dispose() + instance = new ScreenOcrService(deps) + return instance +} + +export function getScreenOcrService(): ScreenOcrService { + if (!instance) { + throw new D3ROError(ErrorCode.ScreenOcrUnavailable, 'ScreenOcrService is not initialized') + } + return instance +} + +export function resetScreenOcrServiceForTests(): void { + instance?.dispose() + instance = null +} diff --git a/apps/desktop/src/main/services/keybinding-accelerator-policy.ts b/apps/desktop/src/main/services/keybinding-accelerator-policy.ts index 9b214a74..e4d1a7a0 100644 --- a/apps/desktop/src/main/services/keybinding-accelerator-policy.ts +++ b/apps/desktop/src/main/services/keybinding-accelerator-policy.ts @@ -24,9 +24,17 @@ const SUGGESTION_ACTIONS: ReadonlySet = new Set = new Set(['screen-ocr']) + /** 액션 활성 판정에 필요한 기능 스위치 */ export interface KeyBindingFeatureFlags { suggestionEnabled: boolean + /** 이 환경에서 화면 OCR 을 쓸 수 있는가 (생략하면 쓸 수 없는 것으로 본다) */ + screenOcrAvailable?: boolean } /** 액션이 쓰이는 기능이 켜져 있는가. 기능 스위치가 없는 액션은 항상 켜져 있다. */ @@ -35,6 +43,7 @@ export function isKeyBindingActionFeatureEnabled( flags: KeyBindingFeatureFlags ): boolean { if (SUGGESTION_ACTIONS.has(actionId)) return flags.suggestionEnabled + if (SCREEN_OCR_ACTIONS.has(actionId)) return flags.screenOcrAvailable === true return true } diff --git a/apps/desktop/src/main/services/screen-ocr/electron-adapters.ts b/apps/desktop/src/main/services/screen-ocr/electron-adapters.ts new file mode 100644 index 00000000..6d557235 --- /dev/null +++ b/apps/desktop/src/main/services/screen-ocr/electron-adapters.ts @@ -0,0 +1,246 @@ +// src/main/services/screen-ocr/electron-adapters.ts +// +// ScreenOcrService 포트의 Electron 어댑터 — 화면 캡처 · 파일 · PDF · 클립보드 · LLM · TTS. +// 정책은 서비스(ScreenOcrService)와 core(@d3ro/core/screen-ocr)에 있고, 여기는 OS/Electron 호출만 한다. + +import { promises as fs } from 'fs' +import { join } from 'path' +import { app, BrowserWindow, clipboard, desktopCapturer, dialog, nativeImage, screen } from 'electron' +import { IPC_CHANNELS } from '@d3ro/core/ipc-channels' +import type { OcrRect } from '@d3ro/core/screen-ocr' +import { configGet, configSet } from '../ConfigService' +import { getLlmGateway } from '../llm/LlmGateway' +import { getTTSPlaybackService } from '../TTSPlaybackService' +import { getLogger } from '../LoggerService' +import { openExternalSafe } from '../../windows/web-contents-hardening' +import { captureScreenRegion } from '../../utils/win32-screen-capture' +import { + hideScreenOcrOverlay, + peekScreenOcrOverlayWindow, + getScreenOcrOverlayWindow, + sendToScreenOcrOverlay, + showScreenOcrOverlay +} from '../../windows/screen-ocr-overlay' +import type { + ScreenOcrDeps, + ScreenOcrFilesPort, + ScreenOcrOverlayPort, + ScreenShot +} from '../ScreenOcrService' +import { createWindowsOcrEngine } from './windows-ocr-engine' + +const logger = getLogger('ScreenOcrAdapters') + +/** 오버레이가 그림을 다 그렸다고 알리기를 기다리는 최대 시간 — 넘으면 그냥 보인다. */ +const OVERLAY_READY_TIMEOUT_MS = 500 + +// ── 화면 캡처 ───────────────────────────────────────── + +/** + * 커서가 있는 모니터를 물리 해상도로 캡처한다. + * Windows 는 GDI 로 그 모니터 한 장만 읽고(수십 ms), 안 되면 desktopCapturer 로 넘어간다. + */ +export async function captureDisplayAtCursor(): Promise { + const display = screen.getDisplayNearestPoint(screen.getCursorScreenPoint()) + const image = captureWithGdi(display) ?? (await captureWithDesktopCapturer(display)) + const size = image.getSize() + return { + png: image.toPNG(), + width: size.width, + height: size.height, + scaleFactor: display.scaleFactor, + displayBounds: { ...display.bounds }, + crop: (rect: OcrRect): Buffer => image.crop(rect).toPNG() + } +} + +function captureWithGdi(display: Electron.Display): Electron.NativeImage | null { + if (process.platform !== 'win32') return null + // 모니터 영역(DIP) → 가상 화면의 물리 픽셀. 배율이 섞인 다중 모니터에서도 OS 가 준 값을 그대로 쓴다. + const pixels = captureScreenRegion(screen.dipToScreenRect(null, display.bounds)) + if (!pixels) return null + const image = nativeImage.createFromBitmap(pixels.bgra, { width: pixels.width, height: pixels.height }) + return image.isEmpty() ? null : image +} + +async function captureWithDesktopCapturer(display: Electron.Display): Promise { + const width = Math.round(display.size.width * display.scaleFactor) + const height = Math.round(display.size.height * display.scaleFactor) + + const sources = await desktopCapturer.getSources({ types: ['screen'], thumbnailSize: { width, height } }) + // display_id 가 비는 환경(드라이버·원격 세션)이 있다 — 모니터가 하나뿐이면 그 하나를 쓴다. + const source = + sources.find((candidate) => candidate.display_id === String(display.id)) ?? + (sources.length === 1 ? sources[0] : undefined) + if (!source || source.thumbnail.isEmpty()) { + throw new Error(`no capture source for display ${display.id} (${sources.length} source(s))`) + } + return source.thumbnail +} + +// ── 오버레이 창 ─────────────────────────────────────── + +/** + * 오버레이 표시는 2단계다: 숨긴 창에 그림을 먼저 보내고, 다 그렸다는 신호(READY)가 오면 보인다. + * 곧바로 보이면 이전 캡처나 빈 바탕이 한 프레임 비친다. + */ +export function createOverlayPort(): ScreenOcrOverlayPort & { notifyReady(sessionId: string): void } { + let pending: { sessionId: string; bounds: Electron.Rectangle; timer: NodeJS.Timeout } | null = null + + const reveal = (): void => { + if (!pending) return + clearTimeout(pending.timer) + const { bounds } = pending + pending = null + showScreenOcrOverlay(bounds) + } + + return { + show(bounds, payload) { + if (pending) clearTimeout(pending.timer) + const win = getScreenOcrOverlayWindow() + // 숨긴 채로 크기를 먼저 맞춘다 — 렌더러가 보이기 전에 최종 크기로 배치를 끝내게. + win.setBounds(bounds) + pending = { + sessionId: payload.sessionId, + bounds, + timer: setTimeout(() => { + logger.warn('Screen OCR overlay did not report ready in time — showing anyway') + reveal() + }, OVERLAY_READY_TIMEOUT_MS) + } + sendToScreenOcrOverlay(IPC_CHANNELS.POPUP_SCREEN_OCR.SESSION, payload) + }, + notifyReady(sessionId) { + if (pending?.sessionId === sessionId) reveal() + }, + sendRecognized(payload) { + sendToScreenOcrOverlay(IPC_CHANNELS.POPUP_SCREEN_OCR.RECOGNIZED, payload) + }, + respond(payload) { + sendToScreenOcrOverlay(IPC_CHANNELS.POPUP_SCREEN_OCR.RESPONSE, payload) + }, + hide() { + if (pending) { + clearTimeout(pending.timer) + pending = null + } + // 그림을 놓게 한다(닫힌 뒤에도 캡처가 렌더러 메모리에 남지 않게). + sendToScreenOcrOverlay(IPC_CHANNELS.POPUP_SCREEN_OCR.SESSION, null) + hideScreenOcrOverlay() + } + } +} + +// ── 파일 ────────────────────────────────────────────── + +function tempDir(): string { + return join(app.getPath('temp'), 'd3ro-voice-screen-ocr') +} + +/** 저장 대화상자가 떠 있는 동안을 알린다 — 그동안은 전역 Esc 잡기를 풀어 대화상자의 Esc 가 살아 있게 한다. */ +export interface DialogGuard { + enter(): void + leave(): void +} + +export function createFilesPort(dialogTitle: (kind: 'pdf' | 'png') => string, guard: DialogGuard): ScreenOcrFilesPort { + return { + async writeTemp(name, data) { + const dir = tempDir() + await fs.mkdir(dir, { recursive: true }) + const path = join(dir, name) + await fs.writeFile(path, data) + return path + }, + async remove(path) { + await fs.rm(path, { force: true }) + }, + async askSavePath(kind, defaultName) { + const options: Electron.SaveDialogOptions = { + title: dialogTitle(kind), + defaultPath: join(app.getPath('documents'), defaultName), + filters: [kind === 'pdf' ? { name: 'PDF', extensions: ['pdf'] } : { name: 'PNG', extensions: ['png'] }] + } + // 오버레이가 최상위 단계라 부모 없이 띄우면 대화상자가 그 뒤에 깔린다. + const parent = peekScreenOcrOverlayWindow() + guard.enter() + try { + const { canceled, filePath } = parent + ? await dialog.showSaveDialog(parent, options) + : await dialog.showSaveDialog(options) + return canceled || !filePath ? null : filePath + } finally { + guard.leave() + } + }, + async write(path, data) { + await fs.writeFile(path, data) + } + } +} + +/** 임시 HTML 한 쪽을 숨긴 창에서 PDF 로 찍는다 (쪽 크기는 문서의 @page 를 따른다). */ +export async function renderHtmlFileToPdf(htmlPath: string): Promise { + const win = new BrowserWindow({ + show: false, + width: 800, + height: 600, + webPreferences: { javascript: false, sandbox: true, contextIsolation: true, nodeIntegration: false } + }) + try { + win.webContents.on('will-navigate', (event) => event.preventDefault()) + win.webContents.setWindowOpenHandler(() => ({ action: 'deny' })) + await win.loadFile(htmlPath) + return await win.webContents.printToPDF({ + printBackground: true, + preferCSSPageSize: true, + margins: { top: 0, bottom: 0, left: 0, right: 0 } + }) + } finally { + win.destroy() + } +} + +// ── 조립 ────────────────────────────────────────────── + +export function createScreenOcrDeps( + overlay: ScreenOcrOverlayPort, + dialogTitle: (kind: 'pdf' | 'png') => string, + strings: () => Record, + guard: DialogGuard +): ScreenOcrDeps { + return { + capture: captureDisplayAtCursor, + engine: createWindowsOcrEngine(), + overlay, + files: createFilesPort(dialogTitle, guard), + renderPdf: renderHtmlFileToPdf, + clipboard: { + writeText: async (text) => { + await clipboard.writeText(text) + }, + writeImage: async (png) => { + await clipboard.writeImage(nativeImage.createFromBuffer(png)) + } + }, + llm: { + translate: (text, targetLanguage, signal, timeoutMs) => + getLlmGateway().processText(text, 'translate', targetLanguage, undefined, { signal, timeoutMs }), + summarize: (text, signal, timeoutMs) => + getLlmGateway().processText(text, 'summarize', undefined, undefined, { signal, timeoutMs }) + }, + tts: { + speak: (text) => getTTSPlaybackService().speak(text), + stop: () => getTTSPlaybackService().stop() + }, + openExternal: (url) => openExternalSafe(url), + config: { + getOcrLanguage: () => configGet('screenOcrLanguage') || 'auto', + setOcrLanguage: (language) => configSet('screenOcrLanguage', language), + getAppLocale: () => (configGet('language') as string) || 'ko', + getTranslationTarget: () => configGet('targetLanguage') || 'English' + }, + strings + } +} diff --git a/apps/desktop/src/main/services/screen-ocr/windows-ocr-engine.ts b/apps/desktop/src/main/services/screen-ocr/windows-ocr-engine.ts new file mode 100644 index 00000000..fac6da15 --- /dev/null +++ b/apps/desktop/src/main/services/screen-ocr/windows-ocr-engine.ts @@ -0,0 +1,267 @@ +// src/main/services/screen-ocr/windows-ocr-engine.ts +// +// Windows 내장 OCR(Windows.Media.Ocr) 어댑터 — PowerToys Text Extractor 와 같은 엔진이다. +// +// 왜 이 엔진인가 (2026-11-20 조사): +// - 설치본에 모델·네이티브 모듈을 더 싣지 않는다(이 저장소는 install script 를 허용하지 않고, +// ffmpeg 처럼 큰 바이너리는 런타임에 내려받는다). +// - 화면 글자 기준 2배 확대 입력에서 한국어·영어 혼합 줄을 0.2~0.3초에 읽는다. +// - WinRT 는 Windows PowerShell 5.1(powershell.exe — 모든 Windows 에 있다)에서 바로 부를 수 있다. +// PowerShell 7(pwsh)은 WinRT 투영이 빠져 쓸 수 없다. +// +// 스크립트는 파일로 쓰지 않고 -EncodedCommand 로 넘긴다 — 스크립트 파일 실행 정책에 걸리지 않고 +// 임시 파일도 남지 않는다. 입력은 환경 변수로만 준다(명령줄에 경로를 이어 붙이지 않는다). + +import { execFile } from 'child_process' +import { join } from 'path' +import { D3ROError, ErrorCode } from '@d3ro/core/errors' +import { parseOcrEngineOutput, type OcrResult } from '@d3ro/core/screen-ocr' + +/** Windows.Media.Ocr.OcrEngine.MaxImageDimension — 이 변 길이를 넘는 그림은 인식하지 못한다. */ +export const WINDOWS_OCR_MAX_DIMENSION = 10000 + +const ENGINE_TIMEOUT_MS = 30_000 +const MAX_OUTPUT_BYTES = 32 * 1024 * 1024 + +export interface OcrRecognizeRequest { + /** 인식할 PNG 파일의 절대 경로 */ + imagePath: string + /** 언어 태그. 'auto' 면 Windows 사용자 언어를 따른다. */ + language: string + /** 인식 전 확대 배율 (좌표는 원본 크기 기준으로 되돌려 준다) */ + scale: number + signal?: AbortSignal +} + +/** OCR 엔진 포트 — 서비스는 이 모양에만 의존한다. */ +export interface OcrEnginePort { + /** 이 환경에서 엔진을 쓸 수 있는지 */ + isSupported(): boolean + /** 그림 한 변의 최대 길이 */ + readonly maxDimension: number + /** 설치된 인식 언어 태그 목록 */ + listLanguages(): Promise + recognize(request: OcrRecognizeRequest): Promise +} + +// 스크립트 본문. 결과는 항상 한 줄 JSON 으로 표준 출력에 쓴다. +// D3RO_OCR_MODE=languages → { languages: [...] } +// 그 외 → { lang, lines: [{ words: [{ t, x, y, w, h }] }] } 또는 { error } +const OCR_SCRIPT = String.raw` +$ErrorActionPreference = 'Stop' +[Console]::OutputEncoding = [System.Text.Encoding]::UTF8 +try { + Add-Type -AssemblyName System.Runtime.WindowsRuntime + $null = [Windows.Media.Ocr.OcrEngine, Windows.Foundation, ContentType = WindowsRuntime] + $null = [Windows.Graphics.Imaging.BitmapDecoder, Windows.Foundation, ContentType = WindowsRuntime] + $null = [Windows.Storage.StorageFile, Windows.Storage, ContentType = WindowsRuntime] + $null = [Windows.Globalization.Language, Windows.Foundation, ContentType = WindowsRuntime] + + if ($env:D3RO_OCR_MODE -eq 'languages') { + $tags = @([Windows.Media.Ocr.OcrEngine]::AvailableRecognizerLanguages | ForEach-Object { $_.LanguageTag }) + [pscustomobject]@{ languages = $tags } | ConvertTo-Json -Depth 3 -Compress + exit 0 + } + + $asTask = [System.WindowsRuntimeSystemExtensions].GetMethods() | Where-Object { + $_.Name -eq 'AsTask' -and $_.GetParameters().Count -eq 1 -and $_.GetParameters()[0].ParameterType.Name -eq 'IAsyncOperation` + '`' + String.raw`1' + } | Select-Object -First 1 + function Await($operation, [Type]$type) { + $task = $asTask.MakeGenericMethod($type).Invoke($null, @($operation)) + $task.Wait() + $task.Result + } + + $scale = [double]::Parse($env:D3RO_OCR_SCALE, [System.Globalization.CultureInfo]::InvariantCulture) + $file = Await ([Windows.Storage.StorageFile]::GetFileFromPathAsync($env:D3RO_OCR_IMAGE)) ([Windows.Storage.StorageFile]) + $stream = Await ($file.OpenAsync([Windows.Storage.FileAccessMode]::Read)) ([Windows.Storage.Streams.IRandomAccessStream]) + try { + $decoder = Await ([Windows.Graphics.Imaging.BitmapDecoder]::CreateAsync($stream)) ([Windows.Graphics.Imaging.BitmapDecoder]) + $transform = New-Object Windows.Graphics.Imaging.BitmapTransform + $transform.ScaledWidth = [uint32][math]::Round($decoder.PixelWidth * $scale) + $transform.ScaledHeight = [uint32][math]::Round($decoder.PixelHeight * $scale) + $transform.InterpolationMode = [Windows.Graphics.Imaging.BitmapInterpolationMode]::Fant + $bitmap = Await ($decoder.GetSoftwareBitmapAsync( + [Windows.Graphics.Imaging.BitmapPixelFormat]::Bgra8, + [Windows.Graphics.Imaging.BitmapAlphaMode]::Premultiplied, + $transform, + [Windows.Graphics.Imaging.ExifOrientationMode]::IgnoreExifOrientation, + [Windows.Graphics.Imaging.ColorManagementMode]::DoNotColorManage)) ([Windows.Graphics.Imaging.SoftwareBitmap]) + } finally { + $stream.Dispose() + } + + $engine = $null + if ($env:D3RO_OCR_LANG -and $env:D3RO_OCR_LANG -ne 'auto') { + $engine = [Windows.Media.Ocr.OcrEngine]::TryCreateFromLanguage((New-Object Windows.Globalization.Language $env:D3RO_OCR_LANG)) + } + if (-not $engine) { $engine = [Windows.Media.Ocr.OcrEngine]::TryCreateFromUserProfileLanguages() } + if (-not $engine) { + $first = [Windows.Media.Ocr.OcrEngine]::AvailableRecognizerLanguages | Select-Object -First 1 + if ($first) { $engine = [Windows.Media.Ocr.OcrEngine]::TryCreateFromLanguage($first) } + } + if (-not $engine) { + '{"error":"no-language"}' + exit 0 + } + + $result = Await ($engine.RecognizeAsync($bitmap)) ([Windows.Media.Ocr.OcrResult]) + $lines = foreach ($line in $result.Lines) { + [pscustomobject]@{ + words = @(foreach ($word in $line.Words) { + $r = $word.BoundingRect + [pscustomobject]@{ + t = $word.Text + x = [math]::Round($r.X / $scale, 2) + y = [math]::Round($r.Y / $scale, 2) + w = [math]::Round($r.Width / $scale, 2) + h = [math]::Round($r.Height / $scale, 2) + } + }) + } + } + [pscustomobject]@{ lang = $engine.RecognizerLanguage.LanguageTag; lines = @($lines) } | ConvertTo-Json -Depth 6 -Compress +} catch { + [pscustomobject]@{ error = 'exception'; message = "$($_.Exception.Message)" } | ConvertTo-Json -Compress + exit 0 +} +` + +function encodeCommand(script: string): string { + return Buffer.from(script, 'utf16le').toString('base64') +} + +function powershellPath(): string { + const root = process.env['SystemRoot'] ?? process.env['windir'] ?? 'C:\\Windows' + return join(root, 'System32', 'WindowsPowerShell', 'v1.0', 'powershell.exe') +} + +/** 엔진 프로세스를 한 번 돌려 표준 출력의 JSON 을 돌려준다. */ +export type OcrProcessRunner = (env: Record, signal?: AbortSignal) => Promise + +const runPowerShell: OcrProcessRunner = (env, signal) => + new Promise((resolve, reject) => { + execFile( + powershellPath(), + ['-NoLogo', '-NoProfile', '-NonInteractive', '-EncodedCommand', encodeCommand(OCR_SCRIPT)], + { + env: { ...process.env, ...env }, + encoding: 'utf8', + timeout: ENGINE_TIMEOUT_MS, + maxBuffer: MAX_OUTPUT_BYTES, + windowsHide: true, + signal + }, + (error, stdout) => { + if (error) { + reject(error) + return + } + resolve(stdout) + } + ) + }) + +/** 표준 출력에서 JSON 한 덩어리를 꺼낸다 (앞뒤에 경고 줄이 섞여도 견딘다). */ +function parseJsonOutput(stdout: string): unknown { + const start = stdout.indexOf('{') + const end = stdout.lastIndexOf('}') + if (start === -1 || end <= start) return null + try { + return JSON.parse(stdout.slice(start, end + 1)) as unknown + } catch { + return null + } +} + +function isRecord(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value) +} + +export interface WindowsOcrEngineDeps { + platform?: NodeJS.Platform + run?: OcrProcessRunner +} + +export function createWindowsOcrEngine(deps: WindowsOcrEngineDeps = {}): OcrEnginePort { + const platform = deps.platform ?? process.platform + const run = deps.run ?? runPowerShell + + const invoke = async (env: Record, signal?: AbortSignal): Promise> => { + let stdout: string + try { + stdout = await run(env, signal) + } catch (error) { + if (signal?.aborted) throw error + throw new D3ROError( + ErrorCode.ScreenOcrRecognitionFailed, + `OCR engine process failed: ${error instanceof Error ? error.message : String(error)}` + ) + } + const parsed = parseJsonOutput(stdout) + if (!isRecord(parsed)) { + throw new D3ROError(ErrorCode.ScreenOcrRecognitionFailed, 'OCR engine returned no parsable output') + } + if (parsed.error === 'no-language') { + throw new D3ROError(ErrorCode.ScreenOcrUnavailable, 'No Windows OCR language is installed') + } + if (typeof parsed.error === 'string') { + throw new D3ROError( + ErrorCode.ScreenOcrRecognitionFailed, + `OCR engine error: ${typeof parsed.message === 'string' ? parsed.message : parsed.error}` + ) + } + return parsed + } + + return { + maxDimension: WINDOWS_OCR_MAX_DIMENSION, + + isSupported: () => platform === 'win32', + + async listLanguages(): Promise { + if (platform !== 'win32') return [] + const parsed = await invoke({ D3RO_OCR_MODE: 'languages' }) + const languages = parsed.languages + // 언어가 하나뿐이면 ConvertTo-Json 이 배열이 아니라 문자열로 낸다. + if (typeof languages === 'string') return [languages] + if (!Array.isArray(languages)) return [] + return languages.filter((tag): tag is string => typeof tag === 'string' && tag.length > 0) + }, + + async recognize(request: OcrRecognizeRequest): Promise { + if (platform !== 'win32') { + throw new D3ROError(ErrorCode.ScreenOcrUnavailable, `Screen OCR is not available on ${platform}`) + } + const parsed = await invoke( + { + D3RO_OCR_MODE: 'recognize', + D3RO_OCR_IMAGE: request.imagePath, + D3RO_OCR_LANG: request.language, + D3RO_OCR_SCALE: String(request.scale) + }, + request.signal + ) + // 줄이 하나뿐이면 ConvertTo-Json 이 lines 를 배열이 아닌 객체로, 단어가 하나뿐이면 words 를 객체로 낸다. + const result = parseOcrEngineOutput(normalizeSingletons(parsed)) + if (!result) { + throw new D3ROError(ErrorCode.ScreenOcrRecognitionFailed, 'OCR engine output has an unexpected shape') + } + return result + } + } +} + +/** PowerShell 5.1 의 ConvertTo-Json 은 원소가 하나인 배열을 풀어 버린다 — 다시 배열로 싼다. */ +function normalizeSingletons(parsed: Record): Record { + const rawLines = parsed.lines + const lines = Array.isArray(rawLines) ? rawLines : isRecord(rawLines) ? [rawLines] : [] + return { + ...parsed, + lines: lines.map((line: unknown) => { + if (!isRecord(line)) return line + const words = line.words + return { ...line, words: Array.isArray(words) ? words : isRecord(words) ? [words] : [] } + }) + } +} diff --git a/apps/desktop/src/main/utils/win32-screen-capture.ts b/apps/desktop/src/main/utils/win32-screen-capture.ts new file mode 100644 index 00000000..140579dd --- /dev/null +++ b/apps/desktop/src/main/utils/win32-screen-capture.ts @@ -0,0 +1,171 @@ +// src/main/utils/win32-screen-capture.ts +// +// koffi(FFI) + GDI BitBlt 로 화면의 한 영역을 물리 픽셀 그대로 읽는다. +// +// 왜 desktopCapturer 가 아닌가 (2026-11-20 실측, 모니터 3대): +// - desktopCapturer.getSources 는 **모든** 모니터를 캡처하고, DXGI 복제가 프레임을 못 받으면 +// 모니터마다 500ms 를 기다린 뒤 GDI 로 넘어간다 — 한 번에 2.9초가 걸렸다. +// - BitBlt 는 필요한 모니터 한 장만 39ms 에 읽고, 결과 그림은 바이트까지 같았다. +// 단축키를 누른 순간의 화면을 얼려야 하는 화면 OCR 에는 이 차이가 결정적이다. +// +// 실패(비 Windows, 모듈 로드 실패, GDI 호출 실패)는 예외 대신 null 로 돌려준다 — +// 호출자는 desktopCapturer 로 넘어간다. + +import { getLogger } from '../services/LoggerService' + +const logger = getLogger('win32-screen-capture') + +type AnyFunc = (...args: unknown[]) => unknown + +interface GdiApi { + GetDC: AnyFunc + ReleaseDC: AnyFunc + CreateCompatibleDC: AnyFunc + CreateCompatibleBitmap: AnyFunc + SelectObject: AnyFunc + BitBlt: AnyFunc + GetDIBits: AnyFunc + DeleteObject: AnyFunc + DeleteDC: AnyFunc +} + +const SRCCOPY = 0x00cc0020 +/** 레이어드(반투명) 창까지 함께 읽는다 */ +const CAPTUREBLT = 0x40000000 +const BI_RGB = 0 +const DIB_RGB_COLORS = 0 +/** 한 변이 이보다 크면 요청이 잘못된 것으로 본다 (8K 가로 = 7680) */ +const MAX_DIMENSION = 16384 + +let api: GdiApi | null = null +let loadFailed = false + +function loadApi(): GdiApi | null { + if (api) return api + if (loadFailed) return null + if (process.platform !== 'win32') { + loadFailed = true + return null + } + try { + // eslint-disable-next-line @typescript-eslint/no-var-requires + const koffi = require('koffi') as { + load: (lib: string) => { func: (signature: string) => AnyFunc } + struct: (name: string, fields: Record) => unknown + } + const user32 = koffi.load('user32.dll') + const gdi32 = koffi.load('gdi32.dll') + koffi.struct('D3RO_BITMAPINFOHEADER', { + biSize: 'uint32_t', + biWidth: 'int32_t', + biHeight: 'int32_t', + biPlanes: 'uint16_t', + biBitCount: 'uint16_t', + biCompression: 'uint32_t', + biSizeImage: 'uint32_t', + biXPelsPerMeter: 'int32_t', + biYPelsPerMeter: 'int32_t', + biClrUsed: 'uint32_t', + biClrImportant: 'uint32_t' + }) + api = { + GetDC: user32.func('void *GetDC(void *hWnd)'), + ReleaseDC: user32.func('int ReleaseDC(void *hWnd, void *hDC)'), + CreateCompatibleDC: gdi32.func('void *CreateCompatibleDC(void *hdc)'), + CreateCompatibleBitmap: gdi32.func('void *CreateCompatibleBitmap(void *hdc, int cx, int cy)'), + SelectObject: gdi32.func('void *SelectObject(void *hdc, void *h)'), + BitBlt: gdi32.func( + 'bool BitBlt(void *hdc, int x, int y, int cx, int cy, void *hdcSrc, int x1, int y1, uint32_t rop)' + ), + GetDIBits: gdi32.func( + 'int GetDIBits(void *hdc, void *hbm, uint32_t start, uint32_t cLines, _Out_ uint8_t *lpvBits, _Inout_ D3RO_BITMAPINFOHEADER *lpbmi, uint32_t usage)' + ), + DeleteObject: gdi32.func('bool DeleteObject(void *ho)'), + DeleteDC: gdi32.func('bool DeleteDC(void *hdc)') + } + return api + } catch (error) { + loadFailed = true + logger.warn( + `koffi/GDI 사용 불가 — 화면 캡처는 desktopCapturer 로 넘어간다 (${ + error instanceof Error ? error.message : String(error) + })` + ) + return null + } +} + +export interface ScreenPixels { + /** BGRA, 위에서 아래로, 알파는 불투명(255)으로 채움 */ + bgra: Buffer + width: number + height: number +} + +/** + * 가상 화면의 한 영역을 읽는다 (좌표·크기는 **물리 픽셀**, 가상 화면 원점 기준). + * Windows 가 아니거나 캡처에 실패하면 null. + */ +export function captureScreenRegion(rect: { x: number; y: number; width: number; height: number }): ScreenPixels | null { + const a = loadApi() + if (!a) return null + const width = Math.round(rect.width) + const height = Math.round(rect.height) + if (width < 1 || height < 1 || width > MAX_DIMENSION || height > MAX_DIMENSION) return null + + let screenDc: unknown = null + let memoryDc: unknown = null + let bitmap: unknown = null + try { + screenDc = a.GetDC(null) + if (!screenDc) return null + memoryDc = a.CreateCompatibleDC(screenDc) + bitmap = memoryDc ? a.CreateCompatibleBitmap(screenDc, width, height) : null + if (!memoryDc || !bitmap) return null + + const previous = a.SelectObject(memoryDc, bitmap) + const copied = a.BitBlt( + memoryDc, + 0, + 0, + width, + height, + screenDc, + Math.round(rect.x), + Math.round(rect.y), + SRCCOPY | CAPTUREBLT + ) as boolean + // GetDIBits 는 비트맵이 DC 에 선택돼 있지 않아야 한다. + a.SelectObject(memoryDc, previous) + if (!copied) return null + + const bgra = Buffer.alloc(width * height * 4) + const header = { + biSize: 40, + biWidth: width, + // 음수 높이 = 위에서 아래로(top-down) + biHeight: -height, + biPlanes: 1, + biBitCount: 32, + biCompression: BI_RGB, + biSizeImage: 0, + biXPelsPerMeter: 0, + biYPelsPerMeter: 0, + biClrUsed: 0, + biClrImportant: 0 + } + const lines = Number(a.GetDIBits(memoryDc, bitmap, 0, height, bgra, header, DIB_RGB_COLORS)) + if (lines !== height) return null + + // GDI 는 알파 채널을 0 으로 둔다 — 그대로 그림으로 만들면 완전히 투명해진다. + for (let i = 3; i < bgra.length; i += 4) bgra[i] = 255 + return { bgra, width, height } + } catch (error) { + logger.warn(`GDI 화면 캡처 실패: ${error instanceof Error ? error.message : String(error)}`) + return null + } finally { + if (bitmap) a.DeleteObject(bitmap) + if (memoryDc) a.DeleteDC(memoryDc) + if (screenDc) a.ReleaseDC(null, screenDc) + } +} diff --git a/apps/desktop/src/main/windows/WindowManager.ts b/apps/desktop/src/main/windows/WindowManager.ts index 9371132d..c9f3ffb0 100644 --- a/apps/desktop/src/main/windows/WindowManager.ts +++ b/apps/desktop/src/main/windows/WindowManager.ts @@ -38,6 +38,7 @@ import { sendToPopupWindow } from './popup-window-factory' import { createElectronScreenGeometry } from './screen-geometry' +import { peekScreenOcrOverlayWindow } from './screen-ocr-overlay' import { installApplicationMenu } from './app-menu' import { launchedAtLogin } from '../services/AutoLaunchService' import { @@ -1061,6 +1062,7 @@ export function reapplyThemeToAllPopups(): void { commandPopupWindow, captionOverlayWindow, suggestionOverlayWindow, + peekScreenOcrOverlayWindow(), ] for (const win of popupWindows) { if (win && !win.isDestroyed()) { diff --git a/apps/desktop/src/main/windows/popup-window-factory.ts b/apps/desktop/src/main/windows/popup-window-factory.ts index 9e985f7f..e22d7cc8 100644 --- a/apps/desktop/src/main/windows/popup-window-factory.ts +++ b/apps/desktop/src/main/windows/popup-window-factory.ts @@ -31,6 +31,16 @@ export interface PopupWindowSpec { * 항상 보이는 창(자막 오버레이)만 true 로 둔다. */ backgroundThrottling?: boolean + /** + * 키보드 포커스를 받을지. 기본 false — 팝업은 쓰던 앱의 포커스를 빼앗지 않는다. + * 화면 OCR 오버레이만 true 다(화면을 덮고 Esc · Ctrl+C 를 직접 받는다). + */ + focusable?: boolean + /** + * 투명 창인지. 기본 true. 화면 전체를 그림으로 덮는 창은 false 로 둔다 — + * 투명 창은 첫 그리기 전에 뒤 화면이 비쳐 깜빡인다. + */ + transparent?: boolean /** 로드 전에 적용할 창 설정 (클릭 통과 등) */ beforeLoad?: (win: BrowserWindow) => void /** 창이 닫혔다 — 소유자가 참조를 지운다 */ @@ -46,11 +56,11 @@ export function buildPopupWindowOptions( ...spec.bounds, show: false, frame: false, - transparent: true, + transparent: spec.transparent ?? true, resizable: false, alwaysOnTop: true, skipTaskbar: true, - focusable: false, + focusable: spec.focusable ?? false, webPreferences: { preload: preloadPath, sandbox: false, diff --git a/apps/desktop/src/main/windows/screen-ocr-overlay.ts b/apps/desktop/src/main/windows/screen-ocr-overlay.ts new file mode 100644 index 00000000..71fc7a14 --- /dev/null +++ b/apps/desktop/src/main/windows/screen-ocr-overlay.ts @@ -0,0 +1,69 @@ +// src/main/windows/screen-ocr-overlay.ts +// +// 화면 OCR 오버레이 창 — 캡처한 화면 한 장으로 그 모니터를 통째로 덮는 팝업. +// 다른 팝업과 달리 포커스를 받는다(Esc · Ctrl+C · Ctrl+A 를 창이 직접 받는다). +// 창 생성 · 보안 · 로드 전 IPC 보류는 popup-window-factory 정본을 그대로 쓴다. + +import type { BrowserWindow } from 'electron' +import { createPopupWindow, sendToPopupWindow } from './popup-window-factory' +import { getLogger } from '../services/LoggerService' + +const logger = getLogger('ScreenOcrOverlay') + +let overlayWindow: BrowserWindow | null = null + +function createOverlayWindow(): BrowserWindow { + return createPopupWindow({ + name: 'screen-ocr', + // 표시할 때 대상 모니터 크기로 다시 맞춘다. + bounds: { width: 800, height: 600 }, + focusable: true, + transparent: false, + onClosed: () => { + overlayWindow = null + } + }) +} + +export function getScreenOcrOverlayWindow(): BrowserWindow { + if (!overlayWindow || overlayWindow.isDestroyed()) { + overlayWindow = createOverlayWindow() + logger.info('ScreenOcr overlay window created (preloaded)') + } + return overlayWindow +} + +/** 창을 새로 만들지 않고 살아 있는 오버레이만 조회한다. */ +export function peekScreenOcrOverlayWindow(): BrowserWindow | null { + return overlayWindow && !overlayWindow.isDestroyed() ? overlayWindow : null +} + +export function isScreenOcrOverlayVisible(): boolean { + const win = peekScreenOcrOverlayWindow() + return win !== null && win.isVisible() +} + +export function sendToScreenOcrOverlay(channel: string, data: unknown): void { + const win = peekScreenOcrOverlayWindow() + if (win) sendToPopupWindow(win, channel, data) +} + +/** + * 오버레이로 모니터 하나를 덮는다 (bounds 는 그 모니터의 DIP 영역). + * 작업 표시줄 위까지 덮어야 얼린 화면과 실제 화면이 어긋나 보이지 않는다 — 'screen-saver' 단계로 올린다. + */ +export function showScreenOcrOverlay(bounds: Electron.Rectangle): void { + const win = getScreenOcrOverlayWindow() + win.setBounds(bounds) + win.setAlwaysOnTop(true, 'screen-saver') + win.show() + // 배율이 다른 모니터로 옮기면 첫 setBounds 가 옛 배율로 계산될 수 있다 — 보인 뒤 한 번 더 맞춘다. + win.setBounds(bounds) + win.moveTop() + win.focus() +} + +export function hideScreenOcrOverlay(): void { + const win = peekScreenOcrOverlayWindow() + if (win && win.isVisible()) win.hide() +} diff --git a/apps/desktop/src/renderer/popups/screen-ocr/index.html b/apps/desktop/src/renderer/popups/screen-ocr/index.html new file mode 100644 index 00000000..75cadfc8 --- /dev/null +++ b/apps/desktop/src/renderer/popups/screen-ocr/index.html @@ -0,0 +1,62 @@ + + + + + + + Screen OCR + + +
+ + + + +
+ + + + + + + +
+ + + + + + diff --git a/apps/desktop/src/renderer/popups/screen-ocr/script.js b/apps/desktop/src/renderer/popups/screen-ocr/script.js new file mode 100644 index 00000000..30c8ae30 --- /dev/null +++ b/apps/desktop/src/renderer/popups/screen-ocr/script.js @@ -0,0 +1,678 @@ +// 화면 OCR 오버레이 스크립트 (2026-11-20) +// +// 캡처한 화면 위에서 글자를 고른다: +// - 글자 선택(기본): 단어에서 끌기 시작하면 글 흐름대로, 빈 곳에서 끌면 사각형으로 고른다. +// 더블클릭 = 단어, 세 번 클릭 = 줄, Shift+클릭 = 이어서 고르기. +// - 영역 선택: 언제나 사각형으로 고른다(그 영역의 그림 · PDF 를 낼 때도 쓴다). +// 선택 계산 · 글 만들기는 @d3ro/core/screen-ocr 정본을 그대로 쓴다(메인 프로세스와 같은 계산). +// 문구는 전부 main 의 _i18n. 좌표는 캡처 그림의 물리 픽셀로 다루고, 그릴 때만 CSS 픽셀로 바꾼다. + +import { + allWords, + detectLinks, + flowSelection, + lineSelection, + nearestWordIndex, + rectFromPoints, + selectionBounds, + selectionHighlightRects, + selectionText, + wordAtPoint, + wordsInRect +} from '@d3ro/core/screen-ocr' + +const CHANNEL = { + SESSION: 'screenOcr:session', + RECOGNIZED: 'screenOcr:recognized', + RESPONSE: 'screenOcr:response', + REQUEST: 'screenOcr:request', + READY: 'screenOcr:ready', + DISMISS: 'screenOcr:dismiss' +} + +/** 이보다 작게 끈 사각형은 클릭으로 본다 (CSS 픽셀) */ +const CLICK_SLOP = 4 +/** 글자만 골랐을 때 그림 · PDF 로 낼 영역에 두르는 여백 (물리 픽셀) */ +const REGION_PADDING = 8 +const TOAST_MS = 1400 +const CLOSE_AFTER_COPY_MS = 420 + +const $ = (id) => document.getElementById(id) +const stage = $('stage') +const shot = $('shot') +const highlights = $('highlights') +const regionEl = $('region') +const toolbar = $('toolbar') +const actions = $('actions') +const statusEl = $('status') +const languageEl = $('language') +const resultEl = $('result') +const resultTitle = $('result-title') +const resultBody = $('result-body') +const toastEl = $('toast') + +const state = { + sessionId: null, + /** 캡처 그림 크기 (물리 픽셀) */ + width: 0, + height: 0, + imageUrl: null, + /** 인식 결과 (없으면 아직 읽는 중이거나 실패) */ + result: null, + recognizing: false, + recognitionError: null, + mode: 'text', + /** 고른 단어 번호 */ + selected: [], + /** 글 흐름 선택의 시작 단어 */ + anchor: null, + /** 사각형으로 고른 영역 (물리 픽셀). 글 흐름으로 골랐으면 null */ + region: null, + drag: null, + speaking: false, + i18n: {}, + nextRequestId: 1, + pending: new Map(), + toastTimer: null, + closeTimer: null +} + +const unsubs = [] + +function tr(key, vars) { + let text = (state.i18n && state.i18n[key]) || '' + if (vars) { + for (const name of Object.keys(vars)) text = text.split('{{' + name + '}}').join(String(vars[name])) + } + return text +} + +function applyI18n() { + document.querySelectorAll('[data-i18n]').forEach((el) => { + el.textContent = tr(el.getAttribute('data-i18n')) + }) + document.querySelectorAll('[data-i18n-label]').forEach((el) => { + el.setAttribute('aria-label', tr(el.getAttribute('data-i18n-label'))) + el.setAttribute('title', tr(el.getAttribute('data-i18n-label'))) + }) +} + +// ── 좌표 ────────────────────────────────────────────── + +function scaleX() { + return state.width > 0 && window.innerWidth > 0 ? state.width / window.innerWidth : 1 +} + +function scaleY() { + return state.height > 0 && window.innerHeight > 0 ? state.height / window.innerHeight : 1 +} + +/** 마우스 이벤트 → 캡처 그림 좌표 */ +function toImagePoint(event) { + return { x: event.clientX * scaleX(), y: event.clientY * scaleY() } +} + +function toCssRect(rect) { + return { + left: rect.x / scaleX(), + top: rect.y / scaleY(), + width: rect.width / scaleX(), + height: rect.height / scaleY() + } +} + +function place(el, rect) { + const css = toCssRect(rect) + el.style.left = css.left + 'px' + el.style.top = css.top + 'px' + el.style.width = css.width + 'px' + el.style.height = css.height + 'px' +} + +// ── 그리기 ──────────────────────────────────────────── + +function selectedText(singleLine) { + return state.result ? selectionText(state.result, state.selected, { singleLine: Boolean(singleLine) }) : '' +} + +/** 그림 · PDF 로 낼 영역: 끌어 고른 사각형 → 고른 글자를 감싼 사각형 → 없으면 화면 전체(null) */ +function exportRegion() { + if (state.region) return state.region + if (!state.result || state.selected.length === 0) return null + const bounds = selectionBounds(state.result, state.selected) + if (!bounds) return null + return { + x: bounds.x - REGION_PADDING, + y: bounds.y - REGION_PADDING, + width: bounds.width + REGION_PADDING * 2, + height: bounds.height + REGION_PADDING * 2 + } +} + +function renderHighlights() { + highlights.textContent = '' + if (!state.result) return + for (const rect of selectionHighlightRects(state.result, state.selected)) { + const el = document.createElement('div') + el.className = 'hl' + place(el, rect) + highlights.appendChild(el) + } +} + +function renderRegion() { + const rect = state.drag && state.drag.kind === 'rect' ? state.drag.rect : state.region + if (!rect || rect.width < 1 || rect.height < 1) { + regionEl.hidden = true + return + } + regionEl.hidden = false + place(regionEl, rect) +} + +function renderStatus() { + statusEl.classList.toggle('error', Boolean(state.recognitionError)) + if (state.recognitionError) { + statusEl.textContent = tr(state.recognitionError) + } else if (state.recognizing) { + statusEl.textContent = tr('statusRecognizing') + } else if (state.result && state.selected.length > 0) { + statusEl.textContent = tr('statusSelected', { count: Array.from(selectedText(false)).length }) + } else if (state.result && state.result.words.length === 0) { + statusEl.textContent = tr('statusEmpty') + } else if (state.result) { + statusEl.textContent = tr('statusReady', { count: state.result.words.length }) + } else { + statusEl.textContent = '' + } +} + +function renderActions() { + const text = selectedText(false) + const hasText = text.trim().length > 0 + for (const id of ['act-copy', 'act-copy-line', 'act-translate', 'act-summarize', 'act-speak', 'act-link']) { + $(id).disabled = !hasText + } + $('act-select-all').disabled = !state.result || state.result.words.length === 0 + $('act-link').textContent = hasText && detectLinks(text).length > 0 ? tr('openLink') : tr('search') + $('act-speak').textContent = state.speaking ? tr('stopSpeak') : tr('speak') + $('act-speak').classList.toggle('on', state.speaking) + if (state.speaking) $('act-speak').disabled = false + positionActions() +} + +/** 동작 막대를 고른 곳 바로 아래(자리가 없으면 위)에 붙인다. 고른 것이 없으면 화면 아래 가운데. */ +function positionActions() { + const region = exportRegion() + if (!region) { + actions.classList.remove('anchored') + actions.style.left = '' + actions.style.top = '' + return + } + actions.classList.add('anchored') + const css = toCssRect(region) + const size = actions.getBoundingClientRect() + const gap = 10 + let top = css.top + css.height + gap + if (top + size.height > window.innerHeight - 8) top = css.top - size.height - gap + if (top < 8) top = Math.max(8, window.innerHeight - size.height - 8) + let left = css.left + css.width / 2 - size.width / 2 + left = Math.max(8, Math.min(left, window.innerWidth - size.width - 8)) + actions.style.left = Math.round(left) + 'px' + actions.style.top = Math.round(top) + 'px' +} + +function renderMode() { + $('mode-text').classList.toggle('active', state.mode === 'text') + $('mode-rect').classList.toggle('active', state.mode === 'rect') + $('mode-text').setAttribute('aria-pressed', String(state.mode === 'text')) + $('mode-rect').setAttribute('aria-pressed', String(state.mode === 'rect')) +} + +function renderLanguages(languages, current) { + languageEl.textContent = '' + const auto = document.createElement('option') + auto.value = 'auto' + auto.textContent = tr('languageAuto') + languageEl.appendChild(auto) + for (const tag of Array.isArray(languages) ? languages : []) { + const option = document.createElement('option') + option.value = tag + option.textContent = tag + languageEl.appendChild(option) + } + if (typeof current === 'string') languageEl.value = current + if (languageEl.selectedIndex === -1) languageEl.value = 'auto' +} + +function render() { + renderHighlights() + renderRegion() + renderStatus() + renderMode() + renderActions() +} + +function showToast(text, isError) { + if (state.toastTimer) clearTimeout(state.toastTimer) + toastEl.textContent = text + toastEl.classList.toggle('error', Boolean(isError)) + toastEl.hidden = false + state.toastTimer = setTimeout(() => { + toastEl.hidden = true + state.toastTimer = null + }, TOAST_MS) +} + +function errorText(kind) { + const key = 'error' + String(kind || 'export').charAt(0).toUpperCase() + String(kind || 'export').slice(1) + return tr(key) || tr('errorExport') +} + +// ── 선택 ────────────────────────────────────────────── + +function setSelection(selected, region) { + state.selected = selected + state.region = region || null + render() +} + +function clearSelection() { + state.anchor = null + setSelection([], null) +} + +function selectAll() { + if (!state.result) return + state.anchor = null + setSelection(allWords(state.result), null) +} + +function setMode(mode) { + state.mode = mode + renderMode() +} + +function onMouseDown(event) { + if (event.button !== 0 || !state.sessionId) return + closeResult() + const point = toImagePoint(event) + const hit = state.result ? wordAtPoint(state.result, point, 3 * scaleX()) : null + + if (state.mode === 'text' && hit !== null) { + if (event.shiftKey && state.anchor !== null) { + setSelection(flowSelection(state.anchor, hit), null) + } else if (event.detail >= 3) { + state.anchor = hit + setSelection(lineSelection(state.result, hit), null) + return + } else if (event.detail === 2) { + state.anchor = hit + setSelection([hit], null) + return + } else { + state.anchor = hit + setSelection([hit], null) + } + state.drag = { kind: 'flow', moved: false } + } else { + state.drag = { + kind: 'rect', + startCss: { x: event.clientX, y: event.clientY }, + start: point, + rect: { x: point.x, y: point.y, width: 0, height: 0 }, + moved: false + } + } + event.preventDefault() +} + +function onMouseMove(event) { + const point = toImagePoint(event) + const drag = state.drag + if (!drag) { + const over = state.mode === 'text' && state.result ? wordAtPoint(state.result, point, 3 * scaleX()) : null + stage.classList.toggle('over-word', over !== null) + return + } + if (drag.kind === 'flow') { + const focus = nearestWordIndex(state.result, point) + if (focus === null || state.anchor === null) return + drag.moved = true + document.body.classList.add('dragging') + state.selected = flowSelection(state.anchor, focus) + renderHighlights() + return + } + const dx = Math.abs(event.clientX - drag.startCss.x) + const dy = Math.abs(event.clientY - drag.startCss.y) + if (!drag.moved && dx < CLICK_SLOP && dy < CLICK_SLOP) return + drag.moved = true + document.body.classList.add('dragging') + drag.rect = rectFromPoints(drag.start, point) + state.selected = state.result ? wordsInRect(state.result, drag.rect) : [] + renderHighlights() + renderRegion() +} + +function onMouseUp(event) { + const drag = state.drag + if (!drag || event.button !== 0) return + state.drag = null + document.body.classList.remove('dragging') + if (drag.kind === 'rect') { + if (!drag.moved) { + // 빈 곳을 그냥 클릭했다 — 선택을 푼다. + clearSelection() + return + } + state.anchor = null + state.region = drag.rect + } + render() +} + +// ── main 과 주고받기 ────────────────────────────────── + +function send(channel, data) { + if (window.popupAPI) window.popupAPI.send(channel, data) +} + +function listen(channel, handler) { + if (!window.popupAPI) return + const unsub = window.popupAPI.on(channel, handler) + if (typeof unsub === 'function') unsubs.push(unsub) +} + +/** 요청을 보내고 응답을 기다린다. 닫히면(새 캡처 포함) 대기 중인 요청은 버린다. */ +function request(payload) { + const sessionId = state.sessionId + if (!sessionId) return Promise.resolve({ ok: false, error: 'invalid' }) + const requestId = state.nextRequestId + state.nextRequestId += 1 + return new Promise((resolve) => { + state.pending.set(requestId, resolve) + send(CHANNEL.REQUEST, { sessionId, requestId, request: payload }) + }) +} + +function dismiss() { + send(CHANNEL.DISMISS, {}) +} + +function closeAfterToast(text) { + showToast(text) + if (state.closeTimer) clearTimeout(state.closeTimer) + state.closeTimer = setTimeout(() => { + state.closeTimer = null + dismiss() + }, CLOSE_AFTER_COPY_MS) +} + +// ── 동작 ────────────────────────────────────────────── + +async function copyText(singleLine) { + const text = selectedText(singleLine) + if (!text.trim()) return + const response = await request({ type: 'copy-text', text }) + if (response.ok) closeAfterToast(tr('toastCopied')) + else showToast(errorText(response.error), true) +} + +async function copyImage() { + const response = await request({ type: 'copy-image', rect: exportRegion() }) + if (response.ok) closeAfterToast(tr('toastImageCopied')) + else showToast(errorText(response.error), true) +} + +async function saveFile(type) { + const response = await request({ type, rect: exportRegion() }) + if (!response.ok) showToast(errorText(response.error), true) + else if (!response.cancelled) showToast(tr('toastSaved')) +} + +function openResult(title) { + resultTitle.textContent = title + resultBody.textContent = tr('working') + resultBody.className = 'result-body pending' + $('result-copy').disabled = true + resultEl.hidden = false +} + +function closeResult() { + resultEl.hidden = true +} + +async function runLlm(type, title) { + const text = selectedText(false) + if (!text.trim()) return + openResult(title) + const response = await request({ type, text }) + if (resultEl.hidden) return + if (response.ok && typeof response.text === 'string') { + resultBody.textContent = response.text + resultBody.className = 'result-body' + $('result-copy').disabled = false + } else { + resultBody.textContent = errorText(response.ok ? 'llm' : response.error) + resultBody.className = 'result-body error' + } +} + +async function toggleSpeak() { + if (state.speaking) { + state.speaking = false + renderActions() + await request({ type: 'stop-speak' }) + return + } + const text = selectedText(true) + if (!text.trim()) return + const response = await request({ type: 'speak', text }) + if (response.ok) { + state.speaking = true + renderActions() + } else { + showToast(errorText(response.error), true) + } +} + +async function openLinkOrSearch() { + const text = selectedText(true) + if (!text.trim()) return + const links = detectLinks(text) + const response = + links.length > 0 ? await request({ type: 'open-link', url: links[0] }) : await request({ type: 'search-web', text }) + if (response.ok) dismiss() + else showToast(errorText(response.error), true) +} + +async function changeLanguage() { + const language = languageEl.value + state.result = null + state.recognizing = true + state.recognitionError = null + state.anchor = null + state.selected = [] + render() + const response = await request({ type: 'set-language', language }) + if (!response.ok) { + state.recognizing = false + state.recognitionError = 'errorRecognition' + render() + } +} + +// ── 캡처 수명 ───────────────────────────────────────── + +function resetSession() { + for (const resolve of state.pending.values()) resolve({ ok: false, error: 'invalid' }) + state.pending.clear() + if (state.imageUrl) URL.revokeObjectURL(state.imageUrl) + if (state.toastTimer) clearTimeout(state.toastTimer) + if (state.closeTimer) clearTimeout(state.closeTimer) + state.toastTimer = null + state.closeTimer = null + state.imageUrl = null + state.sessionId = null + state.result = null + state.recognizing = false + state.recognitionError = null + state.selected = [] + state.anchor = null + state.region = null + state.drag = null + state.speaking = false + shot.removeAttribute('src') + toastEl.hidden = true + closeResult() + document.body.classList.remove('dragging') +} + +function onSession(data) { + resetSession() + if (!data || typeof data !== 'object') { + render() + return + } + state.i18n = data._i18n || {} + state.sessionId = data.sessionId + state.width = Number(data.width) || 0 + state.height = Number(data.height) || 0 + state.recognizing = true + state.mode = 'text' + applyI18n() + renderLanguages(data.languages, data.language) + render() + + const sessionId = data.sessionId + const ready = () => { + // 그림이 실제로 그려진 다음 프레임에 알린다 — 그 전에 창이 보이면 이전 화면이 비친다. + requestAnimationFrame(() => { + if (state.sessionId === sessionId) send(CHANNEL.READY, { sessionId }) + }) + } + state.imageUrl = URL.createObjectURL(new Blob([data.image], { type: 'image/png' })) + shot.onload = ready + // 그림을 못 읽어도 창은 보여야 한다(닫을 수 있게) — 인식 실패는 따로 표시된다. + shot.onerror = ready + shot.src = state.imageUrl +} + +function onRecognized(data) { + if (!data || data.sessionId !== state.sessionId) return + state.recognizing = false + if (Array.isArray(data.languages) && data.languages.length > 0) { + renderLanguages(data.languages, languageEl.value) + } + if (data.ok) { + state.result = data.result + state.recognitionError = null + // 인식이 끝나기 전에 끌어 둔 영역이 있으면 그 안의 글자를 바로 고른다. + state.selected = state.region ? wordsInRect(state.result, state.region) : [] + } else { + state.result = null + state.selected = [] + state.recognitionError = data.error === 'unavailable' ? 'errorUnavailable' : 'errorRecognition' + } + render() +} + +function onResponse(data) { + if (!data || typeof data.requestId !== 'number') return + const resolve = state.pending.get(data.requestId) + if (!resolve) return + state.pending.delete(data.requestId) + resolve(data) +} + +// ── 키보드 ──────────────────────────────────────────── + +function onKeyDown(event) { + if (!state.sessionId) return + const key = event.key + const ctrl = event.ctrlKey || event.metaKey + // 결과 판 안에서 글을 고르고 복사하는 것은 브라우저 기본 동작에 맡긴다. + const inResult = !resultEl.hidden && resultEl.contains(document.activeElement) + + if (key === 'Escape') { + event.preventDefault() + dismiss() + } else if (ctrl && (key === 'c' || key === 'C')) { + if (inResult || String(window.getSelection()) !== '') return + event.preventDefault() + void copyText(event.shiftKey) + } else if (ctrl && (key === 'a' || key === 'A')) { + event.preventDefault() + selectAll() + } else if (ctrl && (key === 's' || key === 'S')) { + event.preventDefault() + void saveFile('save-image') + } else if (ctrl && (key === 'p' || key === 'P')) { + event.preventDefault() + void saveFile('save-pdf') + } else if (key === 'Enter' && document.activeElement === document.body) { + event.preventDefault() + void copyText(false) + } else if (!ctrl && !event.altKey && document.activeElement === document.body) { + if (key === 't' || key === 'T') setMode('text') + else if (key === 'r' || key === 'R') setMode('rect') + } +} + +// ── 연결 ────────────────────────────────────────────── + +function click(id, handler) { + $(id).addEventListener('click', (event) => { + // 버튼이 포커스를 쥐고 있으면 Enter · T · R 단축키가 버튼으로 간다 — 누른 뒤 곧바로 놓는다. + event.currentTarget.blur() + handler() + }) +} + +stage.addEventListener('mousedown', onMouseDown) +window.addEventListener('mousemove', onMouseMove) +window.addEventListener('mouseup', onMouseUp) +window.addEventListener('keydown', onKeyDown) +window.addEventListener('resize', render) +window.addEventListener('contextmenu', (event) => event.preventDefault()) + +click('mode-text', () => setMode('text')) +click('mode-rect', () => setMode('rect')) +click('close', dismiss) +click('flip', () => { + const bottom = toolbar.classList.toggle('bottom') + document.body.classList.toggle('toolbar-bottom', bottom) +}) +click('act-copy', () => void copyText(false)) +click('act-copy-line', () => void copyText(true)) +click('act-translate', () => void runLlm('translate', tr('resultTranslate'))) +click('act-summarize', () => void runLlm('summarize', tr('resultSummary'))) +click('act-speak', () => void toggleSpeak()) +click('act-link', () => void openLinkOrSearch()) +click('act-select-all', selectAll) +click('act-copy-image', () => void copyImage()) +click('act-save-image', () => void saveFile('save-image')) +click('act-save-pdf', () => void saveFile('save-pdf')) +click('result-close', closeResult) +click('result-copy', async () => { + const response = await request({ type: 'copy-text', text: resultBody.textContent || '' }) + if (response.ok) closeAfterToast(tr('toastCopied')) + else showToast(errorText(response.error), true) +}) +languageEl.addEventListener('change', () => void changeLanguage()) + +listen(CHANNEL.SESSION, onSession) +listen(CHANNEL.RECOGNIZED, onRecognized) +listen(CHANNEL.RESPONSE, onResponse) + +window.addEventListener('beforeunload', () => { + resetSession() + while (unsubs.length > 0) { + const unsub = unsubs.pop() + if (typeof unsub === 'function') unsub() + } +}) + +render() diff --git a/apps/desktop/src/renderer/popups/screen-ocr/style.css b/apps/desktop/src/renderer/popups/screen-ocr/style.css new file mode 100644 index 00000000..a5008b3e --- /dev/null +++ b/apps/desktop/src/renderer/popups/screen-ocr/style.css @@ -0,0 +1,311 @@ +/* screen-ocr/style.css — 화면 OCR 오버레이 (2026-11-20). + * 캡처한 화면 한 장이 모니터를 통째로 덮고, 그 위에 선택 강조 · 도구 막대 · 결과 판이 뜬다. + * 색은 main 이 주입하는 CSS 변수만 쓴다(popup-base.css 의 :root 는 주입 전 폴백). */ + +@import '../_shared/popup-base.css'; + +html, +body { + width: 100%; + height: 100%; + background: var(--d3-bg-inset); +} + +#stage { + position: fixed; + inset: 0; + cursor: crosshair; +} + +#stage.over-word { + cursor: text; +} + +#shot { + position: absolute; + inset: 0; + width: 100%; + height: 100%; + pointer-events: none; + -webkit-user-drag: none; +} + +/* 얼린 화면이라는 표시 — 가장자리 한 줄 */ +#frame { + position: absolute; + inset: 0; + border: 2px solid var(--d3-accent-main); + pointer-events: none; +} + +#highlights { + position: absolute; + inset: 0; + pointer-events: none; +} + +.hl { + position: absolute; + border-radius: 2px; + background: color-mix(in srgb, var(--d3-accent-main) 34%, transparent); + box-shadow: 0 0 0 1px color-mix(in srgb, var(--d3-accent-main) 55%, transparent); +} + +/* 영역 선택 — 바깥을 어둡게 눌러 고른 곳만 또렷하게 */ +#region { + position: absolute; + border: 1px solid var(--d3-accent-light); + box-shadow: 0 0 0 9999px rgba(3, 7, 18, 0.42); + pointer-events: none; +} + +.bar { + position: fixed; + left: 50%; + transform: translateX(-50%); + display: flex; + align-items: center; + gap: 6px; + padding: var(--popup-pad); + max-width: calc(100vw - 32px); + white-space: nowrap; + transition: opacity 120ms var(--popup-ease); + z-index: 10; +} + +#toolbar { + top: 14px; +} + +#toolbar.bottom { + top: auto; + bottom: 14px; +} + +#actions { + bottom: 44px; + z-index: 11; +} + +#actions.anchored { + left: 0; + top: 0; + bottom: auto; + transform: none; +} + +/* 끄는 동안은 막대가 글자를 가리지 않게 치운다 */ +body.dragging .bar, +body.dragging #hint { + opacity: 0; + pointer-events: none; +} + +.seg { + display: flex; + gap: 2px; + padding: 2px; + border-radius: var(--popup-radius-item); + background: var(--d3-bg-inset); +} + +.seg-btn, +.act, +.icon-btn { + font: inherit; + color: var(--d3-text-secondary); + background: transparent; + border: 1px solid transparent; + border-radius: var(--popup-radius-item); + padding: 4px 10px; + cursor: pointer; + transition: background 120ms var(--popup-ease), color 120ms var(--popup-ease); +} + +.seg-btn:hover, +.act:hover:not(:disabled), +.icon-btn:hover { + background: var(--d3-overlay-soft); + color: var(--d3-text-primary); +} + +.seg-btn.active { + background: var(--d3-bg-elevated); + color: var(--d3-text-primary); + border-color: var(--d3-border-default); +} + +.act:disabled { + opacity: 0.38; + cursor: default; +} + +.act.primary { + background: var(--d3-accent-solid); + color: var(--d3-accent-ink); +} + +.act.primary:hover:not(:disabled) { + background: var(--d3-accent-solidHover); + color: var(--d3-accent-ink); +} + +.act.on { + color: var(--d3-accent-light); + border-color: var(--d3-border-strong); +} + +.icon-btn { + padding: 4px 8px; + line-height: 1.2; +} + +:is(.seg-btn, .act, .icon-btn, select):focus-visible { + outline: 2px solid var(--d3-accent-light); + outline-offset: 1px; +} + +.sep { + width: 1px; + align-self: stretch; + margin: 2px; + background: var(--d3-border-default); +} + +.status { + min-width: 150px; + padding: 0 6px; + font-size: 12px; + color: var(--d3-text-secondary); + font-variant-numeric: tabular-nums; +} + +.status.error { + color: var(--d3-status-danger); +} + +.lang { + display: flex; + align-items: center; + gap: 6px; + font-size: 12px; + color: var(--d3-text-tertiary); +} + +.lang select { + font: inherit; + font-size: 12px; + color: var(--d3-text-primary); + background: var(--d3-bg-inset); + border: 1px solid var(--d3-border-default); + border-radius: var(--popup-radius-item); + padding: 3px 6px; +} + +.hint { + position: fixed; + left: 50%; + bottom: 14px; + transform: translateX(-50%); + padding: 3px 10px; + border-radius: var(--popup-radius-item); + font-size: 12px; + color: var(--d3-text-secondary); + background: color-mix(in srgb, var(--d3-bg-card) 88%, transparent); + pointer-events: none; + transition: opacity 120ms var(--popup-ease); + z-index: 9; +} + +/* 도구 막대를 아래로 내리면 안내 줄은 위로 올린다 */ +body.toolbar-bottom .hint { + bottom: auto; + top: 14px; +} + +body.toolbar-bottom #actions:not(.anchored) { + bottom: 60px; +} + +#result { + position: fixed; + left: 50%; + top: 50%; + transform: translate(-50%, -50%); + width: min(560px, calc(100vw - 48px)); + max-height: min(60vh, 520px); + display: flex; + flex-direction: column; + z-index: 20; +} + +#result[hidden], +#toast[hidden], +#region[hidden] { + display: none; +} + +.result-head { + display: flex; + align-items: center; + justify-content: space-between; + padding: 8px 8px 6px 14px; + border-bottom: 1px solid var(--d3-border-subtle); +} + +.result-title { + font-size: 12px; + font-weight: 600; + color: var(--d3-text-secondary); +} + +.result-body { + padding: 12px 14px; + overflow-y: auto; + white-space: pre-wrap; + overflow-wrap: anywhere; + user-select: text; + cursor: text; + font-size: 14px; + line-height: 1.6; +} + +.result-body.pending { + color: var(--d3-text-tertiary); +} + +.result-body.error { + color: var(--d3-status-danger); +} + +.result-foot { + display: flex; + justify-content: flex-end; + padding: 6px 8px 8px; + border-top: 1px solid var(--d3-border-subtle); +} + +.toast { + position: fixed; + left: 50%; + top: 50%; + transform: translate(-50%, -50%); + padding: 8px 16px; + font-weight: 600; + z-index: 30; + pointer-events: none; +} + +.toast.error { + color: var(--d3-status-danger); +} + +@media (prefers-reduced-motion: reduce) { + .bar, + .hint, + .seg-btn, + .act, + .icon-btn { + transition: none; + } +} diff --git a/apps/desktop/tests/main/services/keybinding-redteam-r2-16.test.ts b/apps/desktop/tests/main/services/keybinding-redteam-r2-16.test.ts index 76c4d560..d0f91c5a 100644 --- a/apps/desktop/tests/main/services/keybinding-redteam-r2-16.test.ts +++ b/apps/desktop/tests/main/services/keybinding-redteam-r2-16.test.ts @@ -70,6 +70,18 @@ describe('acceleratorsToReserve', () => { ).toEqual(['Control+Shift+V', 'Control+Shift+C']) }) + it('화면OCR단축키예약_쓸수없는환경_키만삼키지않게예약하지않고쓸수있을때만예약한다', () => { + const reserved = (screenOcrAvailable: boolean | undefined): string[] => + acceleratorsToReserve(defaultReservable(), { + platform: 'darwin', + flags: { suggestionEnabled: false, screenOcrAvailable } + }) + expect(reserved(undefined)).not.toContain('Control+Alt+O') + expect(reserved(false)).not.toContain('Control+Alt+O') + expect(reserved(true)).toContain('Control+Alt+O') + expect(isKeyBindingActionFeatureEnabled('screen-ocr', { suggestionEnabled: true })).toBe(false) + }) + it('macOS 에서 제안 기능이 켜져 있으면 suggestion-* 바인딩도 예약한다', () => { expect( acceleratorsToReserve(defaultReservable(), { platform: 'darwin', flags: { suggestionEnabled: true } }) diff --git a/apps/desktop/tests/main/services/screen-ocr-service.test.ts b/apps/desktop/tests/main/services/screen-ocr-service.test.ts new file mode 100644 index 00000000..cd989f57 --- /dev/null +++ b/apps/desktop/tests/main/services/screen-ocr-service.test.ts @@ -0,0 +1,561 @@ +import { describe, expect, it, vi } from 'vitest' +import { D3ROError, ErrorCode } from '@d3ro/core/errors' +import { parseOcrEngineOutput, type OcrRect, type OcrResult } from '@d3ro/core/screen-ocr' +import { + ScreenOcrService, + type ScreenOcrDeps, + type ScreenOcrRecognizedPayload, + type ScreenOcrResponse, + type ScreenOcrSessionPayload +} from '../../../src/main/services/ScreenOcrService' +import { createWindowsOcrEngine } from '../../../src/main/services/screen-ocr/windows-ocr-engine' + +function ocrResult(): OcrResult { + const result = parseOcrEngineOutput({ + lang: 'ko', + lines: [ + { words: [{ t: '화면', x: 10, y: 10, w: 40, h: 20 }, { t: 'OCR', x: 60, y: 10, w: 40, h: 20 }] }, + { words: [{ t: '둘째', x: 10, y: 50, w: 40, h: 20 }] } + ] + }) + if (!result) throw new Error('fixture did not parse') + return result +} + +interface Harness { + service: ScreenOcrService + shown: Array<{ bounds: OcrRect; payload: ScreenOcrSessionPayload }> + recognized: ScreenOcrRecognizedPayload[] + responses: ScreenOcrResponse[] + hidden: { count: number } + temp: Map + written: Map + clipboard: { text: string | null; image: Buffer | null } + crops: OcrRect[] + opened: string[] + config: { language: string } + recognizeCalls: Array<{ imagePath: string; language: string; scale: number }> + finishRecognition: (value: OcrResult | Error) => void + sessionId: () => string + deps: ScreenOcrDeps +} + +interface HarnessOptions { + supported?: boolean + savePath?: string | null + captureError?: Error + /** 인식을 곧바로 끝낼지 (기본 true). false 면 finishRecognition 을 부를 때까지 걸려 있다. */ + autoRecognize?: boolean + openExternal?: (url: string) => Promise + translate?: ScreenOcrDeps['llm']['translate'] + renderPdf?: ScreenOcrDeps['renderPdf'] + clipboardError?: Error +} + +function createHarness(options: HarnessOptions = {}): Harness { + const shown: Harness['shown'] = [] + const recognized: ScreenOcrRecognizedPayload[] = [] + const responses: ScreenOcrResponse[] = [] + const hidden = { count: 0 } + const temp = new Map() + const written = new Map() + const clipboard: Harness['clipboard'] = { text: null, image: null } + const crops: OcrRect[] = [] + const opened: string[] = [] + const config = { language: 'auto' } + const recognizeCalls: Harness['recognizeCalls'] = [] + let pendingRecognition: Array<(value: OcrResult | Error) => void> = [] + + const deps: ScreenOcrDeps = { + capture: async () => { + if (options.captureError) throw options.captureError + return { + png: Buffer.from('full-png'), + width: 1920, + height: 1080, + scaleFactor: 1, + displayBounds: { x: 0, y: 0, width: 1920, height: 1080 }, + crop: (rect) => { + crops.push(rect) + return Buffer.from(`crop-${rect.x}-${rect.y}-${rect.width}-${rect.height}`) + } + } + }, + engine: { + maxDimension: 10000, + isSupported: () => options.supported ?? true, + listLanguages: async () => ['ko', 'en-US'], + recognize: (request) => { + recognizeCalls.push({ imagePath: request.imagePath, language: request.language, scale: request.scale }) + if (options.autoRecognize ?? true) return Promise.resolve(ocrResult()) + return new Promise((resolve, reject) => { + pendingRecognition.push((value) => (value instanceof Error ? reject(value) : resolve(value))) + }) + } + }, + overlay: { + show: (bounds, payload) => shown.push({ bounds, payload }), + sendRecognized: (payload) => recognized.push(payload), + respond: (payload) => responses.push(payload), + hide: () => { + hidden.count += 1 + } + }, + files: { + writeTemp: async (name, data) => { + const path = `/tmp/${name}` + temp.set(path, data) + return path + }, + remove: async (path) => { + temp.delete(path) + }, + askSavePath: async () => (options.savePath === undefined ? '/out/file' : options.savePath), + write: async (path, data) => { + written.set(path, data) + } + }, + renderPdf: options.renderPdf ?? (async () => Buffer.from('%PDF')), + clipboard: { + writeText: async (text) => { + if (options.clipboardError) throw options.clipboardError + clipboard.text = text + }, + writeImage: async (png) => { + clipboard.image = png + } + }, + llm: { + translate: options.translate ?? (async (text, target) => `[${target}] ${text}`), + summarize: async (text) => `요약: ${text}` + }, + tts: { speak: vi.fn(async () => undefined), stop: vi.fn() }, + openExternal: + options.openExternal ?? + (async (url) => { + opened.push(url) + return true + }), + config: { + getOcrLanguage: () => config.language, + setOcrLanguage: (language) => { + config.language = language + }, + getAppLocale: () => 'ko', + getTranslationTarget: () => 'English' + }, + strings: () => ({ copy: '복사' }), + now: () => new Date(2026, 10, 20, 14, 5, 9).getTime() + } + + return { + service: new ScreenOcrService(deps), + shown, + recognized, + responses, + hidden, + temp, + written, + clipboard, + crops, + opened, + config, + recognizeCalls, + finishRecognition: (value) => { + const waiting = pendingRecognition + pendingRecognition = [] + for (const finish of waiting) finish(value) + }, + sessionId: () => shown[shown.length - 1].payload.sessionId, + deps + } +} + +/** 시작하고 인식이 끝나 오버레이에 결과가 갈 때까지 기다린다. */ +async function open(h: Harness): Promise { + await h.service.start() + await vi.waitFor(() => expect(h.recognized).toHaveLength(1)) +} + +describe('ScreenOcrService 캡처 수명주기', () => { + it('화면OCR시작_캡처성공_오버레이를화면크기로띄우고인식결과를보낸다', async () => { + const h = createHarness() + await open(h) + + expect(h.shown).toHaveLength(1) + expect(h.shown[0].bounds).toEqual({ x: 0, y: 0, width: 1920, height: 1080 }) + expect(h.shown[0].payload).toMatchObject({ width: 1920, height: 1080, language: 'auto', _i18n: { copy: '복사' } }) + expect(h.recognized[0]).toMatchObject({ ok: true, languages: ['ko', 'en-US'] }) + expect(h.recognizeCalls[0]).toMatchObject({ language: 'auto', scale: 2 }) + expect(h.service.isOpen).toBe(true) + }) + + it('화면OCR시작_인식이끝난뒤_캡처임시파일이디스크에남지않는다', async () => { + const h = createHarness() + await open(h) + expect(h.recognizeCalls[0].imagePath).toMatch(/\.png$/u) + expect(h.temp.size).toBe(0) + }) + + it('화면OCR시작_인식이실패해도_캡처임시파일을지우고실패를오버레이에알린다', async () => { + const h = createHarness({ autoRecognize: false }) + await h.service.start() + await vi.waitFor(() => expect(h.recognizeCalls).toHaveLength(1)) + h.finishRecognition(new Error('engine crashed')) + await vi.waitFor(() => expect(h.recognized).toHaveLength(1)) + + expect(h.recognized[0]).toMatchObject({ ok: false, error: 'recognition' }) + expect(h.temp.size).toBe(0) + }) + + it('화면OCR시작_설치된인식언어가없음_쓸수없음으로알린다', async () => { + const h = createHarness({ autoRecognize: false }) + await h.service.start() + await vi.waitFor(() => expect(h.recognizeCalls).toHaveLength(1)) + h.finishRecognition(new D3ROError(ErrorCode.ScreenOcrUnavailable, 'no language')) + await vi.waitFor(() => expect(h.recognized).toHaveLength(1)) + expect(h.recognized[0]).toMatchObject({ ok: false, error: 'unavailable' }) + }) + + it('화면OCR시작_지원하지않는플랫폼_오버레이를띄우지않고ScreenOcrUnavailable을던진다', async () => { + const h = createHarness({ supported: false }) + await expect(h.service.start()).rejects.toMatchObject({ code: ErrorCode.ScreenOcrUnavailable }) + expect(h.shown).toHaveLength(0) + expect(h.service.isOpen).toBe(false) + }) + + it('화면OCR시작_캡처실패_ScreenOcrCaptureFailed를던지고열린상태로남지않는다', async () => { + const failing = createHarness({ captureError: new Error('no source') }) + await expect(failing.service.start()).rejects.toMatchObject({ code: ErrorCode.ScreenOcrCaptureFailed }) + expect(failing.shown).toHaveLength(0) + expect(failing.service.isOpen).toBe(false) + }) + + it('화면OCR시작_이미열려있을때다시시작_새캡처를겹쳐만들지않는다', async () => { + const h = createHarness() + await open(h) + await h.service.start() + expect(h.shown).toHaveLength(1) + }) + + it('화면OCR토글_열려있을때단축키_닫고오버레이를숨긴다', async () => { + const h = createHarness() + await open(h) + await h.service.toggle() + expect(h.service.isOpen).toBe(false) + expect(h.hidden.count).toBe(1) + }) + + it('화면OCR닫기_인식이끝나기전에닫음_늦게온결과를오버레이에보내지않는다', async () => { + const h = createHarness({ autoRecognize: false }) + await h.service.start() + await vi.waitFor(() => expect(h.recognizeCalls).toHaveLength(1)) + h.service.close() + h.finishRecognition(ocrResult()) + await vi.waitFor(() => expect(h.temp.size).toBe(0)) + expect(h.recognized).toHaveLength(0) + }) + + it('화면OCR닫기_읽어주기를쓰지않은캡처_다른기능의음성재생을멈추지않는다', async () => { + const h = createHarness() + await open(h) + h.service.close() + expect(h.deps.tts.stop).not.toHaveBeenCalled() + }) + + it('화면OCR닫기_읽어주는중에닫음_읽기를멈춘다', async () => { + const h = createHarness() + await open(h) + await h.service.handleRequest(h.sessionId(), 1, { type: 'speak', text: '읽어 줘' }) + expect(h.deps.tts.speak).toHaveBeenCalledWith('읽어 줘') + h.service.close() + expect(h.deps.tts.stop).toHaveBeenCalledTimes(1) + }) +}) + +describe('ScreenOcrService 요청 처리', () => { + it('글자복사_선택한글_클립보드에쓰고성공으로응답한다', async () => { + const h = createHarness() + await open(h) + await h.service.handleRequest(h.sessionId(), 7, { type: 'copy-text', text: '화면 OCR\n둘째' }) + expect(h.clipboard.text).toBe('화면 OCR\n둘째') + expect(h.responses).toEqual([{ requestId: 7, ok: true }]) + }) + + it('글자복사_클립보드쓰기실패_성공으로꾸미지않고실패로응답한다', async () => { + const h = createHarness({ clipboardError: new Error('clipboard locked') }) + await open(h) + await h.service.handleRequest(h.sessionId(), 2, { type: 'copy-text', text: '화면' }) + expect(h.responses).toEqual([{ requestId: 2, ok: false, error: 'export' }]) + }) + + it('글자복사_빈글_클립보드를건드리지않고선택없음으로거절한다', async () => { + const h = createHarness() + await open(h) + await h.service.handleRequest(h.sessionId(), 1, { type: 'copy-text', text: ' ' }) + expect(h.clipboard.text).toBeNull() + expect(h.responses).toEqual([{ requestId: 1, ok: false, error: 'empty' }]) + }) + + it('요청처리_끝난캡처의요청_아무것도하지않고거절한다', async () => { + const h = createHarness() + await open(h) + const stale = h.sessionId() + h.service.close() + await h.service.handleRequest(stale, 3, { type: 'copy-text', text: '늦은 요청' }) + expect(h.clipboard.text).toBeNull() + expect(h.responses).toEqual([{ requestId: 3, ok: false, error: 'invalid' }]) + }) + + it('이미지복사_영역지정_그림밖부분을잘라낸영역만복사한다', async () => { + const h = createHarness() + await open(h) + await h.service.handleRequest(h.sessionId(), 1, { + type: 'copy-image', + rect: { x: 1900, y: -20, width: 100, height: 120 } + }) + expect(h.crops).toEqual([{ x: 1900, y: 0, width: 20, height: 100 }]) + expect(h.clipboard.image?.toString()).toBe('crop-1900-0-20-100') + }) + + it('이미지복사_영역없음_화면전체를복사한다', async () => { + const h = createHarness() + await open(h) + await h.service.handleRequest(h.sessionId(), 1, { type: 'copy-image', rect: null }) + expect(h.clipboard.image?.toString()).toBe('full-png') + expect(h.crops).toEqual([]) + }) + + it('이미지복사_그림밖영역_복사하지않고거절한다', async () => { + const h = createHarness() + await open(h) + await h.service.handleRequest(h.sessionId(), 1, { + type: 'copy-image', + rect: { x: 5000, y: 5000, width: 10, height: 10 } + }) + expect(h.clipboard.image).toBeNull() + expect(h.responses).toEqual([{ requestId: 1, ok: false, error: 'invalid' }]) + }) + + it('이미지저장_저장위치선택_그림을파일로쓴다', async () => { + const h = createHarness({ savePath: '/out/shot.png' }) + await open(h) + await h.service.handleRequest(h.sessionId(), 1, { type: 'save-image', rect: null }) + expect(h.written.get('/out/shot.png')?.toString()).toBe('full-png') + expect(h.responses).toEqual([{ requestId: 1, ok: true }]) + }) + + it('이미지저장_대화상자취소_파일을쓰지않고취소로응답한다', async () => { + const h = createHarness({ savePath: null }) + await open(h) + await h.service.handleRequest(h.sessionId(), 1, { type: 'save-image', rect: null }) + expect(h.written.size).toBe(0) + expect(h.responses).toEqual([{ requestId: 1, ok: true, cancelled: true }]) + }) + + it('PDF저장_영역지정_그영역의글자만영역기준좌표로넣은문서를PDF로쓴다', async () => { + let html = '' + const h = createHarness({ + savePath: '/out/doc.pdf', + renderPdf: async (htmlPath) => { + html = String(h.temp.get(htmlPath)) + return Buffer.from('%PDF-1.7') + } + }) + await open(h) + // 첫 줄(y 10~30)만 덮는 영역 + await h.service.handleRequest(h.sessionId(), 1, { + type: 'save-pdf', + rect: { x: 0, y: 0, width: 200, height: 40 } + }) + + expect(h.written.get('/out/doc.pdf')?.toString()).toBe('%PDF-1.7') + expect(html).toContain('@page{size:200px 40px;margin:0}') + expect(html).toContain('>화면') + expect(html).toContain('>OCR') + expect(html).not.toContain('둘째') + expect(h.responses).toEqual([{ requestId: 1, ok: true }]) + }) + + it('PDF저장_PDF생성실패_임시HTML을지우고내보내기실패로응답한다', async () => { + const h = createHarness({ + renderPdf: async () => { + throw new Error('print failed') + } + }) + await open(h) + await h.service.handleRequest(h.sessionId(), 1, { type: 'save-pdf', rect: null }) + expect(h.written.size).toBe(0) + expect(h.temp.size).toBe(0) + expect(h.responses).toEqual([{ requestId: 1, ok: false, error: 'export' }]) + }) + + it('번역_외국어글_앱언어를대상으로번역한결과를돌려준다', async () => { + const h = createHarness() + await open(h) + await h.service.handleRequest(h.sessionId(), 1, { type: 'translate', text: 'The quick brown fox' }) + expect(h.responses).toEqual([{ requestId: 1, ok: true, text: '[Korean] The quick brown fox' }]) + }) + + it('번역_LLM이빈응답_성공으로꾸미지않고실패로응답한다', async () => { + const h = createHarness({ translate: async () => ' ' }) + await open(h) + await h.service.handleRequest(h.sessionId(), 1, { type: 'translate', text: 'hello' }) + expect(h.responses).toEqual([{ requestId: 1, ok: false, error: 'llm' }]) + }) + + it('번역_LLM호출실패_실패종류를llm으로응답한다', async () => { + const h = createHarness({ + translate: async () => { + throw new Error('server unreachable') + } + }) + await open(h) + await h.service.handleRequest(h.sessionId(), 1, { type: 'translate', text: 'hello' }) + expect(h.responses).toEqual([{ requestId: 1, ok: false, error: 'llm' }]) + }) + + it('번역_처리중에닫음_닫힌오버레이로응답을보내지않는다', async () => { + let finish: (text: string) => void = () => undefined + const h = createHarness({ + translate: () => + new Promise((resolve) => { + finish = resolve + }) + }) + await open(h) + const pending = h.service.handleRequest(h.sessionId(), 1, { type: 'translate', text: 'hello' }) + h.service.close() + finish('안녕') + await pending + expect(h.responses).toEqual([]) + }) + + it('요약_선택한글_요약결과를돌려준다', async () => { + const h = createHarness() + await open(h) + await h.service.handleRequest(h.sessionId(), 1, { type: 'summarize', text: '긴 글' }) + expect(h.responses).toEqual([{ requestId: 1, ok: true, text: '요약: 긴 글' }]) + }) + + it('웹검색_선택한글_줄바꿈을정리한검색주소를연다', async () => { + const h = createHarness() + await open(h) + await h.service.handleRequest(h.sessionId(), 1, { type: 'search-web', text: '화면 OCR\n둘째' }) + expect(h.opened).toEqual([`https://www.google.com/search?q=${encodeURIComponent('화면 OCR 둘째')}`]) + }) + + it('링크열기_정책이막은주소_링크실패로응답한다', async () => { + const h = createHarness({ openExternal: async () => false }) + await open(h) + await h.service.handleRequest(h.sessionId(), 1, { type: 'open-link', url: 'file:///C:/secret' }) + expect(h.responses).toEqual([{ requestId: 1, ok: false, error: 'link' }]) + }) + + it('인식언어변경_설치된언어_설정을바꾸고그언어로다시인식한다', async () => { + const h = createHarness() + await open(h) + await h.service.handleRequest(h.sessionId(), 1, { type: 'set-language', language: 'en-US' }) + await vi.waitFor(() => expect(h.recognized).toHaveLength(2)) + expect(h.config.language).toBe('en-US') + expect(h.recognizeCalls[1].language).toBe('en-US') + expect(h.responses).toEqual([{ requestId: 1, ok: true }]) + }) + + it('인식언어변경_설치되지않은언어_설정을바꾸지않고거절한다', async () => { + const h = createHarness() + await open(h) + await h.service.handleRequest(h.sessionId(), 1, { type: 'set-language', language: 'xx-YY' }) + expect(h.config.language).toBe('auto') + expect(h.recognizeCalls).toHaveLength(1) + expect(h.responses).toEqual([{ requestId: 1, ok: false, error: 'invalid' }]) + }) +}) + +describe('Windows OCR 엔진 어댑터', () => { + it('엔진인식_정상출력_정본결과로옮기고입력은환경변수로만넘긴다', async () => { + const calls: Array> = [] + const engine = createWindowsOcrEngine({ + platform: 'win32', + run: async (env) => { + calls.push(env) + return 'WARNING: noise\n{"lang":"ko","lines":[{"words":[{"t":"화면","x":1,"y":2,"w":30,"h":12}]}]}\n' + } + }) + const result = await engine.recognize({ imagePath: 'C:\\tmp\\a b.png', language: 'ko', scale: 1.5 }) + expect(result.words.map((w) => w.text)).toEqual(['화면']) + expect(calls[0]).toEqual({ + D3RO_OCR_MODE: 'recognize', + D3RO_OCR_IMAGE: 'C:\\tmp\\a b.png', + D3RO_OCR_LANG: 'ko', + D3RO_OCR_SCALE: '1.5' + }) + }) + + it('엔진인식_줄과단어가하나뿐이라배열이풀린출력_그대로읽는다', async () => { + const engine = createWindowsOcrEngine({ + platform: 'win32', + run: async () => '{"lang":"en-US","lines":{"words":{"t":"Hello","x":1,"y":2,"w":30,"h":12}}}' + }) + const result = await engine.recognize({ imagePath: 'x.png', language: 'auto', scale: 2 }) + expect(result.words.map((w) => w.text)).toEqual(['Hello']) + expect(result.lines).toHaveLength(1) + }) + + it('엔진인식_글자가없는화면_빈결과를돌려준다', async () => { + const engine = createWindowsOcrEngine({ platform: 'win32', run: async () => '{"lang":"ko","lines":[]}' }) + const result = await engine.recognize({ imagePath: 'x.png', language: 'auto', scale: 2 }) + expect(result.words).toEqual([]) + }) + + it('엔진인식_설치된언어없음_ScreenOcrUnavailable', async () => { + const engine = createWindowsOcrEngine({ platform: 'win32', run: async () => '{"error":"no-language"}' }) + await expect(engine.recognize({ imagePath: 'x.png', language: 'auto', scale: 2 })).rejects.toMatchObject({ + code: ErrorCode.ScreenOcrUnavailable + }) + }) + + it('엔진인식_엔진예외와해석불가출력_ScreenOcrRecognitionFailed', async () => { + const failing = createWindowsOcrEngine({ + platform: 'win32', + run: async () => '{"error":"exception","message":"bad image"}' + }) + await expect(failing.recognize({ imagePath: 'x.png', language: 'auto', scale: 2 })).rejects.toMatchObject({ + code: ErrorCode.ScreenOcrRecognitionFailed + }) + const garbage = createWindowsOcrEngine({ platform: 'win32', run: async () => 'not json at all' }) + await expect(garbage.recognize({ imagePath: 'x.png', language: 'auto', scale: 2 })).rejects.toMatchObject({ + code: ErrorCode.ScreenOcrRecognitionFailed + }) + }) + + it('엔진인식_프로세스실행실패_ScreenOcrRecognitionFailed', async () => { + const engine = createWindowsOcrEngine({ + platform: 'win32', + run: async () => { + throw new Error('spawn ENOENT') + } + }) + await expect(engine.recognize({ imagePath: 'x.png', language: 'auto', scale: 2 })).rejects.toMatchObject({ + code: ErrorCode.ScreenOcrRecognitionFailed + }) + }) + + it('엔진지원여부_Windows가아님_지원하지않고프로세스를띄우지않는다', async () => { + const run = vi.fn(async () => '{}') + const engine = createWindowsOcrEngine({ platform: 'darwin', run }) + expect(engine.isSupported()).toBe(false) + expect(await engine.listLanguages()).toEqual([]) + await expect(engine.recognize({ imagePath: 'x.png', language: 'auto', scale: 2 })).rejects.toMatchObject({ + code: ErrorCode.ScreenOcrUnavailable + }) + expect(run).not.toHaveBeenCalled() + }) + + it('언어목록_하나뿐이라문자열로온출력_목록으로돌려준다', async () => { + const single = createWindowsOcrEngine({ platform: 'win32', run: async () => '{"languages":"ko"}' }) + expect(await single.listLanguages()).toEqual(['ko']) + const many = createWindowsOcrEngine({ platform: 'win32', run: async () => '{"languages":["ko","en-US"]}' }) + expect(await many.listLanguages()).toEqual(['ko', 'en-US']) + }) +}) diff --git a/apps/desktop/tests/main/windows/popup-window-options.test.ts b/apps/desktop/tests/main/windows/popup-window-options.test.ts new file mode 100644 index 00000000..c5fdc6c3 --- /dev/null +++ b/apps/desktop/tests/main/windows/popup-window-options.test.ts @@ -0,0 +1,45 @@ +import { describe, expect, it, vi } from 'vitest' + +vi.mock('electron', () => ({ + BrowserWindow: class {}, + nativeTheme: { shouldUseDarkColors: true }, + shell: { openExternal: vi.fn() }, + app: { getPath: () => '', isPackaged: false } +})) + +vi.mock('@electron-toolkit/utils', () => ({ is: { dev: false } })) + +import { buildPopupWindowOptions } from '../../../src/main/windows/popup-window-factory' + +const spec = { name: 'history-popup', bounds: { width: 380, height: 200 }, onClosed: () => undefined } + +describe('buildPopupWindowOptions', () => { + it('팝업창옵션_기본팝업_포커스를빼앗지않는투명창이다', () => { + const options = buildPopupWindowOptions(spec, '/preload/popup.js') + expect(options).toMatchObject({ + width: 380, + height: 200, + show: false, + frame: false, + transparent: true, + focusable: false, + alwaysOnTop: true, + skipTaskbar: true + }) + expect(options.webPreferences).toMatchObject({ + preload: '/preload/popup.js', + contextIsolation: true, + nodeIntegration: false, + backgroundThrottling: false + }) + }) + + it('팝업창옵션_화면OCR오버레이_키를받는불투명창이되고보안설정은그대로다', () => { + const options = buildPopupWindowOptions( + { ...spec, name: 'screen-ocr', focusable: true, transparent: false }, + '/preload/popup.js' + ) + expect(options).toMatchObject({ focusable: true, transparent: false, frame: false, alwaysOnTop: true }) + expect(options.webPreferences).toMatchObject({ contextIsolation: true, nodeIntegration: false }) + }) +}) diff --git a/docs/map/04-desktop-app.md b/docs/map/04-desktop-app.md index 65c3302a..27b72971 100644 --- a/docs/map/04-desktop-app.md +++ b/docs/map/04-desktop-app.md @@ -30,7 +30,7 @@ Singleton + `EventEmitter` pattern (`getXService()` accessors). | `VoiceModeService` | Orchestrator: 9-state `RecognitionState` + 4-state `AudioState`, dual-condition flush, action queue, and secure audio buffer zeroing (`secureWipeAudioChunks` / `secureZeroBuffer`) on cancel/cleanup (2026-11-13 `W5-01`). Events: session-started/completed/cancelled, transcription-update, audio-level, recognition/audio-state-changed, premium-llm-fallback, error | | `AudioCaptureService` | Mic PCM16 16kHz mono (bundled SoX on Windows, node-record-lpcm16 elsewhere) with `secureWipeAudioChunks` on stop/dispose. Spawns hidden (`windowsHide`); a missing SoX fails with the exact fix command | | `LocalSTTService` | faster-whisper Python sidecar manager (state machine, dual-flush, model download/cancel, background warm-up, live partial transcription, `sidecarBaseUrl` getter for `POST /diarize`, and **2026-11-13** `releasePrimaryModel()` / 15-min `IDLE_MODEL_UNLOAD_MS` timer via `POST /unload` `{ slot: "primary" }` so switching to cloud STT or going idle frees the 1.6–3.1 GB Whisper model while keeping the sidecar HTTP server alive for `/uia/focus`). Connects over IPv4 loopback (`getSidecarBaseUrl`) and fails fast with an actionable message when the bundled engine or virtualenv is missing | -| `KeyBindingService` | uiohook-napi global hooking for **keyboard and mouse**, driven by the `@d3ro/core/keybinding` contract: 6 rebindable actions (dictation, hands-free, command, caption, history-popup, command-popup), several bindings per action, structural reserved-combo checks. Events: `triggered` (in-process payload carries `actionId`, `type` (`pressed`/`released`), `isDoublePress`, `holdMode`, `timestamp`; the renderer-facing `keybinding:triggered` event is the narrower `KeyBindingTriggeredEvent`, `keybinding.ts:1092`), `changed`, `error`. `globalShortcut` is used only to mute the macOS system beep, and only for accelerators it registered itself. Mouse events cannot be suppressed by uiohook, so a bound button also performs its native action | +| `KeyBindingService` | uiohook-napi global hooking for **keyboard and mouse**, driven by the `@d3ro/core/keybinding` contract: rebindable actions (dictation, hands-free, command, caption, history-popup, command-popup, screen-ocr, and the suggestion-* set), several bindings per action, structural reserved-combo checks. Events: `triggered` (in-process payload carries `actionId`, `type` (`pressed`/`released`), `isDoublePress`, `holdMode`, `timestamp`; the renderer-facing `keybinding:triggered` event is the narrower `KeyBindingTriggeredEvent`, `keybinding.ts:1092`), `changed`, `error`. `globalShortcut` is used only to mute the macOS system beep, and only for accelerators it registered itself. Mouse events cannot be suppressed by uiohook, so a bound button also performs its native action | | `TextInsertService` | Clipboard save→set→Ctrl+V→restore (`NutKeyboardPort`: uses native `koffi` Win32 `SendInput` via `utils/win32-unicode-input.ts` on Windows without loading `@nut-tree-fork/nut-js`; `nut-js` used on macOS/Linux) | | `SoundEffectService` | Preloaded WAV feedback (start/stop/error/cancel/chime) | @@ -62,6 +62,7 @@ Singleton + `EventEmitter` pattern (`getXService()` accessors). | `VoiceCommandService` | Keyword → command rule matching | | `ChainService` | Multi-step LLM pipelines (LLMChain). Each step resolves its instruction through `llm-prompts.ts` (`ChainService.ts:196`); before that, chain steps sent placeholders through unsubstituted | | `ScreenContextService` | Active-window + selected-text context (on Windows uses `sendWin32CopyChord()` via `koffi` `SendInput` without loading `nut-js`) | +| `ScreenOcrService` | Screen text capture (CAP-17): owns one capture session — capture → overlay → async recognition → overlay requests (copy text/image, save PNG/searchable PDF, translate, summarize, read aloud, open link/search, language switch) → close. All OS access is behind ports (`services/screen-ocr/electron-adapters.ts`); engine port `OcrEnginePort` has one adapter, `windows-ocr-engine.ts` (Windows.Media.Ocr). Assembled in `src/main/screen-ocr-wiring.ts` (key binding, overlay IPC with sender check, `Escape` capture while open except during a save dialog). Events: `opened`, `closed`. | ### Phase 10+ features | Service | Purpose | @@ -168,7 +169,7 @@ Preload exposes **`window.electronAPI`** with 37 namespaces: `platform, audio, c ## 4. Windows & popups -`windows/WindowManager.ts` manages 7 windows: main (borderless, custom TitleBar; macOS `hiddenInset`) created at boot, and 6 vanilla popups (`recording-tip`, `result-popup`, `history-popup`, `command-popup`, `caption-overlay`, `suggestion-overlay`) created **lazily on first demand** (`peekHistoryPopupWindow` / `peekCommandPopupWindow` inspect existing windows without instantiating hidden Chromium renderers at startup). Injects popup theme CSS + i18n strings via `@d3ro/i18n/main`; 2-phase resize. `windows/TrayManager.ts` — tray icon + menu + double-click show. +`windows/WindowManager.ts` manages 7 windows: main (borderless, custom TitleBar; macOS `hiddenInset`) created at boot, and 6 vanilla popups (`recording-tip`, `result-popup`, `history-popup`, `command-popup`, `caption-overlay`, `suggestion-overlay`); an 8th window, the `screen-ocr` overlay, lives in `windows/screen-ocr-overlay.ts` and reuses the same `popup-window-factory` (spec flags `focusable: true`, `transparent: false`) created **lazily on first demand** (`peekHistoryPopupWindow` / `peekCommandPopupWindow` inspect existing windows without instantiating hidden Chromium renderers at startup). Injects popup theme CSS + i18n strings via `@d3ro/i18n/main`; 2-phase resize. `windows/TrayManager.ts` — tray icon + menu + double-click show. **Popup invariants** (each shipped broken once — do not regress): @@ -186,6 +187,7 @@ Vanilla popups (`src/renderer/popups/`): | `command-popup` | Command selection. Opened by the `command-popup` action (default `Ctrl+Shift+C`, rebindable) | | `caption-overlay` | Live caption overlay (font/opacity/maxLines) | | `suggestion-overlay` | Next-sentence ghost text: caret-anchored (`anchorFloatingPanel`), non-focusable, click-through unless `suggestionOverlayInteractive`; accept/next/dismiss come from global key bindings (the window never owns focus). Its narrow layout shows a one-line local provenance/count summary only, never raw memory evidence. Presentation includes candidate, generating, warm-up and partial-text states; its X hides the panel before main-process dismissal. | +| `screen-ocr` | Screen text capture overlay (CAP-17): the only **focusable, opaque** popup; covers one monitor with the frozen capture at `screen-saver` level. 2-phase show — main sends `screenOcr:session`, the popup paints the image and answers `screenOcr:ready`, then the window is shown. Imports `@d3ro/core/screen-ocr` for selection math (same code as main). Channels `POPUP_SCREEN_OCR` (`session`, `recognized`, `request`/`response`, `ready`, `dismiss`). | --- diff --git a/docs/map/10-feature-catalog.md b/docs/map/10-feature-catalog.md index 1b8db475..edeec413 100644 --- a/docs/map/10-feature-catalog.md +++ b/docs/map/10-feature-catalog.md @@ -30,6 +30,7 @@ Status quick-reference: `[x]` done+verified · `[~]` partial/unverified · `[ ]` | CAP-14 | Recording persistence / crash recovery | [x] | [ ] | [x] | [-] | Desktop WAV persist; mobile durable queue + process-kill WAV recovery | | CAP-15 | Android foreground recording service | [-] | [-] | [x] | [-] | Mobile API 34 FGS + persistent notification (SSOT R-005 GREEN) | | CAP-16 | Rebindable global key bindings (keyboard + mouse) | [x] | [-] | [-] | [-] | Contract SSOT `packages/core/src/keybinding.ts`: `KEY_CATALOG` (10 groups), `KEYBINDING_ACTIONS` (6 actions), `validateBinding`, `detectBindingConflicts`. Multiple bindings per action persist as one `AppConfig.keyBindings` map (`packages/core/src/types.ts`), replacing the four singular `*Shortcut` fields; `ConfigService` migrates legacy values once. `KeyBindingService` hooks keyboard **and** mouse via uiohook — MB1 is not bindable, MB2/MB3 need a modifier, MB4/MB5 are free. **Verified 2026-09-21 on Windows by a manual run** and **2026-10-30 unit-tested** in `KeyBindingService.test.ts` (11 tests, GAP-KEY-01 `[x]`), with dedicated `command` action toggle/hold + forced voice-command matching in `VoiceModeService.ts` / `voice-intent.ts` (`voice-intent.test.ts` 19 tests, GAP-KEY-03 `[x]`). **Still open:** macOS/Linux mouse behavior (`11` GAP-KEY-02). W/M `[-]`: no OS-level global binding surface exists there. B `[-]`: device-local setting. | +| CAP-17 | Screen text capture (OCR overlay) | [~] | [-] | [-] | [-] | Desktop/Windows. The rebindable `screen-ocr` action (default `Ctrl+Alt+O`, CAP-16) freezes the monitor under the cursor and covers it with the `screen-ocr` popup; text is selected by flow drag (start on a word), rectangle drag (start on empty space or in Area mode), double-click word, triple-click line, Shift+click extend, `Ctrl+A`. Actions: copy (multi-line / one line), copy image, save PNG, **searchable PDF** (image above a real text layer), translate and summarize through `LlmGateway`, read aloud (`TTSPlaybackService`), open link / web search, recognition language switch (`AppConfig.screenOcrLanguage`). Pure selection/text/PDF logic `packages/core/src/screen-ocr.ts`; service `apps/desktop/src/main/services/ScreenOcrService.ts`; engine `services/screen-ocr/windows-ocr-engine.ts` (Windows.Media.Ocr via `powershell.exe -EncodedCommand`, 2x upscale); capture `utils/win32-screen-capture.ts` (GDI BitBlt, 39 ms vs 2.9 s for `desktopCapturer` on a 3-monitor PC). The capture touches disk only as one temp PNG that is deleted right after recognition. **Verified 2026-11-20:** core 34 + desktop 41 tests, real-engine probe (Korean+English screen, 0.64 s), headless overlay run (flow/rect selection, copy, translate, PDF request, release on close) and a printed PDF whose text layer is searchable. **Not yet verified:** the live hotkey journey in the running app (focus, mixed-DPI multi-monitor) — `GAP-OCR-01`. macOS has no engine and fails closed with a message — `GAP-OCR-02`. | --- diff --git a/docs/map/11-gap-backlog.md b/docs/map/11-gap-backlog.md index d71696bf..451cecae 100644 --- a/docs/map/11-gap-backlog.md +++ b/docs/map/11-gap-backlog.md @@ -21,6 +21,8 @@ Legend: `[ ]` open · `[~]` in progress · `[!]` blocked externally · `[x]` res | ID | Area | Gap | Evidence | Suggested next step | |---|---|---|---|---| +| GAP-OCR-01 | Desktop | Screen text capture (CAP-17) has not been driven end-to-end in the running app: global hotkey → overlay takes focus → drag select → copy/PDF on a real desktop, including a mixed-DPI multi-monitor setup. Unit, real-engine and headless-overlay checks pass. | `apps/desktop/src/main/screen-ocr-wiring.ts`, `apps/desktop/src/main/windows/screen-ocr-overlay.ts`, `apps/desktop/tests/main/services/screen-ocr-service.test.ts` | Run the installed build: press `Ctrl+Alt+O` on each monitor, confirm the frozen image lines up with the real screen, Esc/Ctrl+C work without a first click, and a saved PDF is searchable. Then flip CAP-17 to `[x]`. | +| GAP-OCR-02 | Desktop | Screen text capture is Windows-only and uses a single recognizer language per pass. macOS has no engine (fails closed with a message); with only the Korean Windows OCR pack installed, small English UI text is sometimes misread. | `apps/desktop/src/main/services/screen-ocr/windows-ocr-engine.ts` (`OcrEnginePort`) | Add a macOS Vision adapter behind `OcrEnginePort`; consider a second pass with another installed language for mixed-language screens, or a bundled ONNX recognizer downloaded at runtime. | | GAP-QA-01 | Quality | Extreme Red Team: headful end-to-end bug hunting across real desktop Electron, Web Next.js, and CI pipelines. | `red_team_log.md`, `apps/desktop/tests/e2e`, `apps/web/e2e/smoke.spec.ts` | `[x]` 2026-09-15: 18 scenarios executed, 14 defects caught and 100% resolved (infinite chunking loop DEF-008, IPC signature mismatch DEF-004, markdown editor typing rollback DEF-006, Web RSC Link serialization DEF-012, secret scanner lookahead DEF-013, etc.). All 18 scenarios GREEN with zero regressions. **2026-11-04 rework:** the red-team cycle specs were one-off bug hunts (machine screenshot paths, fixed sleeps, conditional assertions) and were replaced by 5 behavior-scoped protected journeys + a shared Electron fixture; 4 stale/duplicate feature specs, the orphaned root `tests/e2e`, and 7 ad-hoc Playwright runner scripts were deleted. Duplicates merged. New `npm run test:hygiene` gate blocks the patterns that made them meaningless. Log: `docs/tdd-red/06-repo-application.md`. | | GAP-REL-01 | Release | Official release publication to Forgejo and active public download center deployment. | `scripts/ci/publish-forgejo-release.mjs`, `apps/web/src/app/download/page.tsx`, `site/src/sections/Download.tsx`, `apps/web/e2e/smoke.spec.ts` | `[~]` 2026-09-15: v1.1.0 release assets (`D3RO-Voice-Setup-1.1.0-x64.exe`, `.blockmap`, `latest.yml`, `update-policy.json`) published to canonical Forgejo registry and release hub. 2026-09-16: the published 1.1.0 installer carries no Authenticode signature, so it does not satisfy the release policy; product version moved to `1.2.0` and publication must come from CI with the signing gate GREEN. Download centers in `apps/web` (`/download`) and `site` (`#download`) link the canonical Forgejo feed. 2026-09-19: `1.3.7` (overlay fix) is published to the canonical updater feed (`latest.yml` = 1.3.7, 90.6MiB); the CI signing gate that blocks tag-driven publication is still unresolved, so this went out through the local updater path (GAP-REL-06). 2026-09-23: `1.5.0` published the same way (updater feed `latest.yml` = 1.5.0, installer 91.2MiB / 95,612,108 bytes, sha512 matches the local build and the feed serves it anonymously); the runtime and portable channels were republished first. **2026-11-14 (`v1.17.1`):** Desktop `1.17.1` (`6c6875b`, `D3RO-Voice-Setup-1.17.1-x64.exe` `95,294,301` bytes + Ed25519 signed `latest.yml`/`.sig`), `runtime-1.17.1`/`runtime-latest`, `portable-1.17.1`/`portable-latest`, 및 Landing `site/` (`https://d3ro.chanpaca.net/release-identity.json` = `1.17.1`/`6c6875b`)가 프로덕션에 게시됐다(단 Authenticode SmartScreen 평판 `GAP-REL-02`/`06` 및 Release #339 ~91MB 첨부 `GAP-REL-16` 예외는 유지). | | GAP-REL-08 | Release | 다운로드 센터가 **존재하지 않는 설치 파일**을 가리켰다. `apps/web/src/lib/desktop-release.ts`와 `site/src/release.ts`의 `DESKTOP_VERSION`이 `1.2.0`에 멈춰 있어 설치 URL이 `D3RO-Voice-Setup-1.2.0-x64.exe`였고, 그 경로는 피드에서 404다(실측: 1.2.0=404, 1.3.7=206). `version:sync`가 이 두 표면을 덮지 않아 계속 어긋났다. | `scripts/ci/sync-version.mjs`, `apps/web/src/lib/desktop-release.ts`, `site/src/release.ts` | `[x]` 2026-09-19: 두 다운로드 계약 파일을 `sync-version.mjs` 대상에 추가해 버전·릴리스일이 SSOT에서 자동 반영되도록 하고, 현재 값(1.3.7 / 2026-09-19)으로 정정했다. `version:check`·typecheck·site 빌드 GREEN. | diff --git a/memory/project_status.md b/memory/project_status.md index 37aefc28..ca13cc2c 100644 --- a/memory/project_status.md +++ b/memory/project_status.md @@ -1,5 +1,17 @@ # D3RO-VOICE 프로젝트 현황 +## 화면 글자 읽기(OCR) 오버레이 — CAP-17 (2026-11-20, 구현·검증 완료 / 미커밋·미배포) + +- 요청: 단축키로 화면을 캡처해 덮고, 그 위 글자를 사각형/일반 끌기로 고르기 + PDF 저장 등 부가 기능. +- 구현: `screen-ocr` 재바인딩 액션(기본 `Ctrl+Alt+O`) → 커서가 있는 모니터를 GDI 로 캡처(`utils/win32-screen-capture.ts`) → `screen-ocr` 팝업(포커스 받는 유일한 팝업, 2단계 표시)이 화면을 덮음 → Windows 내장 OCR(`services/screen-ocr/windows-ocr-engine.ts`, powershell.exe -EncodedCommand, 2배 확대) 결과로 단어 상자 선택. +- 선택: 단어에서 끌면 글 흐름, 빈 곳에서 끌면 사각형(영역 모드는 항상 사각형), 더블클릭 단어, 세 번 클릭 줄, Shift+클릭 이어 고르기, Ctrl+A. +- 동작: 복사 / 한 줄 복사 / 이미지 복사 / PNG 저장 / 검색 가능한 PDF(그림 아래 글자 층) / 번역·요약(LlmGateway) / 읽어 주기(TTS) / 링크 열기·웹 검색 / 인식 언어 변경(`screenOcrLanguage`). +- 정본: 순수 로직 `packages/core/src/screen-ocr.ts`, 서비스 `ScreenOcrService.ts`(포트 주입), 조립 `screen-ocr-wiring.ts`, 창 `windows/screen-ocr-overlay.ts`, 문구 41키 × 12로케일. +- 실측: desktopCapturer 는 모니터 3대에서 2.9초(DXGI 500ms 대기 × 모니터) → GDI BitBlt 39ms 로 교체(폴백으로만 남김). 실제 화면 2560×1440 인식 0.76초(프로세스 포함 1.2초). +- 검증: core 34 + 데스크톱 42 신규 테스트, 데스크톱 전체 214 files / 2,933 GREEN(Node 24.19.0), workspace typecheck·lint·hygiene·design·i18n·contract GREEN, 실엔진 프로브, 헤드리스 Electron 43 에서 오버레이(흐름/사각형 선택·복사·번역·PDF 요청·닫힘 정리) + 인쇄한 PDF 글자 층 검색 확인. +- **남은 것:** 실행 중인 앱에서 단축키→포커스→선택 실사용 확인(`GAP-OCR-01`, 그래서 카탈로그는 `[~]`), macOS 엔진 없음(`GAP-OCR-02`, fail-closed 안내). +- 함정: 기본 셸 Node 는 23 — 테스트는 `PATH="$APPDATA/npm:$PATH"`(Node 24.19.0, better-sqlite3 ABI 137)로 돌려야 한다. 한국어 OCR 팩만 있으면 작은 영문 UI 글자를 가끔 오독한다(영어 언어 팩 설치 시 오버레이에서 언어 전환 가능). + ## 학습 어휘를 회의록 전사 교열에 연결 (2026-11-05) - 사용자 보고: "키보드로 치는 건 다 학습하는데 회의록 전사에 활용을 못 한다, 전사가 엉망". diff --git a/packages/core/__tests__/screen-ocr.test.ts b/packages/core/__tests__/screen-ocr.test.ts new file mode 100644 index 00000000..fcaf5db6 --- /dev/null +++ b/packages/core/__tests__/screen-ocr.test.ts @@ -0,0 +1,279 @@ +import { describe, expect, it } from 'vitest' +import { + allWords, + buildSearchablePdfHtml, + chooseOcrScale, + clampRect, + detectLinks, + flowSelection, + lineSelection, + nearestWordIndex, + parseOcrEngineOutput, + pickTranslationTarget, + rectFromPoints, + selectionBounds, + selectionHighlightRects, + selectionText, + wordAtPoint, + wordsInRect, + wordsRelativeTo, + type OcrResult +} from '../src/screen-ocr' + +/** 한 줄에 단어를 폭 40, 간격 10 으로 늘어놓는다. */ +function line(y: number, texts: string[], startX = 10): { words: Array> } { + return { words: texts.map((t, i) => ({ t, x: startX + i * 50, y, w: 40, h: 20 })) } +} + +function sample(): OcrResult { + const result = parseOcrEngineOutput({ + lang: 'ko', + lines: [line(10, ['화면', 'OCR', '기능']), line(50, ['둘째', '줄']), line(90, ['Third', 'line', 'here'])] + }) + if (!result) throw new Error('sample did not parse') + return result +} + +describe('parseOcrEngineOutput', () => { + it('엔진출력해석_정상입력_읽는순서로단어번호와줄범위를매긴다', () => { + const result = sample() + expect(result.language).toBe('ko') + expect(result.words.map((w) => w.text)).toEqual(['화면', 'OCR', '기능', '둘째', '줄', 'Third', 'line', 'here']) + expect(result.words.map((w) => w.index)).toEqual([0, 1, 2, 3, 4, 5, 6, 7]) + expect(result.lines.map((l) => [l.start, l.count])).toEqual([ + [0, 3], + [3, 2], + [5, 3] + ]) + expect(result.lines[0]).toMatchObject({ x: 10, y: 10, width: 140, height: 20 }) + }) + + it('엔진출력해석_모양이틀린최상위_null을돌려줘빈결과로꾸미지않는다', () => { + expect(parseOcrEngineOutput(null)).toBeNull() + expect(parseOcrEngineOutput({ lang: 'ko' })).toBeNull() + expect(parseOcrEngineOutput({ lines: 'nope' })).toBeNull() + }) + + it('엔진출력해석_망가진단어와빈줄_버리고나머지번호는이어진다', () => { + const result = parseOcrEngineOutput({ + lang: 'en', + lines: [ + { words: [{ t: '', x: 0, y: 0, w: 10, h: 10 }, { t: 'ok', x: 'NaN', y: 0, w: 10, h: 10 }] }, + { words: [{ t: 'zero', x: 0, y: 0, w: 0, h: 10 }, { t: 'kept', x: 5, y: 5, w: 10, h: 10 }] }, + 'garbage' + ] + }) + expect(result?.words.map((w) => [w.text, w.index, w.line])).toEqual([['kept', 0, 0]]) + expect(result?.lines).toHaveLength(1) + }) +}) + +describe('사각형 선택', () => { + it('사각형선택_중심이든단어만_살짝스친옆줄은고르지않는다', () => { + const result = sample() + // 첫 줄 전체를 덮고 둘째 줄(y 50~70) 윗부분 4px 만 스친다. + expect(wordsInRect(result, { x: 0, y: 0, width: 200, height: 54 })).toEqual([0, 1, 2]) + }) + + it('사각형선택_절반넘게걸친단어만_절반이안되면고르지않는다', () => { + const result = sample() + // 첫 단어(x 10~50)의 왼쪽 45% 만 덮으면 고르지 않는다. 높이 20 중 12 만 덮어도 폭을 다 덮으면(60%) 고른다. + expect(wordsInRect(result, { x: 0, y: 0, width: 28, height: 40 })).toEqual([]) + expect(wordsInRect(result, { x: 0, y: 12, width: 60, height: 12 })).toEqual([0]) + }) + + it('rectFromPoints_거꾸로끌어도_양수크기사각형이된다', () => { + expect(rectFromPoints({ x: 100, y: 80 }, { x: 20, y: 30 })).toEqual({ x: 20, y: 30, width: 80, height: 50 }) + }) + + it('clampRect_그림밖으로나간영역_안쪽으로자르고완전히밖이면null', () => { + expect(clampRect({ x: -10, y: -5, width: 50.4, height: 30 }, { width: 100, height: 20 })).toEqual({ + x: 0, + y: 0, + width: 41, + height: 20 + }) + expect(clampRect({ x: 200, y: 0, width: 50, height: 10 }, { width: 100, height: 100 })).toBeNull() + expect(clampRect({ x: 10, y: 10, width: 0, height: 0 }, { width: 100, height: 100 })).toBeNull() + }) +}) + +describe('글 흐름 선택', () => { + it('wordAtPoint_단어위와빈곳_단어번호또는null', () => { + const result = sample() + expect(wordAtPoint(result, { x: 70, y: 20 })).toBe(1) + expect(wordAtPoint(result, { x: 300, y: 300 })).toBeNull() + }) + + it('nearestWordIndex_줄사이여백과줄오른쪽바깥_가까운줄의가까운단어를고른다', () => { + const result = sample() + // 둘째 줄(y 50~70) 오른쪽 바깥 → 둘째 줄의 마지막 단어 + expect(nearestWordIndex(result, { x: 400, y: 60 })).toBe(4) + // 셋째 줄 아래 여백 → 셋째 줄에서 가로로 가장 가까운 단어 + expect(nearestWordIndex(result, { x: 65, y: 400 })).toBe(6) + }) + + it('nearestWordIndex_단어가없는결과_null', () => { + const empty = parseOcrEngineOutput({ lang: 'ko', lines: [] }) + expect(empty && nearestWordIndex(empty, { x: 0, y: 0 })).toBeNull() + }) + + it('flowSelection_뒤에서앞으로끌어도_사이단어를전부고른다', () => { + expect(flowSelection(6, 2)).toEqual([2, 3, 4, 5, 6]) + expect(flowSelection(3, 3)).toEqual([3]) + }) + + it('lineSelection과allWords_줄전체와전체단어', () => { + const result = sample() + expect(lineSelection(result, 4)).toEqual([3, 4]) + expect(lineSelection(result, 99)).toEqual([]) + expect(allWords(result)).toHaveLength(8) + }) +}) + +describe('선택 → 텍스트', () => { + it('selectionText_여러줄선택_줄바꿈을유지하고단어는공백으로잇는다', () => { + const result = sample() + expect(selectionText(result, [1, 2, 3, 4, 5])).toBe('OCR 기능\n둘째 줄\nThird') + }) + + it('selectionText_한줄옵션_줄바꿈없이잇는다', () => { + const result = sample() + expect(selectionText(result, [2, 3, 4], { singleLine: true })).toBe('기능 둘째 줄') + }) + + it('selectionText_순서가뒤섞이고겹치고범위밖인번호_정리해서읽는순서로낸다', () => { + const result = sample() + expect(selectionText(result, [4, 0, 0, 99, -1, 3])).toBe('화면\n둘째 줄') + }) + + it('selectionText_일본어중국어처럼글자마다단어인줄_공백없이잇는다', () => { + const result = parseOcrEngineOutput({ + lang: 'ja', + lines: [line(0, ['日', '本', '語', 'OCR', 'テ', 'ス', 'ト']), line(40, ['次', 'の', '行'])] + }) + if (!result) throw new Error('no result') + expect(selectionText(result, allWords(result))).toBe('日本語 OCR テスト\n次の行') + expect(selectionText(result, allWords(result), { singleLine: true })).toBe('日本語 OCR テスト次の行') + }) + + it('selectionText_아무것도안고름_빈문자열', () => { + expect(selectionText(sample(), [])).toBe('') + }) +}) + +describe('강조 사각형과 경계', () => { + it('selectionHighlightRects_같은줄에서이어진단어_하나로잇고끊긴곳은나눈다', () => { + const result = sample() + const rects = selectionHighlightRects(result, [0, 1, 3, 5, 7]) + expect(rects).toEqual([ + { x: 10, y: 10, width: 90, height: 20 }, + { x: 10, y: 50, width: 40, height: 20 }, + { x: 10, y: 90, width: 40, height: 20 }, + { x: 110, y: 90, width: 40, height: 20 } + ]) + }) + + it('selectionBounds_고른단어를모두감싼다_없으면null', () => { + const result = sample() + expect(selectionBounds(result, [1, 4])).toEqual({ x: 60, y: 10, width: 40, height: 60 }) + expect(selectionBounds(result, [])).toBeNull() + }) + + it('wordsRelativeTo_잘라낸영역기준좌표로옮긴다', () => { + const result = sample() + const moved = wordsRelativeTo(result, { x: 50, y: 40, width: 200, height: 40 }) + expect(moved.map((w) => [w.text, w.x, w.y])).toEqual([['줄', 10, 10]]) + }) +}) + +describe('chooseOcrScale', () => { + it('인식배율_100퍼센트화면_2배로키운다', () => { + expect(chooseOcrScale(1920, 1080, 1, 10000)).toBe(2) + }) + + it('인식배율_고배율화면_이미큰글자는덜키운다', () => { + expect(chooseOcrScale(2880, 1800, 1.5, 10000)).toBe(1.33) + expect(chooseOcrScale(3840, 2160, 2, 10000)).toBe(1) + }) + + it('인식배율_엔진최대변길이를넘지않는다', () => { + const scale = chooseOcrScale(7680, 2160, 1, 10000) + expect(scale).toBe(1.3) + expect(7680 * scale).toBeLessThanOrEqual(10000) + }) + + it('인식배율_배율값이이상하면_1배율화면으로본다', () => { + expect(chooseOcrScale(1000, 1000, Number.NaN, 10000)).toBe(2) + expect(chooseOcrScale(0, 0, 1, 10000)).toBe(1) + }) +}) + +describe('detectLinks', () => { + it('링크찾기_주소와메일_열수있는URL로돌려준다', () => { + expect(detectLinks('보기: https://example.com/path?q=1, 그리고 www.d3ro.io. 문의 help@example.com')).toEqual([ + 'mailto:help@example.com', + 'https://example.com/path?q=1', + 'https://www.d3ro.io' + ]) + }) + + it('링크찾기_주소가없는글_빈목록', () => { + expect(detectLinks('그냥 문장입니다 3.14 e.g.')).toEqual([]) + }) + + it('링크찾기_같은주소가여러번_한번만', () => { + expect(detectLinks('https://a.io https://a.io')).toEqual(['https://a.io']) + }) +}) + +describe('pickTranslationTarget', () => { + it('번역대상_외국어글_사용자의앱언어로옮긴다', () => { + expect(pickTranslationTarget('The quick brown fox', 'ko', 'English')).toBe('Korean') + expect(pickTranslationTarget('안녕하세요 반갑습니다', 'en', 'Japanese')).toBe('English') + }) + + it('번역대상_이미사용자언어인글_설정의번역대상언어로옮긴다', () => { + expect(pickTranslationTarget('안녕하세요 반갑습니다', 'ko', 'English')).toBe('English') + expect(pickTranslationTarget('Hello there, friend', 'en', 'Japanese')).toBe('Japanese') + expect(pickTranslationTarget('こんにちは、世界', 'ja', 'English')).toBe('English') + }) + + it('번역대상_모르는앱언어_설정값을그대로쓴다', () => { + expect(pickTranslationTarget('hello', 'xx', 'German')).toBe('German') + }) +}) + +describe('buildSearchablePdfHtml', () => { + const html = buildSearchablePdfHtml({ + imageDataUrl: 'data:image/png;base64,AAAA', + width: 800, + height: 600, + language: 'ko', + words: [ + { text: '화면 &"', x: 10, y: 20, width: 100, height: 20 }, + { text: ' ', x: 0, y: 0, width: 10, height: 10 }, + { text: 'zero', x: 0, y: 0, width: 0, height: 10 } + ] + }) + + it('PDF문서_쪽크기가그림크기와같다', () => { + expect(html).toContain('@page{size:800px 600px;margin:0}') + expect(html).toContain('viewBox="0 0 800 600"') + }) + + it('PDF문서_글자층이그림아래에깔린다', () => { + expect(html.indexOf(' { + expect(html).toContain('textLength="100.0"') + expect(html).toContain('화면 <b>&"') + expect(html).not.toContain('') + }) + + it('PDF문서_빈글자와크기없는단어_글자층에넣지않는다', () => { + expect(html.match(/ { + return typeof value === 'object' && value !== null && !Array.isArray(value) +} + +function finiteNumber(value: unknown): number | null { + return typeof value === 'number' && Number.isFinite(value) ? value : null +} + +function unionRect(rects: readonly OcrRect[]): OcrRect { + let left = Infinity + let top = Infinity + let right = -Infinity + let bottom = -Infinity + for (const r of rects) { + left = Math.min(left, r.x) + top = Math.min(top, r.y) + right = Math.max(right, r.x + r.width) + bottom = Math.max(bottom, r.y + r.height) + } + return { x: left, y: top, width: right - left, height: bottom - top } +} + +/** + * 엔진이 준 `{ lang, lines: [{ words: [{ t, x, y, w, h }] }] }` 를 정본 표현으로 옮긴다. + * 모양이 어긋난 단어(글자 없음 · 좌표가 수가 아님 · 크기 0)는 버리고, 단어가 하나도 안 남은 줄도 버린다. + * 최상위 모양이 틀리면 null — 호출자는 인식 실패로 다룬다(빈 결과로 꾸미지 않는다). + */ +export function parseOcrEngineOutput(raw: unknown): OcrResult | null { + if (!isRecord(raw) || !Array.isArray(raw.lines)) return null + const language = typeof raw.lang === 'string' ? raw.lang : '' + const lines: OcrLine[] = [] + const words: OcrWord[] = [] + + for (const rawLine of raw.lines) { + if (!isRecord(rawLine) || !Array.isArray(rawLine.words)) continue + const start = words.length + for (const rawWord of rawLine.words) { + if (!isRecord(rawWord)) continue + const text = typeof rawWord.t === 'string' ? rawWord.t : '' + const x = finiteNumber(rawWord.x) + const y = finiteNumber(rawWord.y) + const width = finiteNumber(rawWord.w) + const height = finiteNumber(rawWord.h) + if (text.length === 0 || x === null || y === null || width === null || height === null) continue + if (width <= 0 || height <= 0) continue + words.push({ text, x, y, width, height, line: lines.length, index: words.length }) + } + const count = words.length - start + if (count === 0) continue + lines.push({ ...unionRect(words.slice(start)), start, count }) + } + + return { language, lines, words } +} + +// ============================================================ +// 기하 +// ============================================================ + +/** 두 점으로 사각형을 만든다 (끌기 방향과 무관하게 양수 크기). */ +export function rectFromPoints(a: OcrPoint, b: OcrPoint): OcrRect { + return { + x: Math.min(a.x, b.x), + y: Math.min(a.y, b.y), + width: Math.abs(a.x - b.x), + height: Math.abs(a.y - b.y) + } +} + +/** 사각형을 경계 안으로 자르고 정수 픽셀로 맞춘다. 남는 면적이 없으면 null. */ +export function clampRect(rect: OcrRect, bounds: { width: number; height: number }): OcrRect | null { + const left = Math.max(0, Math.floor(rect.x)) + const top = Math.max(0, Math.floor(rect.y)) + const right = Math.min(bounds.width, Math.ceil(rect.x + rect.width)) + const bottom = Math.min(bounds.height, Math.ceil(rect.y + rect.height)) + if (right - left < 1 || bottom - top < 1) return null + return { x: left, y: top, width: right - left, height: bottom - top } +} + +function containsPoint(rect: OcrRect, point: OcrPoint, pad = 0): boolean { + return ( + point.x >= rect.x - pad && + point.x <= rect.x + rect.width + pad && + point.y >= rect.y - pad && + point.y <= rect.y + rect.height + pad + ) +} + +// ============================================================ +// 선택 계산 +// ============================================================ + +/** + * 사각형 선택 — 중심이 사각형 안에 있거나 면적의 절반 이상이 걸친 단어. + * 살짝 스친 옆 줄까지 딸려 오지 않게 "닿기만 하면 선택"으로 하지 않는다. + */ +export function wordsInRect(result: OcrResult, rect: OcrRect): number[] { + const out: number[] = [] + const right = rect.x + rect.width + const bottom = rect.y + rect.height + for (const word of result.words) { + const cx = word.x + word.width / 2 + const cy = word.y + word.height / 2 + if (cx >= rect.x && cx <= right && cy >= rect.y && cy <= bottom) { + out.push(word.index) + continue + } + const overlapW = Math.min(right, word.x + word.width) - Math.max(rect.x, word.x) + const overlapH = Math.min(bottom, word.y + word.height) - Math.max(rect.y, word.y) + if (overlapW > 0 && overlapH > 0 && overlapW * overlapH >= word.width * word.height * 0.5) { + out.push(word.index) + } + } + return out +} + +/** 점 바로 아래(여유 pad 픽셀)의 단어 번호. 없으면 null. */ +export function wordAtPoint(result: OcrResult, point: OcrPoint, pad = 2): number | null { + for (const word of result.words) { + if (containsPoint(word, point, pad)) return word.index + } + return null +} + +/** + * 글 흐름 선택의 끝점 — 점에서 가장 가까운 줄을 고르고, 그 줄 안에서 가장 가까운 단어를 고른다. + * 줄 밖 여백으로 끌어도 선택이 이어지게 하려는 것이다(일반 글 선택과 같은 느낌). + */ +export function nearestWordIndex(result: OcrResult, point: OcrPoint): number | null { + if (result.words.length === 0) return null + let bestLine: OcrLine | null = null + let bestDistance = Infinity + for (const line of result.lines) { + const dy = point.y < line.y ? line.y - point.y : point.y > line.y + line.height ? point.y - (line.y + line.height) : 0 + const dx = point.x < line.x ? line.x - point.x : point.x > line.x + line.width ? point.x - (line.x + line.width) : 0 + // 세로 거리를 훨씬 무겁게 본다 — 같은 높이의 줄이 먼 옆 단(column)이라도 위아래 줄보다 우선이다. + const distance = dy * 4 + dx + if (distance < bestDistance) { + bestDistance = distance + bestLine = line + } + } + if (!bestLine) return null + + let best = bestLine.start + let bestDx = Infinity + for (let i = bestLine.start; i < bestLine.start + bestLine.count; i += 1) { + const word = result.words[i] + const dx = point.x < word.x ? word.x - point.x : point.x > word.x + word.width ? point.x - (word.x + word.width) : 0 + if (dx < bestDx) { + bestDx = dx + best = i + } + } + return best +} + +/** 글 흐름 선택 — 시작 단어부터 끝 단어까지 읽는 순서로 전부. */ +export function flowSelection(anchor: number, focus: number): number[] { + const from = Math.min(anchor, focus) + const to = Math.max(anchor, focus) + const out: number[] = [] + for (let i = from; i <= to; i += 1) out.push(i) + return out +} + +/** 단어가 속한 줄 전체. */ +export function lineSelection(result: OcrResult, wordIndex: number): number[] { + const word = result.words[wordIndex] + if (!word) return [] + const line = result.lines[word.line] + return flowSelection(line.start, line.start + line.count - 1) +} + +export function allWords(result: OcrResult): number[] { + return result.words.map((word) => word.index) +} + +// ============================================================ +// 선택 → 텍스트 +// ============================================================ + +/** 한자 · 히라가나 · 가타카나 — 띄어 쓰지 않는 글자. (한글은 띄어 쓴다) */ +function isUnspacedScript(char: string): boolean { + const code = char.codePointAt(0) ?? 0 + return ( + (code >= 0x3040 && code <= 0x30ff) || // 히라가나 · 가타카나 + (code >= 0x3400 && code <= 0x4dbf) || // 한자 확장 A + (code >= 0x4e00 && code <= 0x9fff) || // 한자 + (code >= 0x3000 && code <= 0x303f) || // CJK 문장부호 + (code >= 0xff00 && code <= 0xffef) // 전각 문자 + ) +} + +function lastChar(text: string): string { + const chars = Array.from(text) + return chars[chars.length - 1] ?? '' +} + +function firstChar(text: string): string { + return Array.from(text)[0] ?? '' +} + +/** 한 줄 안에서 이웃한 두 단어 사이에 공백을 넣을지. 일본어 · 중국어는 엔진이 글자마다 단어로 준다. */ +function joiner(previous: string, next: string): string { + return isUnspacedScript(lastChar(previous)) && isUnspacedScript(firstChar(next)) ? '' : ' ' +} + +export interface SelectionTextOptions { + /** true 면 줄바꿈 없이 한 줄로 잇는다 */ + singleLine?: boolean +} + +/** + * 선택한 단어들을 글로 만든다. 줄은 인식된 줄 순서를 따르고, 같은 줄의 단어는 왼쪽부터 잇는다. + * 번호가 범위를 벗어나거나 겹쳐도 무시한다. + */ +export function selectionText( + result: OcrResult, + selected: readonly number[], + options: SelectionTextOptions = {} +): string { + const picked = Array.from(new Set(selected)) + .filter((index) => Number.isInteger(index) && index >= 0 && index < result.words.length) + .sort((a, b) => a - b) + const lines: string[] = [] + let currentLine = -1 + let buffer = '' + for (const index of picked) { + const word = result.words[index] + if (word.line !== currentLine) { + if (currentLine !== -1) lines.push(buffer) + currentLine = word.line + buffer = word.text + } else { + buffer += joiner(buffer, word.text) + word.text + } + } + if (currentLine !== -1) lines.push(buffer) + + if (!options.singleLine) return lines.join('\n') + let joined = '' + for (const line of lines) { + joined = joined.length === 0 ? line : joined + joiner(joined, line) + line + } + return joined +} + +/** 선택한 단어를 모두 감싸는 사각형. 선택이 비면 null. */ +export function selectionBounds(result: OcrResult, selected: readonly number[]): OcrRect | null { + const rects = selected.map((index) => result.words[index]).filter((word): word is OcrWord => Boolean(word)) + return rects.length === 0 ? null : unionRect(rects) +} + +/** + * 강조 표시용 사각형 — 같은 줄에서 연달아 선택된 단어는 하나로 잇는다(단어 사이 틈이 비어 보이지 않게). + * 높이는 그 줄의 높이로 맞춰 글자마다 들쭉날쭉하지 않게 한다. + */ +export function selectionHighlightRects(result: OcrResult, selected: readonly number[]): OcrRect[] { + const picked = Array.from(new Set(selected)) + .filter((index) => Number.isInteger(index) && index >= 0 && index < result.words.length) + .sort((a, b) => a - b) + const out: OcrRect[] = [] + let runStart = -1 + let previous = -1 + const flush = (): void => { + if (runStart === -1) return + const first = result.words[runStart] + const last = result.words[previous] + const line = result.lines[first.line] + out.push({ x: first.x, y: line.y, width: last.x + last.width - first.x, height: line.height }) + } + for (const index of picked) { + const word = result.words[index] + const continues = previous !== -1 && index === previous + 1 && result.words[previous].line === word.line + if (!continues) { + flush() + runStart = index + } + previous = index + } + flush() + return out +} + +// ============================================================ +// 인식 배율 +// ============================================================ + +/** + * OCR 에 넣기 전 확대 배율. 화면 글자(100% 배율에서 9~10pt)는 그대로 넣으면 인식률이 뚝 떨어지고 + * 2배로 키우면 크게 좋아진다(2026-11-20 실측). 이미 고배율(HiDPI) 화면이면 덜 키운다. + * 엔진의 최대 변 길이를 넘지 않게 줄인다. + */ +export function chooseOcrScale( + width: number, + height: number, + displayScaleFactor: number, + maxDimension: number +): number { + const factor = Number.isFinite(displayScaleFactor) && displayScaleFactor > 0 ? displayScaleFactor : 1 + const wanted = Math.max(1, 2 / factor) + const longest = Math.max(width, height) + if (longest <= 0) return 1 + const allowed = maxDimension / longest + const scale = Math.min(wanted, allowed) + return Math.round(scale * 100) / 100 +} + +// ============================================================ +// 링크 · 번역 대상 +// ============================================================ + +const URL_PATTERN = /\b((?:https?:\/\/|www\.)[^\s<>"'`]+[^\s<>"'`.,;:!?)\]}])/giu +const EMAIL_PATTERN = /\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}\b/giu + +/** 글에서 열 수 있는 주소를 찾는다 (http/https, www. 로 시작하는 주소, 메일 주소). 중복은 뺀다. */ +export function detectLinks(text: string): string[] { + const out: string[] = [] + const add = (value: string): void => { + if (!out.includes(value)) out.push(value) + } + for (const match of text.matchAll(EMAIL_PATTERN)) add(`mailto:${match[0]}`) + for (const match of text.matchAll(URL_PATTERN)) { + const raw = match[1] + add(/^https?:\/\//iu.test(raw) ? raw : `https://${raw}`) + } + return out +} + +/** 앱 언어 코드 → 번역 프롬프트에 넣을 언어 이름. */ +const LOCALE_LANGUAGE_NAMES: Readonly> = { + ko: 'Korean', + en: 'English', + ja: 'Japanese', + zh: 'Chinese (Simplified)', + 'zh-TW': 'Chinese (Traditional)', + es: 'Spanish', + fr: 'French', + de: 'German', + pt: 'Portuguese', + ru: 'Russian', + th: 'Thai', + vi: 'Vietnamese' +} + +function scriptShare(text: string, test: (code: number) => boolean): number { + let letters = 0 + let hits = 0 + for (const char of text) { + if (!/\p{L}/u.test(char)) continue + letters += 1 + if (test(char.codePointAt(0) ?? 0)) hits += 1 + } + return letters === 0 ? 0 : hits / letters +} + +const isHangul = (code: number): boolean => (code >= 0xac00 && code <= 0xd7a3) || (code >= 0x1100 && code <= 0x11ff) +const isKana = (code: number): boolean => code >= 0x3040 && code <= 0x30ff +const isHan = (code: number): boolean => (code >= 0x4e00 && code <= 0x9fff) || (code >= 0x3400 && code <= 0x4dbf) +const isCyrillic = (code: number): boolean => code >= 0x0400 && code <= 0x04ff +const isThai = (code: number): boolean => code >= 0x0e00 && code <= 0x0e7f +const isLatin = (code: number): boolean => + (code >= 0x41 && code <= 0x5a) || (code >= 0x61 && code <= 0x7a) || (code >= 0xc0 && code <= 0x24f) + +/** 글이 이미 사용자의 언어로 보이는지 (문자 체계로만 어림한다). */ +function looksLikeLocale(text: string, locale: string): boolean { + switch (locale) { + case 'ko': + return scriptShare(text, isHangul) >= 0.3 + case 'ja': + return scriptShare(text, isKana) >= 0.15 + case 'zh': + case 'zh-TW': + return scriptShare(text, isHan) >= 0.5 && scriptShare(text, isKana) < 0.05 + case 'ru': + return scriptShare(text, isCyrillic) >= 0.5 + case 'th': + return scriptShare(text, isThai) >= 0.5 + default: + // 라틴 문자 언어끼리는 문자로 구분할 수 없다 — 라틴 글이면 "이미 사용자 언어"로 본다. + return scriptShare(text, isLatin) >= 0.7 + } +} + +/** + * 번역 대상 언어를 고른다 — 기본은 사용자의 앱 언어로 옮긴다. + * 글이 이미 그 언어로 보이면 설정의 번역 대상 언어(configuredTarget)로 옮긴다. + */ +export function pickTranslationTarget(text: string, appLocale: string, configuredTarget: string): string { + const own = LOCALE_LANGUAGE_NAMES[appLocale] + if (!own) return configuredTarget + return looksLikeLocale(text, appLocale) ? configuredTarget : own +} + +// ============================================================ +// 검색 가능한 PDF 문서 +// ============================================================ + +function escapeHtml(value: string): string { + return value + .replace(/&/gu, '&') + .replace(//gu, '>') + .replace(/"/gu, '"') +} + +export interface SearchablePdfInput { + /** 그림의 data URL (image/png) */ + imageDataUrl: string + /** 그림 크기 (물리 픽셀) — 쪽 크기가 된다 */ + width: number + height: number + /** 그림 좌표계의 단어 (잘라낸 영역이면 그 영역 기준으로 옮긴 좌표) */ + words: ReadonlyArray> + /** 문서 lang 속성 */ + language: string +} + +/** + * 그림 한 장 + 글자 층으로 된 한 쪽짜리 HTML — printToPDF 로 넘기면 검색·복사가 되는 PDF 가 된다. + * + * 글자를 투명하게 칠하지 않고 **그림 아래에** 깐다: 투명 글자는 PDF 백엔드가 그리지 않고 버릴 수 있지만, + * 가려진 글자는 내용 스트림에 그대로 남아 보기 프로그램이 찾고 고를 수 있다. + * 글자 폭은 SVG textLength 로 단어 상자에 정확히 맞춘다(측정용 스크립트가 필요 없다). + */ +export function buildSearchablePdfHtml(input: SearchablePdfInput): string { + const width = Math.max(1, Math.round(input.width)) + const height = Math.max(1, Math.round(input.height)) + const texts = input.words + .filter((word) => word.text.trim().length > 0 && word.width > 0 && word.height > 0) + .map((word) => { + const fontSize = Math.max(1, word.height * 0.9) + const baseline = word.y + word.height * 0.82 + return ( + `${escapeHtml(word.text)}` + ) + }) + .join('') + + return ( + `` + + `` + + `
` + + `${texts}` + + `` + + `
` + ) +} + +/** 사각형 안에 중심이 든 단어를 그 사각형 기준 좌표로 옮긴다 (영역만 PDF/그림으로 낼 때). */ +export function wordsRelativeTo(result: OcrResult, rect: OcrRect): OcrWord[] { + return wordsInRect(result, rect).map((index) => { + const word = result.words[index] + return { ...word, x: word.x - rect.x, y: word.y - rect.y } + }) +} diff --git a/packages/core/src/types.ts b/packages/core/src/types.ts index d73f8a74..c383cfad 100644 --- a/packages/core/src/types.ts +++ b/packages/core/src/types.ts @@ -443,6 +443,8 @@ export interface AppConfig { handsFreeEnabled: boolean /** 스크린 컨텍스트 캡처 활성화 (Phase 10.2) */ screenContextEnabled: boolean + /** 화면 OCR 인식 언어 태그 (예: 'ko', 'en-US'). 'auto' 면 Windows 사용자 언어를 따른다. */ + screenOcrLanguage: string /** Phase 15.5: HuggingFace 토큰 (화자 구분용) */ hfToken: string /** Phase 15.5: 화자 구분 활성화 */ diff --git a/packages/i18n/src/locales/de.json b/packages/i18n/src/locales/de.json index fd7e8b2c..596998b5 100644 --- a/packages/i18n/src/locales/de.json +++ b/packages/i18n/src/locales/de.json @@ -2500,5 +2500,46 @@ "web.push.unconfigured": "Benachrichtigungen sind auf dem Server nicht konfiguriert.", "web.push.blocked": "Benachrichtigungen sind blockiert. Erlaube sie in den Website-Einstellungen des Browsers.", "web.push.revoked": "Dieser Browser wurde aus dem Konto entfernt. Lösche die gespeicherten Websitedaten, bevor du ihn erneut hinzufügst.", - "web.push.error": "Benachrichtigungen konnten nicht geändert werden. Prüfe die Browserberechtigung und versuche es erneut." + "web.push.error": "Benachrichtigungen konnten nicht geändert werden. Prüfe die Browserberechtigung und versuche es erneut.", + "keybinding.action.screenOcr": "Bildschirmtext erfassen (OCR)", + "keybinding.action.screenOcr.desc": "Nimmt den Bildschirm auf, damit Sie den Text darauf durch Ziehen auswählen können.", + "screenOcr.dialog.savePdf": "Als PDF speichern", + "screenOcr.dialog.saveImage": "Als Bild speichern", + "screenOcr.errorCapture": "Der Bildschirm konnte nicht aufgenommen werden.", + "screenOcr.modeText": "Text", + "screenOcr.modeRect": "Bereich", + "screenOcr.copy": "Kopieren", + "screenOcr.copyLine": "Als eine Zeile kopieren", + "screenOcr.copyImage": "Bild kopieren", + "screenOcr.saveImage": "Bild speichern", + "screenOcr.savePdf": "PDF speichern", + "screenOcr.translate": "Übersetzen", + "screenOcr.summarize": "Zusammenfassen", + "screenOcr.speak": "Vorlesen", + "screenOcr.stopSpeak": "Vorlesen stoppen", + "screenOcr.openLink": "Link öffnen", + "screenOcr.search": "Im Web suchen", + "screenOcr.selectAll": "Alles auswählen", + "screenOcr.close": "Schließen", + "screenOcr.language": "Sprache", + "screenOcr.languageAuto": "Automatisch", + "screenOcr.statusRecognizing": "Text wird gelesen…", + "screenOcr.statusReady": "{{count}} Wörter · zum Auswählen ziehen", + "screenOcr.statusEmpty": "Kein Text gefunden", + "screenOcr.statusSelected": "{{count}} Zeichen ausgewählt", + "screenOcr.hint": "Ziehen zum Auswählen · Doppelklick auf ein Wort · Strg+A alles · Strg+C kopieren · Esc schließen", + "screenOcr.toastCopied": "Kopiert", + "screenOcr.toastImageCopied": "Bild kopiert", + "screenOcr.toastSaved": "Gespeichert", + "screenOcr.working": "Wird verarbeitet…", + "screenOcr.resultTranslate": "Übersetzung", + "screenOcr.resultSummary": "Zusammenfassung", + "screenOcr.errorRecognition": "Der Text konnte nicht gelesen werden", + "screenOcr.errorUnavailable": "Die Texterfassung ist auf diesem PC nicht verfügbar. Installieren Sie in den Windows-Einstellungen ein Sprachpaket mit optischer Zeichenerkennung.", + "screenOcr.errorExport": "Speichern oder Kopieren fehlgeschlagen", + "screenOcr.errorLlm": "Keine Antwort von der KI. Prüfen Sie die Modelleinstellungen.", + "screenOcr.errorTts": "Vorlesen fehlgeschlagen", + "screenOcr.errorLink": "Der Link konnte nicht geöffnet werden", + "screenOcr.errorInvalid": "Diese Aufnahme wurde bereits geschlossen. Nehmen Sie erneut auf.", + "screenOcr.errorEmpty": "Kein Text ausgewählt" } diff --git a/packages/i18n/src/locales/en.json b/packages/i18n/src/locales/en.json index b6731bde..84f383a6 100644 --- a/packages/i18n/src/locales/en.json +++ b/packages/i18n/src/locales/en.json @@ -2500,5 +2500,46 @@ "web.push.unconfigured": "Notifications are not configured on the server.", "web.push.blocked": "Notifications are blocked. Allow them in this site's browser settings.", "web.push.revoked": "This browser was removed from the account. Clear this site's stored data before adding it again.", - "web.push.error": "Notifications could not be changed. Check the browser permission and try again." + "web.push.error": "Notifications could not be changed. Check the browser permission and try again.", + "keybinding.action.screenOcr": "Screen text capture (OCR)", + "keybinding.action.screenOcr.desc": "Captures the screen so you can drag to select the text on it.", + "screenOcr.dialog.savePdf": "Save as PDF", + "screenOcr.dialog.saveImage": "Save as image", + "screenOcr.errorCapture": "Could not capture the screen.", + "screenOcr.modeText": "Text", + "screenOcr.modeRect": "Area", + "screenOcr.copy": "Copy", + "screenOcr.copyLine": "Copy as one line", + "screenOcr.copyImage": "Copy image", + "screenOcr.saveImage": "Save image", + "screenOcr.savePdf": "Save PDF", + "screenOcr.translate": "Translate", + "screenOcr.summarize": "Summarize", + "screenOcr.speak": "Read aloud", + "screenOcr.stopSpeak": "Stop reading", + "screenOcr.openLink": "Open link", + "screenOcr.search": "Search the web", + "screenOcr.selectAll": "Select all", + "screenOcr.close": "Close", + "screenOcr.language": "Language", + "screenOcr.languageAuto": "Auto", + "screenOcr.statusRecognizing": "Reading text…", + "screenOcr.statusReady": "{{count}} words · drag to select", + "screenOcr.statusEmpty": "No text found", + "screenOcr.statusSelected": "{{count}} characters selected", + "screenOcr.hint": "Drag to select · double-click a word · Ctrl+A all · Ctrl+C copy · Esc close", + "screenOcr.toastCopied": "Copied", + "screenOcr.toastImageCopied": "Image copied", + "screenOcr.toastSaved": "Saved", + "screenOcr.working": "Working…", + "screenOcr.resultTranslate": "Translation", + "screenOcr.resultSummary": "Summary", + "screenOcr.errorRecognition": "Could not read the text", + "screenOcr.errorUnavailable": "Screen text capture is not available on this PC. Install a language pack with optical character recognition in Windows Settings.", + "screenOcr.errorExport": "Could not save or copy", + "screenOcr.errorLlm": "No response from the AI. Check your model settings.", + "screenOcr.errorTts": "Could not read aloud", + "screenOcr.errorLink": "Could not open the link", + "screenOcr.errorInvalid": "This capture is already closed. Capture again.", + "screenOcr.errorEmpty": "No text selected" } diff --git a/packages/i18n/src/locales/es.json b/packages/i18n/src/locales/es.json index 1e169cb1..acb347ea 100644 --- a/packages/i18n/src/locales/es.json +++ b/packages/i18n/src/locales/es.json @@ -2500,5 +2500,46 @@ "web.push.unconfigured": "Las notificaciones no están configuradas en el servidor.", "web.push.blocked": "Las notificaciones están bloqueadas. Permítelas en la configuración del sitio del navegador.", "web.push.revoked": "Este navegador se eliminó de la cuenta. Borra los datos guardados del sitio antes de volver a añadirlo.", - "web.push.error": "No se pudieron cambiar las notificaciones. Revisa el permiso del navegador e inténtalo de nuevo." + "web.push.error": "No se pudieron cambiar las notificaciones. Revisa el permiso del navegador e inténtalo de nuevo.", + "keybinding.action.screenOcr": "Capturar texto de pantalla (OCR)", + "keybinding.action.screenOcr.desc": "Captura la pantalla para seleccionar su texto arrastrando.", + "screenOcr.dialog.savePdf": "Guardar como PDF", + "screenOcr.dialog.saveImage": "Guardar como imagen", + "screenOcr.errorCapture": "No se pudo capturar la pantalla.", + "screenOcr.modeText": "Texto", + "screenOcr.modeRect": "Área", + "screenOcr.copy": "Copiar", + "screenOcr.copyLine": "Copiar en una línea", + "screenOcr.copyImage": "Copiar imagen", + "screenOcr.saveImage": "Guardar imagen", + "screenOcr.savePdf": "Guardar PDF", + "screenOcr.translate": "Traducir", + "screenOcr.summarize": "Resumir", + "screenOcr.speak": "Leer en voz alta", + "screenOcr.stopSpeak": "Detener lectura", + "screenOcr.openLink": "Abrir enlace", + "screenOcr.search": "Buscar en la web", + "screenOcr.selectAll": "Seleccionar todo", + "screenOcr.close": "Cerrar", + "screenOcr.language": "Idioma", + "screenOcr.languageAuto": "Automático", + "screenOcr.statusRecognizing": "Leyendo texto…", + "screenOcr.statusReady": "{{count}} palabras · arrastra para seleccionar", + "screenOcr.statusEmpty": "No se encontró texto", + "screenOcr.statusSelected": "{{count}} caracteres seleccionados", + "screenOcr.hint": "Arrastra para seleccionar · doble clic en una palabra · Ctrl+A todo · Ctrl+C copiar · Esc cerrar", + "screenOcr.toastCopied": "Copiado", + "screenOcr.toastImageCopied": "Imagen copiada", + "screenOcr.toastSaved": "Guardado", + "screenOcr.working": "Procesando…", + "screenOcr.resultTranslate": "Traducción", + "screenOcr.resultSummary": "Resumen", + "screenOcr.errorRecognition": "No se pudo leer el texto", + "screenOcr.errorUnavailable": "La captura de texto no está disponible en este PC. Instala un paquete de idioma con reconocimiento óptico de caracteres en Configuración de Windows.", + "screenOcr.errorExport": "No se pudo guardar ni copiar", + "screenOcr.errorLlm": "La IA no respondió. Revisa la configuración del modelo.", + "screenOcr.errorTts": "No se pudo leer en voz alta", + "screenOcr.errorLink": "No se pudo abrir el enlace", + "screenOcr.errorInvalid": "Esta captura ya se cerró. Vuelve a capturar.", + "screenOcr.errorEmpty": "No hay texto seleccionado" } diff --git a/packages/i18n/src/locales/fr.json b/packages/i18n/src/locales/fr.json index 5e50fcd5..4a696668 100644 --- a/packages/i18n/src/locales/fr.json +++ b/packages/i18n/src/locales/fr.json @@ -2500,5 +2500,46 @@ "web.push.unconfigured": "Les notifications ne sont pas configurées sur le serveur.", "web.push.blocked": "Les notifications sont bloquées. Autorisez-les dans les paramètres du site du navigateur.", "web.push.revoked": "Ce navigateur a été retiré du compte. Effacez les données enregistrées du site avant de l’ajouter à nouveau.", - "web.push.error": "Impossible de modifier les notifications. Vérifiez l’autorisation du navigateur et réessayez." + "web.push.error": "Impossible de modifier les notifications. Vérifiez l’autorisation du navigateur et réessayez.", + "keybinding.action.screenOcr": "Capture de texte à l’écran (OCR)", + "keybinding.action.screenOcr.desc": "Capture l’écran pour sélectionner son texte en faisant glisser.", + "screenOcr.dialog.savePdf": "Enregistrer en PDF", + "screenOcr.dialog.saveImage": "Enregistrer comme image", + "screenOcr.errorCapture": "Impossible de capturer l’écran.", + "screenOcr.modeText": "Texte", + "screenOcr.modeRect": "Zone", + "screenOcr.copy": "Copier", + "screenOcr.copyLine": "Copier sur une ligne", + "screenOcr.copyImage": "Copier l’image", + "screenOcr.saveImage": "Enregistrer l’image", + "screenOcr.savePdf": "Enregistrer en PDF", + "screenOcr.translate": "Traduire", + "screenOcr.summarize": "Résumer", + "screenOcr.speak": "Lire à voix haute", + "screenOcr.stopSpeak": "Arrêter la lecture", + "screenOcr.openLink": "Ouvrir le lien", + "screenOcr.search": "Rechercher sur le Web", + "screenOcr.selectAll": "Tout sélectionner", + "screenOcr.close": "Fermer", + "screenOcr.language": "Langue", + "screenOcr.languageAuto": "Auto", + "screenOcr.statusRecognizing": "Lecture du texte…", + "screenOcr.statusReady": "{{count}} mots · faites glisser pour sélectionner", + "screenOcr.statusEmpty": "Aucun texte trouvé", + "screenOcr.statusSelected": "{{count}} caractères sélectionnés", + "screenOcr.hint": "Glisser pour sélectionner · double-clic sur un mot · Ctrl+A tout · Ctrl+C copier · Échap fermer", + "screenOcr.toastCopied": "Copié", + "screenOcr.toastImageCopied": "Image copiée", + "screenOcr.toastSaved": "Enregistré", + "screenOcr.working": "Traitement…", + "screenOcr.resultTranslate": "Traduction", + "screenOcr.resultSummary": "Résumé", + "screenOcr.errorRecognition": "Impossible de lire le texte", + "screenOcr.errorUnavailable": "La capture de texte n’est pas disponible sur ce PC. Installez un module linguistique avec reconnaissance optique de caractères dans les Paramètres Windows.", + "screenOcr.errorExport": "Impossible d’enregistrer ou de copier", + "screenOcr.errorLlm": "Aucune réponse de l’IA. Vérifiez les réglages du modèle.", + "screenOcr.errorTts": "Impossible de lire à voix haute", + "screenOcr.errorLink": "Impossible d’ouvrir le lien", + "screenOcr.errorInvalid": "Cette capture est déjà fermée. Recommencez la capture.", + "screenOcr.errorEmpty": "Aucun texte sélectionné" } diff --git a/packages/i18n/src/locales/ja.json b/packages/i18n/src/locales/ja.json index e07fc617..9395d6bd 100644 --- a/packages/i18n/src/locales/ja.json +++ b/packages/i18n/src/locales/ja.json @@ -2500,5 +2500,46 @@ "web.push.unconfigured": "サーバーに通知機能が設定されていません。", "web.push.blocked": "通知がブロックされています。ブラウザのサイト設定で許可してください。", "web.push.revoked": "このブラウザはアカウントから削除されました。再度追加する前に、このサイトの保存データを消去してください。", - "web.push.error": "通知設定を変更できませんでした。ブラウザの権限を確認して、もう一度お試しください。" + "web.push.error": "通知設定を変更できませんでした。ブラウザの権限を確認して、もう一度お試しください。", + "keybinding.action.screenOcr": "画面の文字を読み取る (OCR)", + "keybinding.action.screenOcr.desc": "画面をキャプチャし、その上の文字をドラッグして選択します。", + "screenOcr.dialog.savePdf": "PDF として保存", + "screenOcr.dialog.saveImage": "画像として保存", + "screenOcr.errorCapture": "画面をキャプチャできませんでした。", + "screenOcr.modeText": "テキスト選択", + "screenOcr.modeRect": "範囲選択", + "screenOcr.copy": "コピー", + "screenOcr.copyLine": "1 行でコピー", + "screenOcr.copyImage": "画像をコピー", + "screenOcr.saveImage": "画像を保存", + "screenOcr.savePdf": "PDF を保存", + "screenOcr.translate": "翻訳", + "screenOcr.summarize": "要約", + "screenOcr.speak": "読み上げ", + "screenOcr.stopSpeak": "読み上げを停止", + "screenOcr.openLink": "リンクを開く", + "screenOcr.search": "ウェブ検索", + "screenOcr.selectAll": "すべて選択", + "screenOcr.close": "閉じる", + "screenOcr.language": "認識言語", + "screenOcr.languageAuto": "自動", + "screenOcr.statusRecognizing": "文字を読み取り中…", + "screenOcr.statusReady": "{{count}} 語 · ドラッグして選択", + "screenOcr.statusEmpty": "文字が見つかりませんでした", + "screenOcr.statusSelected": "{{count}} 文字を選択", + "screenOcr.hint": "ドラッグで選択 · ダブルクリックで単語 · Ctrl+A すべて · Ctrl+C コピー · Esc 閉じる", + "screenOcr.toastCopied": "コピーしました", + "screenOcr.toastImageCopied": "画像をコピーしました", + "screenOcr.toastSaved": "保存しました", + "screenOcr.working": "処理中…", + "screenOcr.resultTranslate": "翻訳結果", + "screenOcr.resultSummary": "要約結果", + "screenOcr.errorRecognition": "文字を読み取れませんでした", + "screenOcr.errorUnavailable": "この PC では画面の文字読み取りを使用できません。Windows の設定で言語パック(光学式文字認識)をインストールしてください。", + "screenOcr.errorExport": "保存またはコピーできませんでした", + "screenOcr.errorLlm": "AI から応答がありませんでした。モデル設定を確認してください。", + "screenOcr.errorTts": "読み上げできませんでした", + "screenOcr.errorLink": "リンクを開けませんでした", + "screenOcr.errorInvalid": "このキャプチャはすでに閉じられています。もう一度キャプチャしてください。", + "screenOcr.errorEmpty": "テキストが選択されていません" } diff --git a/packages/i18n/src/locales/ko.json b/packages/i18n/src/locales/ko.json index 285f7cc5..3ebcc69c 100644 --- a/packages/i18n/src/locales/ko.json +++ b/packages/i18n/src/locales/ko.json @@ -2500,5 +2500,46 @@ "web.push.unconfigured": "서버에 알림 기능이 설정되지 않았습니다.", "web.push.blocked": "브라우저에서 알림이 차단되었습니다. 사이트 설정에서 알림을 허용해 주세요.", "web.push.revoked": "이 브라우저가 계정에서 해제되었습니다. 다시 추가하려면 이 사이트의 저장 데이터를 삭제해 주세요.", - "web.push.error": "알림 설정을 바꾸지 못했습니다. 브라우저 권한을 확인한 뒤 다시 시도해 주세요." + "web.push.error": "알림 설정을 바꾸지 못했습니다. 브라우저 권한을 확인한 뒤 다시 시도해 주세요.", + "keybinding.action.screenOcr": "화면 글자 읽기 (OCR)", + "keybinding.action.screenOcr.desc": "화면을 캡처해 그 위의 글자를 끌어서 고릅니다.", + "screenOcr.dialog.savePdf": "PDF로 저장", + "screenOcr.dialog.saveImage": "이미지로 저장", + "screenOcr.errorCapture": "화면을 캡처하지 못했습니다.", + "screenOcr.modeText": "글자 선택", + "screenOcr.modeRect": "영역 선택", + "screenOcr.copy": "복사", + "screenOcr.copyLine": "한 줄로 복사", + "screenOcr.copyImage": "이미지 복사", + "screenOcr.saveImage": "이미지 저장", + "screenOcr.savePdf": "PDF 저장", + "screenOcr.translate": "번역", + "screenOcr.summarize": "요약", + "screenOcr.speak": "읽어 주기", + "screenOcr.stopSpeak": "읽기 중지", + "screenOcr.openLink": "링크 열기", + "screenOcr.search": "웹 검색", + "screenOcr.selectAll": "전체 선택", + "screenOcr.close": "닫기", + "screenOcr.language": "인식 언어", + "screenOcr.languageAuto": "자동", + "screenOcr.statusRecognizing": "글자를 읽는 중…", + "screenOcr.statusReady": "단어 {{count}}개 · 끌어서 고르세요", + "screenOcr.statusEmpty": "글자를 찾지 못했습니다", + "screenOcr.statusSelected": "{{count}}자 선택함", + "screenOcr.hint": "끌어서 선택 · 더블클릭 단어 · Ctrl+A 전체 · Ctrl+C 복사 · Esc 닫기", + "screenOcr.toastCopied": "복사함", + "screenOcr.toastImageCopied": "이미지를 복사함", + "screenOcr.toastSaved": "저장함", + "screenOcr.working": "처리 중…", + "screenOcr.resultTranslate": "번역 결과", + "screenOcr.resultSummary": "요약 결과", + "screenOcr.errorRecognition": "글자를 읽지 못했습니다", + "screenOcr.errorUnavailable": "이 PC에서는 화면 글자 읽기를 쓸 수 없습니다. Windows 설정에서 언어 팩(광학 문자 인식)을 설치해 주세요.", + "screenOcr.errorExport": "저장하거나 복사하지 못했습니다", + "screenOcr.errorLlm": "AI 응답을 받지 못했습니다. 모델 설정을 확인해 주세요.", + "screenOcr.errorTts": "읽어 주지 못했습니다", + "screenOcr.errorLink": "링크를 열지 못했습니다", + "screenOcr.errorInvalid": "이 캡처는 이미 닫혔습니다. 다시 캡처해 주세요.", + "screenOcr.errorEmpty": "선택한 글자가 없습니다" } diff --git a/packages/i18n/src/locales/pt.json b/packages/i18n/src/locales/pt.json index 317b5a32..578c5bc6 100644 --- a/packages/i18n/src/locales/pt.json +++ b/packages/i18n/src/locales/pt.json @@ -2500,5 +2500,46 @@ "web.push.unconfigured": "As notificações não estão configuradas no servidor.", "web.push.blocked": "As notificações estão bloqueadas. Permita-as nas configurações do site no navegador.", "web.push.revoked": "Este navegador foi removido da conta. Limpe os dados salvos deste site antes de adicioná-lo novamente.", - "web.push.error": "Não foi possível alterar as notificações. Verifique a permissão do navegador e tente novamente." + "web.push.error": "Não foi possível alterar as notificações. Verifique a permissão do navegador e tente novamente.", + "keybinding.action.screenOcr": "Capturar texto da tela (OCR)", + "keybinding.action.screenOcr.desc": "Captura a tela para você selecionar o texto arrastando.", + "screenOcr.dialog.savePdf": "Salvar como PDF", + "screenOcr.dialog.saveImage": "Salvar como imagem", + "screenOcr.errorCapture": "Não foi possível capturar a tela.", + "screenOcr.modeText": "Texto", + "screenOcr.modeRect": "Área", + "screenOcr.copy": "Copiar", + "screenOcr.copyLine": "Copiar em uma linha", + "screenOcr.copyImage": "Copiar imagem", + "screenOcr.saveImage": "Salvar imagem", + "screenOcr.savePdf": "Salvar PDF", + "screenOcr.translate": "Traduzir", + "screenOcr.summarize": "Resumir", + "screenOcr.speak": "Ler em voz alta", + "screenOcr.stopSpeak": "Parar leitura", + "screenOcr.openLink": "Abrir link", + "screenOcr.search": "Pesquisar na web", + "screenOcr.selectAll": "Selecionar tudo", + "screenOcr.close": "Fechar", + "screenOcr.language": "Idioma", + "screenOcr.languageAuto": "Automático", + "screenOcr.statusRecognizing": "Lendo o texto…", + "screenOcr.statusReady": "{{count}} palavras · arraste para selecionar", + "screenOcr.statusEmpty": "Nenhum texto encontrado", + "screenOcr.statusSelected": "{{count}} caracteres selecionados", + "screenOcr.hint": "Arraste para selecionar · clique duplo em uma palavra · Ctrl+A tudo · Ctrl+C copiar · Esc fechar", + "screenOcr.toastCopied": "Copiado", + "screenOcr.toastImageCopied": "Imagem copiada", + "screenOcr.toastSaved": "Salvo", + "screenOcr.working": "Processando…", + "screenOcr.resultTranslate": "Tradução", + "screenOcr.resultSummary": "Resumo", + "screenOcr.errorRecognition": "Não foi possível ler o texto", + "screenOcr.errorUnavailable": "A captura de texto não está disponível neste PC. Instale um pacote de idioma com reconhecimento óptico de caracteres nas Configurações do Windows.", + "screenOcr.errorExport": "Não foi possível salvar ou copiar", + "screenOcr.errorLlm": "A IA não respondeu. Verifique as configurações do modelo.", + "screenOcr.errorTts": "Não foi possível ler em voz alta", + "screenOcr.errorLink": "Não foi possível abrir o link", + "screenOcr.errorInvalid": "Esta captura já foi fechada. Capture novamente.", + "screenOcr.errorEmpty": "Nenhum texto selecionado" } diff --git a/packages/i18n/src/locales/ru.json b/packages/i18n/src/locales/ru.json index 3d0073c6..763a385f 100644 --- a/packages/i18n/src/locales/ru.json +++ b/packages/i18n/src/locales/ru.json @@ -2500,5 +2500,46 @@ "web.push.unconfigured": "Уведомления не настроены на сервере.", "web.push.blocked": "Уведомления заблокированы. Разрешите их в настройках сайта в браузере.", "web.push.revoked": "Этот браузер удалён из аккаунта. Перед повторным добавлением очистите сохранённые данные сайта.", - "web.push.error": "Не удалось изменить уведомления. Проверьте разрешение браузера и повторите попытку." + "web.push.error": "Не удалось изменить уведомления. Проверьте разрешение браузера и повторите попытку.", + "keybinding.action.screenOcr": "Распознавание текста на экране (OCR)", + "keybinding.action.screenOcr.desc": "Делает снимок экрана, чтобы выделять текст на нём перетаскиванием.", + "screenOcr.dialog.savePdf": "Сохранить как PDF", + "screenOcr.dialog.saveImage": "Сохранить как изображение", + "screenOcr.errorCapture": "Не удалось сделать снимок экрана.", + "screenOcr.modeText": "Текст", + "screenOcr.modeRect": "Область", + "screenOcr.copy": "Копировать", + "screenOcr.copyLine": "Копировать одной строкой", + "screenOcr.copyImage": "Копировать изображение", + "screenOcr.saveImage": "Сохранить изображение", + "screenOcr.savePdf": "Сохранить PDF", + "screenOcr.translate": "Перевести", + "screenOcr.summarize": "Кратко", + "screenOcr.speak": "Прочитать вслух", + "screenOcr.stopSpeak": "Остановить чтение", + "screenOcr.openLink": "Открыть ссылку", + "screenOcr.search": "Искать в интернете", + "screenOcr.selectAll": "Выделить всё", + "screenOcr.close": "Закрыть", + "screenOcr.language": "Язык", + "screenOcr.languageAuto": "Авто", + "screenOcr.statusRecognizing": "Распознаём текст…", + "screenOcr.statusReady": "Слов: {{count}} · выделяйте перетаскиванием", + "screenOcr.statusEmpty": "Текст не найден", + "screenOcr.statusSelected": "Выделено символов: {{count}}", + "screenOcr.hint": "Перетащите для выделения · двойной щелчок — слово · Ctrl+A всё · Ctrl+C копировать · Esc закрыть", + "screenOcr.toastCopied": "Скопировано", + "screenOcr.toastImageCopied": "Изображение скопировано", + "screenOcr.toastSaved": "Сохранено", + "screenOcr.working": "Обработка…", + "screenOcr.resultTranslate": "Перевод", + "screenOcr.resultSummary": "Краткое содержание", + "screenOcr.errorRecognition": "Не удалось распознать текст", + "screenOcr.errorUnavailable": "Распознавание текста на этом ПК недоступно. Установите в параметрах Windows языковой пакет с оптическим распознаванием символов.", + "screenOcr.errorExport": "Не удалось сохранить или скопировать", + "screenOcr.errorLlm": "ИИ не ответил. Проверьте настройки модели.", + "screenOcr.errorTts": "Не удалось прочитать вслух", + "screenOcr.errorLink": "Не удалось открыть ссылку", + "screenOcr.errorInvalid": "Этот снимок уже закрыт. Сделайте новый.", + "screenOcr.errorEmpty": "Текст не выделен" } diff --git a/packages/i18n/src/locales/th.json b/packages/i18n/src/locales/th.json index 4bd9214f..83cce1c0 100644 --- a/packages/i18n/src/locales/th.json +++ b/packages/i18n/src/locales/th.json @@ -2500,5 +2500,46 @@ "web.push.unconfigured": "ยังไม่ได้ตั้งค่าการแจ้งเตือนบนเซิร์ฟเวอร์", "web.push.blocked": "การแจ้งเตือนถูกบล็อก โปรดอนุญาตในการตั้งค่าเว็บไซต์ของเบราว์เซอร์", "web.push.revoked": "เบราว์เซอร์นี้ถูกนำออกจากบัญชีแล้ว โปรดล้างข้อมูลที่เว็บไซต์นี้บันทึกไว้ก่อนเพิ่มอีกครั้ง", - "web.push.error": "เปลี่ยนการตั้งค่าการแจ้งเตือนไม่สำเร็จ โปรดตรวจสอบสิทธิ์ของเบราว์เซอร์แล้วลองอีกครั้ง" + "web.push.error": "เปลี่ยนการตั้งค่าการแจ้งเตือนไม่สำเร็จ โปรดตรวจสอบสิทธิ์ของเบราว์เซอร์แล้วลองอีกครั้ง", + "keybinding.action.screenOcr": "จับข้อความบนหน้าจอ (OCR)", + "keybinding.action.screenOcr.desc": "จับภาพหน้าจอเพื่อลากเลือกข้อความบนนั้น", + "screenOcr.dialog.savePdf": "บันทึกเป็น PDF", + "screenOcr.dialog.saveImage": "บันทึกเป็นรูปภาพ", + "screenOcr.errorCapture": "จับภาพหน้าจอไม่ได้", + "screenOcr.modeText": "เลือกข้อความ", + "screenOcr.modeRect": "เลือกพื้นที่", + "screenOcr.copy": "คัดลอก", + "screenOcr.copyLine": "คัดลอกเป็นบรรทัดเดียว", + "screenOcr.copyImage": "คัดลอกรูปภาพ", + "screenOcr.saveImage": "บันทึกรูปภาพ", + "screenOcr.savePdf": "บันทึก PDF", + "screenOcr.translate": "แปล", + "screenOcr.summarize": "สรุป", + "screenOcr.speak": "อ่านออกเสียง", + "screenOcr.stopSpeak": "หยุดอ่าน", + "screenOcr.openLink": "เปิดลิงก์", + "screenOcr.search": "ค้นหาบนเว็บ", + "screenOcr.selectAll": "เลือกทั้งหมด", + "screenOcr.close": "ปิด", + "screenOcr.language": "ภาษา", + "screenOcr.languageAuto": "อัตโนมัติ", + "screenOcr.statusRecognizing": "กำลังอ่านข้อความ…", + "screenOcr.statusReady": "{{count}} คำ · ลากเพื่อเลือก", + "screenOcr.statusEmpty": "ไม่พบข้อความ", + "screenOcr.statusSelected": "เลือก {{count}} อักขระ", + "screenOcr.hint": "ลากเพื่อเลือก · ดับเบิลคลิกเลือกคำ · Ctrl+A ทั้งหมด · Ctrl+C คัดลอก · Esc ปิด", + "screenOcr.toastCopied": "คัดลอกแล้ว", + "screenOcr.toastImageCopied": "คัดลอกรูปภาพแล้ว", + "screenOcr.toastSaved": "บันทึกแล้ว", + "screenOcr.working": "กำลังประมวลผล…", + "screenOcr.resultTranslate": "ผลการแปล", + "screenOcr.resultSummary": "ผลสรุป", + "screenOcr.errorRecognition": "อ่านข้อความไม่ได้", + "screenOcr.errorUnavailable": "พีซีนี้ใช้การจับข้อความบนหน้าจอไม่ได้ โปรดติดตั้งชุดภาษาที่มีการรู้จำอักขระด้วยแสงในการตั้งค่า Windows", + "screenOcr.errorExport": "บันทึกหรือคัดลอกไม่ได้", + "screenOcr.errorLlm": "AI ไม่ตอบกลับ โปรดตรวจสอบการตั้งค่าโมเดล", + "screenOcr.errorTts": "อ่านออกเสียงไม่ได้", + "screenOcr.errorLink": "เปิดลิงก์ไม่ได้", + "screenOcr.errorInvalid": "การจับภาพนี้ปิดไปแล้ว โปรดจับภาพใหม่", + "screenOcr.errorEmpty": "ยังไม่ได้เลือกข้อความ" } diff --git a/packages/i18n/src/locales/vi.json b/packages/i18n/src/locales/vi.json index 80aeb579..0ce7e95c 100644 --- a/packages/i18n/src/locales/vi.json +++ b/packages/i18n/src/locales/vi.json @@ -2500,5 +2500,46 @@ "web.push.unconfigured": "Thông báo chưa được cấu hình trên máy chủ.", "web.push.blocked": "Thông báo đang bị chặn. Hãy cho phép trong phần cài đặt trang web của trình duyệt.", "web.push.revoked": "Trình duyệt này đã bị xóa khỏi tài khoản. Hãy xóa dữ liệu đã lưu của trang web trước khi thêm lại.", - "web.push.error": "Không thể thay đổi thông báo. Hãy kiểm tra quyền của trình duyệt rồi thử lại." + "web.push.error": "Không thể thay đổi thông báo. Hãy kiểm tra quyền của trình duyệt rồi thử lại.", + "keybinding.action.screenOcr": "Nhận dạng chữ trên màn hình (OCR)", + "keybinding.action.screenOcr.desc": "Chụp màn hình để bạn kéo chọn chữ trên đó.", + "screenOcr.dialog.savePdf": "Lưu thành PDF", + "screenOcr.dialog.saveImage": "Lưu thành ảnh", + "screenOcr.errorCapture": "Không chụp được màn hình.", + "screenOcr.modeText": "Chọn chữ", + "screenOcr.modeRect": "Chọn vùng", + "screenOcr.copy": "Sao chép", + "screenOcr.copyLine": "Sao chép thành một dòng", + "screenOcr.copyImage": "Sao chép ảnh", + "screenOcr.saveImage": "Lưu ảnh", + "screenOcr.savePdf": "Lưu PDF", + "screenOcr.translate": "Dịch", + "screenOcr.summarize": "Tóm tắt", + "screenOcr.speak": "Đọc to", + "screenOcr.stopSpeak": "Dừng đọc", + "screenOcr.openLink": "Mở liên kết", + "screenOcr.search": "Tìm trên web", + "screenOcr.selectAll": "Chọn tất cả", + "screenOcr.close": "Đóng", + "screenOcr.language": "Ngôn ngữ", + "screenOcr.languageAuto": "Tự động", + "screenOcr.statusRecognizing": "Đang đọc chữ…", + "screenOcr.statusReady": "{{count}} từ · kéo để chọn", + "screenOcr.statusEmpty": "Không tìm thấy chữ", + "screenOcr.statusSelected": "Đã chọn {{count}} ký tự", + "screenOcr.hint": "Kéo để chọn · nhấp đúp chọn từ · Ctrl+A tất cả · Ctrl+C sao chép · Esc đóng", + "screenOcr.toastCopied": "Đã sao chép", + "screenOcr.toastImageCopied": "Đã sao chép ảnh", + "screenOcr.toastSaved": "Đã lưu", + "screenOcr.working": "Đang xử lý…", + "screenOcr.resultTranslate": "Bản dịch", + "screenOcr.resultSummary": "Bản tóm tắt", + "screenOcr.errorRecognition": "Không đọc được chữ", + "screenOcr.errorUnavailable": "Máy tính này không dùng được tính năng nhận dạng chữ. Hãy cài gói ngôn ngữ có nhận dạng ký tự quang học trong Cài đặt Windows.", + "screenOcr.errorExport": "Không lưu hoặc sao chép được", + "screenOcr.errorLlm": "AI không phản hồi. Hãy kiểm tra cài đặt mô hình.", + "screenOcr.errorTts": "Không đọc to được", + "screenOcr.errorLink": "Không mở được liên kết", + "screenOcr.errorInvalid": "Ảnh chụp này đã đóng. Hãy chụp lại.", + "screenOcr.errorEmpty": "Chưa chọn chữ" } diff --git a/packages/i18n/src/locales/zh-TW.json b/packages/i18n/src/locales/zh-TW.json index cf43fa59..e54de3b9 100644 --- a/packages/i18n/src/locales/zh-TW.json +++ b/packages/i18n/src/locales/zh-TW.json @@ -2500,5 +2500,46 @@ "web.push.unconfigured": "伺服器尚未設定通知功能。", "web.push.blocked": "通知已遭封鎖。請在瀏覽器的網站設定中允許通知。", "web.push.revoked": "此瀏覽器已從帳號移除。再次加入前,請清除此網站儲存的資料。", - "web.push.error": "無法變更通知設定。請檢查瀏覽器權限後再試一次。" + "web.push.error": "無法變更通知設定。請檢查瀏覽器權限後再試一次。", + "keybinding.action.screenOcr": "螢幕文字辨識 (OCR)", + "keybinding.action.screenOcr.desc": "擷取螢幕,並拖曳選取其中的文字。", + "screenOcr.dialog.savePdf": "另存為 PDF", + "screenOcr.dialog.saveImage": "另存為圖片", + "screenOcr.errorCapture": "無法擷取螢幕。", + "screenOcr.modeText": "選取文字", + "screenOcr.modeRect": "選取區域", + "screenOcr.copy": "複製", + "screenOcr.copyLine": "複製為一行", + "screenOcr.copyImage": "複製圖片", + "screenOcr.saveImage": "儲存圖片", + "screenOcr.savePdf": "儲存 PDF", + "screenOcr.translate": "翻譯", + "screenOcr.summarize": "摘要", + "screenOcr.speak": "朗讀", + "screenOcr.stopSpeak": "停止朗讀", + "screenOcr.openLink": "開啟連結", + "screenOcr.search": "網頁搜尋", + "screenOcr.selectAll": "全選", + "screenOcr.close": "關閉", + "screenOcr.language": "辨識語言", + "screenOcr.languageAuto": "自動", + "screenOcr.statusRecognizing": "正在辨識文字…", + "screenOcr.statusReady": "{{count}} 個詞 · 拖曳選取", + "screenOcr.statusEmpty": "找不到文字", + "screenOcr.statusSelected": "已選取 {{count}} 個字元", + "screenOcr.hint": "拖曳選取 · 按兩下選詞 · Ctrl+A 全選 · Ctrl+C 複製 · Esc 關閉", + "screenOcr.toastCopied": "已複製", + "screenOcr.toastImageCopied": "已複製圖片", + "screenOcr.toastSaved": "已儲存", + "screenOcr.working": "處理中…", + "screenOcr.resultTranslate": "翻譯結果", + "screenOcr.resultSummary": "摘要結果", + "screenOcr.errorRecognition": "無法辨識文字", + "screenOcr.errorUnavailable": "此電腦無法使用螢幕文字辨識。請在 Windows 設定中安裝含光學字元辨識的語言套件。", + "screenOcr.errorExport": "無法儲存或複製", + "screenOcr.errorLlm": "未收到 AI 回應。請檢查模型設定。", + "screenOcr.errorTts": "無法朗讀", + "screenOcr.errorLink": "無法開啟連結", + "screenOcr.errorInvalid": "此擷取已關閉。請重新擷取。", + "screenOcr.errorEmpty": "未選取文字" } diff --git a/packages/i18n/src/locales/zh.json b/packages/i18n/src/locales/zh.json index cce877aa..22cd64d7 100644 --- a/packages/i18n/src/locales/zh.json +++ b/packages/i18n/src/locales/zh.json @@ -2500,5 +2500,46 @@ "web.push.unconfigured": "服务器尚未配置通知功能。", "web.push.blocked": "通知已被阻止。请在浏览器的网站设置中允许通知。", "web.push.revoked": "此浏览器已从账户中移除。再次添加前,请清除此网站存储的数据。", - "web.push.error": "无法更改通知设置。请检查浏览器权限后重试。" + "web.push.error": "无法更改通知设置。请检查浏览器权限后重试。", + "keybinding.action.screenOcr": "屏幕文字识别 (OCR)", + "keybinding.action.screenOcr.desc": "截取屏幕,并拖动选择其中的文字。", + "screenOcr.dialog.savePdf": "另存为 PDF", + "screenOcr.dialog.saveImage": "另存为图片", + "screenOcr.errorCapture": "无法截取屏幕。", + "screenOcr.modeText": "选择文字", + "screenOcr.modeRect": "选择区域", + "screenOcr.copy": "复制", + "screenOcr.copyLine": "复制为一行", + "screenOcr.copyImage": "复制图片", + "screenOcr.saveImage": "保存图片", + "screenOcr.savePdf": "保存 PDF", + "screenOcr.translate": "翻译", + "screenOcr.summarize": "摘要", + "screenOcr.speak": "朗读", + "screenOcr.stopSpeak": "停止朗读", + "screenOcr.openLink": "打开链接", + "screenOcr.search": "网页搜索", + "screenOcr.selectAll": "全选", + "screenOcr.close": "关闭", + "screenOcr.language": "识别语言", + "screenOcr.languageAuto": "自动", + "screenOcr.statusRecognizing": "正在识别文字…", + "screenOcr.statusReady": "{{count}} 个词 · 拖动选择", + "screenOcr.statusEmpty": "未找到文字", + "screenOcr.statusSelected": "已选择 {{count}} 个字符", + "screenOcr.hint": "拖动选择 · 双击选词 · Ctrl+A 全选 · Ctrl+C 复制 · Esc 关闭", + "screenOcr.toastCopied": "已复制", + "screenOcr.toastImageCopied": "已复制图片", + "screenOcr.toastSaved": "已保存", + "screenOcr.working": "处理中…", + "screenOcr.resultTranslate": "翻译结果", + "screenOcr.resultSummary": "摘要结果", + "screenOcr.errorRecognition": "无法识别文字", + "screenOcr.errorUnavailable": "此电脑无法使用屏幕文字识别。请在 Windows 设置中安装带光学字符识别的语言包。", + "screenOcr.errorExport": "无法保存或复制", + "screenOcr.errorLlm": "未收到 AI 响应。请检查模型设置。", + "screenOcr.errorTts": "无法朗读", + "screenOcr.errorLink": "无法打开链接", + "screenOcr.errorInvalid": "此截图已关闭。请重新截取。", + "screenOcr.errorEmpty": "未选择文字" }