A rebindable screen-ocr action (default Ctrl+Alt+O) freezes the monitor under the cursor and
covers it with a focusable overlay. Text is selected by flow drag, rectangle drag, double/triple
click, Shift+click or Ctrl+A, then copied, saved as PNG or searchable PDF, translated,
summarized, read aloud, opened as a link or searched.
- Selection, text assembly and the PDF document are pure functions in @d3ro/core/screen-ocr,
shared by the main process and the overlay.
- Recognition uses Windows.Media.Ocr through powershell.exe -EncodedCommand with 2x upscaling;
the capture reaches disk only as a temp PNG that is removed right after recognition.
- Capture uses GDI BitBlt for the target monitor (39 ms) because desktopCapturer captured all
three monitors and took 2.9 s when DXGI duplication timed out.
- The accelerator is reserved only where the feature is available, so macOS does not swallow
the key for a feature it cannot run (fails closed with a message).
Map: CAP-17 [~], GAP-OCR-01 (live hotkey journey not yet driven in the running app),
GAP-OCR-02 (macOS engine, mixed-language accuracy).
Adds history share (Markdown file / original audio) with 12-locale strings, pins Maestro to
the target device with --udid, allows cleartext only for explicit local E2E artifacts, and
fixes small-screen scroll/keyboard races in the Maestro flows.
Evidence: docs/evidence/2026-11-17-backlog-operations.md.
The local model follows the polish instructions only most of the time: in
repeated runs it added a [00:00] or [화자 1] marker that was never in the
audio, and it turned "사실 거의 근간" into "사실 근간". Both change what the
meeting record says, so they are now enforced in code instead of hoped for
in the prompt: markers absent from the original chunk are stripped, and
meaning qualifiers the speaker said are restored from that chunk.
The instructions themselves were re-measured on the model: a fixed removal
list, an explicit qualifier example and a marker ban keep the filler cleanup
without touching ordinary words, and the same wording is used by the app,
the tests and the probe. Unknown-word repair still depends on the model.
Learned text becomes part of a prompt, so a dictionary word or phrase
containing a newline could add its own instruction line. Every value that
reaches the recognition hint or the polish block is now flattened to one
line and length-capped, and a test proves an injected line stays plain text.
Also cover the invariants that keep the correction safe (random inputs,
Hangul untouched, idempotent, no token growth, large transcript) and the
file-transcription wiring that attaches the learned hint to every chunk and
corrects each chunk and segment before it is stored.
AI polishing of a recorded or pasted transcript was a generic rewrite: it
cleaned fillers but left names and in-house terms mangled even when the app
already knew them. Polishing now corrects dictionary pronunciations and
typed identifiers deterministically before and after the model, and hands
the model the learned term list plus the user's own expressions with
explicit rules: rewrite a misheard name, product or in-house term to the
learned spelling, and never swap an ordinary noun or ending on similarity
alone. Dictation output is re-checked the same way after the model runs.
The first cut replaced any Hangul word within one jamo of a learned term.
Measured against the real local corpus (three stored meeting transcripts)
it changed 286 of 3,005 words: 하면 became 라면, 이제 became 이게, 매일 became
메일. Most of what the app learns from typing is ordinary grammar, so
distance alone cannot separate a misheard word from a correct one.
Deterministic correction now covers only dictionary pronunciations and the
Latin identifiers the user types, preserving their casing. Korean terms
still bias recognition through the prompt hint and guide the polish
prompt. Re-measured on the same transcripts: 0 of 3,005 words changed.
Hangul-layout typing artifacts (dkssudgktpdy) are dropped from the hint.
The tree carried uncommitted work from several areas (web push, mobile
screens, admin charts, test-hygiene cleanup, CI and map docs). It is landed
as one commit so the release pipeline can start from a clean tree; the
desktop transcript-correction behaviour shipped in the previous commit and
is unchanged here.
Meeting and caption transcription never used what the app had already
learned: speech recognition ran with a fixed Korean prompt, the refine
prompt had no glossary, refined lines stopped inside the caption service,
and the saved transcript was a raw join of segments. Names and jargon the
user types every day were therefore re-mangled on every recording.
The learned corpus (user dictionary plus personal phrases) now feeds one
glossary that biases recognition, fixes mistyped words deterministically
right after each line is finalized, and is handed to the local model when
it polishes a line. Refined lines also reach the meeting transcript and
screen through a new segment-updated event.
Corrections never touch words the user actually wrote, refuse ties between
equally likely candidates, and keep particles intact.
- Settings > Input: the excluded-apps field was reset from every telemetry
snapshot (several per second while typing), so edits vanished before blur
saved them; it now syncs only when the saved list changes
- Templates: a failed save (e.g. deleted on another device) closed the dialog
and dropped the edits; it now stays open and shows why, and a failed
session start is reported too
The lock desktop swallows keyup, so a hold-to-talk recording never ended,
the first shortcut after unlocking was dropped as auto-repeat, and Win from
Win+L made the next suggestion accept wait out the full 1.5s modifier timeout.
On powerMonitor lock-screen/suspend the chord machine now emits released for
emitted triggers and clears its state, and modifier tracking forgets held keys.
The session ignores trailing whitespace but acceptance compared the raw
prefix, so typing one more space after the fix appeared made accept drop
the correction silently and leave the typo in place.
Tags over 80 characters now show a length notice, and adding a tag that
already exists in another case just closes the input, rather than both
landing on a '#tag · retry' badge that repeats the same rejection.
While suggestions are on, typos are no longer replaced silently. The popup shows
the fix next to the candidate and applies it only when the candidate is accepted.
The per-minute request cap rises from 6 to 20 so suggestions keep appearing after
accepting one and continuing to type.
Typo auto-correct (INPUT-19)
- TypoCorrectionService + core typo-correction: 2-set Korean layout
mistypes both ways, always-wrong spellings, near-misses of the user's
frequent words, optional model check re-validated in core; applied by
backspace N + paste and cancelled on new keys or window change.
Prediction quality (INPUT-20/21)
- gemma models are sent raw with the user's unfinished sentence pre-filled
into the model turn, so the model truly continues the text (mid-word
joins and spacing exact). Other models keep the instruction prompt.
- Context: text before the caret (opening lines and a fact list of names,
dates, amounts outside the 400-char window), messages just sent in the
same chat window (memory only, 30 min), personal-graph follow-ups and
IDF-ranked related sentences with Korean stemming.
- Guards: first sentence only, restated-prefix stripping (also fixes the
instruction path), repeat drop, no invented dates/names/numbers
(one retry), plain -다 register, no emoji unless the user uses them,
search-box keywords. Fill loop mixes instruction prompts for variety.
- Opt-in real-model probe: tests/integration/suggestion-quality.ollama.test.ts
Suggestions now actually show up
- Terminal (Claude Code): caret trust from the input line, not the whole
screen (status line below the box blocked every keystroke); Korean
composition counts as typing at the end; a grey placeholder filling the
empty box at once is not user text.
- Chrome: UIA focus flicker while typing no longer resets "edited".
- KakaoTalk/RichEdit: trailing paragraph mark ignored for edit position
and stripped so accepted text keeps its separating space.
- While the overlay is visible the suggestion shortcuts are captured so
Ctrl+Alt+Enter no longer also types a newline; released when hidden.
- nut-js preloaded after boot (first accept waited 4.8 s).