59 KiB
04 — Desktop App (Electron) Map
Surface:
apps/desktopStack: Electron 33 + React 19 + MUI 7 + Vite (electron-vite) + better-sqlite3/drizzle + uiohook-napi + nut-js Source root:apps/desktop/src(main/,preload/,renderer/)
1. Process architecture
| Layer | Path | Contents |
|---|---|---|
| Main | src/main/ |
Services, IPC handlers, windows, bootstrap/lifecycle, DB |
| Preload | src/preload/ |
index.ts exposes window.electronAPI; popup.ts exposes window.popupAPI |
| Renderer | src/renderer/ |
React app: AppLayout + 7 pages + modals + 5 vanilla popups |
Main entry src/main/index.ts: sets app name/AppUserModelId, disables GPU acceleration, EPIPE/uncaught handlers, registers d3ro-voice:// deep-link protocol (Supabase OAuth implicit + PKCE), single-instance lock, then bootstrap() + setupLifecycle(). The explicit manual-QA argument --preview-update-dialogs is isolated before protocol registration/bootstrap: it bypasses the single-instance lock, invokes the real update-dialog prompter for four consent/restart variants, touches no feed/download/install/settings state, then quits. This path is unit-tested, but its packaged native dialogs still require an external-terminal visual run.
Bootstrap src/main/bootstrap.ts: ordered BootstrapStep[] — logger, config, database (critical), license, create-windows (critical), tray, ipc-handlers (critical), custom-instructions, voice-commands, sound-effects, auto-launch, popup-preload (no-op at boot since 2026-11-13; popups are created lazily on first demand), key-bindings, voice-mode, stt-warmup (only when sttProvider === 'local'), llm-polling, meeting-summary-wiring, meeting-mode, cloud-sync, auto-update. Wires VoiceMode events to sound + history persistence, and subscribes to KeyBindingService triggered for the history-popup / command-popup actions (bootstrap.ts:159) — those two were hardcoded accelerators before and are now rebindable like everything else. ConfigEffects also propagates inputTelemetryEnabled / inputTelemetryPaused dynamically to InputTelemetryService.applyConfig.
2. Main services (src/main/services/)
Singleton + EventEmitter pattern (getXService() accessors).
Core voice pipeline
| Service | Purpose |
|---|---|
VoiceModeService |
Orchestrator: 9-state RecognitionState + 4-state AudioState, dual-condition flush, action queue, and secure audio buffer zeroing (secureWipeAudioChunks / secureZeroBuffer) on cancel/cleanup (2026-11-13 W5-01). Events: session-started/completed/cancelled, transcription-update, audio-level, recognition/audio-state-changed, premium-llm-fallback, error |
AudioCaptureService |
Mic PCM16 16kHz mono (bundled SoX on Windows, node-record-lpcm16 elsewhere) with secureWipeAudioChunks on stop/dispose. Spawns hidden (windowsHide); a missing SoX fails with the exact fix command |
LocalSTTService |
faster-whisper Python sidecar manager (state machine, dual-flush, model download/cancel, background warm-up, live partial transcription, sidecarBaseUrl getter for POST /diarize, and 2026-11-13 releasePrimaryModel() / 15-min IDLE_MODEL_UNLOAD_MS timer via POST /unload { slot: "primary" } so switching to cloud STT or going idle frees the 1.6–3.1 GB Whisper model while keeping the sidecar HTTP server alive for /uia/focus). Connects over IPv4 loopback (getSidecarBaseUrl) and fails fast with an actionable message when the bundled engine or virtualenv is missing |
KeyBindingService |
uiohook-napi global hooking for keyboard and mouse, driven by the @d3ro/core/keybinding contract: 6 rebindable actions (dictation, hands-free, command, caption, history-popup, command-popup), several bindings per action, structural reserved-combo checks. Events: triggered (in-process payload carries actionId, type (pressed/released), isDoublePress, holdMode, timestamp; the renderer-facing keybinding:triggered event is the narrower KeyBindingTriggeredEvent, keybinding.ts:1092), changed, error. globalShortcut is used only to mute the macOS system beep, and only for accelerators it registered itself. Mouse events cannot be suppressed by uiohook, so a bound button also performs its native action |
TextInsertService |
Clipboard save→set→Ctrl+V→restore (NutKeyboardPort: uses native koffi Win32 SendInput via utils/win32-unicode-input.ts on Windows without loading @nut-tree-fork/nut-js; nut-js used on macOS/Linux) |
SoundEffectService |
Preloaded WAV feedback (start/stop/error/cancel/chime) |
STT engine layer (services/stt/)
| File | Purpose |
|---|---|
STTManager |
Dispatcher across local + 6 cloud providers, auto-fallback (events provider-changed, config-changed, fallback-to-local). transcribePartial/warmUpLocal route to the local engine only |
types.ts |
ISTTDriver contract |
audio-utils.ts |
pcmToWav, createProbeWav |
| `drivers/OpenAI | Groq |
LLM layer
| Service | Purpose |
|---|---|
LocalLLMService |
Ollama REST (models, pull w/ progress, server start, NDJSON streaming). The 2026-09-22 runaway guard gives each request its own AbortController (external caller signals are relayed and detached on completion), requires done before an NDJSON stream succeeds, and clears incomplete streams. Generation/stream requests are bounded to 2048 tokens / 120 s; chat is bounded to 512 / 60 s. Ollama server spawn and polling are deduplicated, and lifecycle dispose terminates owned work. |
PremiumLLMService |
Claude via Supabase llm-proxy, local fallback |
OnlineLLMService |
JWT-authenticated .NET backend client; 2026-11-13 (W5-01): applies reversible redactPII / redactPIIMessages before cloud dispatch and rehydrateText on completion (pii-zdr-wiring.test.ts 3/3 PASS) |
llm-prompts.ts |
SSOT for prompt resolution, placeholder substitution, and argument placement. resolveSystemPrompt (:118) maps an LLMAction to its base prompt and handles custom explicitly instead of dropping silently to refine. renderInstructionPrompt (:78) substitutes {{text}} / {{userPrompt}} / {{targetLanguage}} and warns by name for any placeholder left standing rather than letting it reach the model. buildInstructionInvocation (:101) decides where an instruction goes in processText(text, action, targetLanguage, customPrompt): the instruction becomes the system prompt and the transcript the text, except for instructions that spell out {{text}}, which keep the old meaning for backward compatibility. resolveTargetLanguage (:50) is the one place translate targets are decided (still English, see 11 GAP-LLM-01). All three LLM entry paths call the same functions — VoiceModeService (:838), ChainService (:196), and the LLM.PROCESS IPC handler (llm-handlers.ts:96) — so no caller re-implements the rules |
Memory & knowledge
| Service | Purpose |
|---|---|
HistoryService |
SQLite history CRUD/search/stats |
UnifiedSearchService |
Read-only SQLite search across history title/content and meeting title/transcript/memos; 2026-11-13: uses FTS5 tokenize = 'trigram' virtual tables (history_fts, meeting_sessions_fts, meeting_memos_fts) for queries of ≥3 characters with automatic trigger sync and <3-char LIKE fallback |
DictionaryService |
Custom vocabulary CRUD/search + cloud sync hooks + JSON/CSV import/export (dictionary:import/export, save/open dialogs) |
MemoService |
Memo tags over history (memo_tags) |
RAGService |
Local RAG: nomic-embed-text embeddings, cosine search over rag_chunks (2026-11-13: SqliteChunkStore persists binary Float32Array embedding_blob alongside JSON fallback, caches StoredChunk[] in a DB-scoped WeakMap for zero-JSON.parse queries, and wraps removeDocument in a transaction) |
CustomInstructionService |
User LLM commands (5 built-ins) |
VoiceCommandService |
Keyword → command rule matching |
ChainService |
Multi-step LLM pipelines (LLMChain). Each step resolves its instruction through llm-prompts.ts (ChainService.ts:196); before that, chain steps sent placeholders through unsubstituted |
ScreenContextService |
Active-window + selected-text context (on Windows uses sendWin32CopyChord() via koffi SendInput without loading nut-js) |
Phase 10+ features
| Service | Purpose |
|---|---|
CaptionService |
Live captions from system/loopback audio; caption overlay (events segment, segment-updated, state-changed, session-saved, error). Since 2026-11-05 every recognition call carries the learned-vocabulary STT hint, each final line passes the deterministic glossary correction, and the LLM refine prompt receives the user-term/style block |
TranscriptGlossaryService |
Learned-vocabulary gateway for transcription (@d3ro/core/transcript-correction): merges user dictionary words + pronunciations and personal-graph phrases into a 15 s cached glossary, and exposes the STT initial_prompt hint, deterministic word correction (dictionary variants + 1-jamo learned-term match that never touches words the user actually wrote), and the LLM 교열 reference block. Sources are injectable; DB failures fall back to an empty glossary so transcription never blocks |
FileTranscriptionService |
Audio/video file → ffmpeg → 30s chunks → STT merge (events progress, complete, error, state-changed). Shared media-format SSOT is file-transcription/media-formats.ts; meeting import can skip duplicate history persistence and retain the normalized WAV |
MeetingSummaryService |
Post-caption LLM summary |
DictationTemplateService |
Field-by-field voice form filling |
VoiceConversationService |
STT→LLM→TTS loop, 10-turn memory. Local conversations are single-flight and pass their request signal through to local LLM chat, so a voice cancel aborts its own active request rather than only changing UI/watchdog state. |
TTSPlaybackService |
Platform TTS (macOS say, Windows SAPI), sentence queue |
VoiceActionService |
Voice → LLM JSON action plan → OS execution (dangerous blocked) |
Phase 12–15
| Service | Purpose |
|---|---|
MeetingModeService |
Meeting recording and existing-media import: live/file transcript, normalized meeting audio (meeting/disk-wav-audio-sink.ts DiskWavAudioSink streams PCM16 directly to disk with O(1) memory and backpatches the 44-byte RIFF/WAV header on finalize(), 2026-11-13 GAP-PERF-01), timestamp memos (meeting_memos.revision CAS synced), doc generation/export, and speaker diarization (diarizeSession calls sidecar POST /diarize via LocalSTTService.sidecarBaseUrl — pyannote.audio when installed + energy-VAD acoustic turn segmentation — and falls back to LLM speaker estimation while preserving [화자 N] markers across polishTranscript, GAP-MTG-02). meeting/imported-recording.ts maps file-transcription segments to the canonical meeting transcript. Account revision checks before/after WAV copy and after transcription reject even A→B→A changes; unsuccessful imports remove only their own generated audio. Since 2026-11-05 refined caption lines reach the meeting transcript (meetingMode:segmentUpdated) instead of the raw text, and auto-polish receives the learned-vocabulary block |
MeetingDocTemplateService |
Meeting-doc templates (built-ins + CRUD) |
Input intelligence (2026-09-21; Local Flow Intelligence extension 2026-09-22)
| Service / file | Purpose |
|---|---|
InputTelemetryService |
Global input telemetry and context observer: keystroke/click/scroll counters, mouse travel (Manhattan sum of per-axis deltas), active time, per-hour-per-app buckets flushed every 5 s; typed-text learning via UIA snapshot diffs. Since 2026-10-26, the UIA context observer runs by default whenever typoAutoCorrectEnabled !== false (or suggestionEnabled === true), while SQLite input_activity persistence only runs when inputTelemetryEnabled === true (opt-in). Since 2026-10-27, includes TerminalKeystrokeTracker (@d3ro/core/input-intelligence) + isForegroundKoreanIme (utils/win32-foreground.ts) to reconstruct the active terminal/Vim line (reconcileWithScreen for UIA screen buffers like Windows Terminal/pwsh/cmd, reconcileWithXtermValue for xterm.js <textarea class="xterm-helper-textarea"> controls in Agent Switchboard/VS Code/Cursor where .value stays populated until Enter/Ctrl+C, and getSnapshot for canvas/remote terminals with len=0 like Kitedesk/WezTerm/Alacritty), preserving trailing spaces stripped by grid terminals and tracking Vim modes (insert, normal, command). Programmatic keys (_programmaticKeys) no longer advance _lastKeyAt, and Enter-triggered chat submissions that clear or shrink the input field rebase focus text immediately so cleared/toolbar text does not trigger post-send suggestions. Since 2026-10-29, tracks _typedInCurrentFocus and _lastTypedHwnd and forces immediate foreground sampling on the first keystroke after a mouse click (_rebaseFocusText) or window switch, so typing pc (ㅔㅊ) immediately after clicking into a search box or new field sets editedSinceFocus = true on the very first snapshot. It extends the telemetry store with Flow Radar (hourly activity density + chars + edit stability; a potential-focus score, not a real session), Edit Friction (chars/backspaces; edits per 100 chars), App Quality (per-app suggestion total/accepted/accept rate/average latency), Privacy Receipt (actual row counts and retention), and Smart Exclusion evidence. Raw key events, key codes and content streams are never retained; separately, opt-in learned text may be retained in typing_samples and personal_phrases under the stated receipt policy. |
TypoCorrectionService |
Built-in automatic Han/Eng layout conversion and typo auto-correction (INPUT-19). Runs on each typed-context: Araso-style layout conversions (source: 'layout') fire after 25 ms on word boundaries (Space/punctuation, 40 ms quick snapshot) or 135 ms at token tail (150 ms quick snapshot, even when Chromium/Discord keeps isComposing=true, committing IME composition first only when the trailing character is Hangul/jamo), replacing 2-set Han/Eng layout errors, mixed English+Korean words (githubdp→github에, PDFvkdlf→PDF파일, ㅖㅇㄹ파일→PDF파일), lingering Shift on double consonants (djEJs→어떤), reverse Hangul-to-English (ㅎ샤ㅗㅕㅠ에→github에, ㅑㄹ ㅑ 쇼ㅔㄷ ㅗ디ㅣㅐ 소두 ㅑㅅ 초뭏ㄷㄴ ㅑㅜ 10 ㄴㄷ채ㅜㅇㄴ→if I type hello then it changes in 10 seconds), lookahead English clauses (새 ㅠㄷ →to be , 새 애 this →to do this ), and partially-converted tokens (안녕gktpdy→안녕하세요). In terminals and Vim (terminalMode: true), layout-only conversion stays active while spelling/model checks are suppressed: converts CLI commands (퍄ㅡ →vim , ㅜ퍄ㅡ →nvim , ㅎ샤 →git , ㅣㄴ →ls , 채ㅜㄴㅅ →const ), Vim Insert-mode typing (stripping leading i/a/o/I/A/O/s/S mode-entry keys when present in raw keystroke buffers), and Vim Command-line ex-commands (:ㅈㅂ→:wq, :ㅈㅂ!→:wq!, :ㅈ→:w, :ㅂ→:q, :ㅂ!→:q!, :ㅌ→:x, :ㅍ네→:vsp, :네→:sp, :ㅜㅐㅗ→:noh, preserving :), while blocking all conversions in Vim Normal mode (vimMode === 'normal'). Since 2026-10-28, on Windows, TextInsertService.replaceBeforeCaret (method === 'keyboard') executes an atomic single-syscall sendWin32UnicodeReplacement (utils/win32-unicode-input.ts) packing (optional VK_SPACE) + N × VK_BACK + M × KEYEVENTF_UNICODE into one SendInput call (zero JS sleep, zero clipboard access), and TypoCorrectionService delta-rebases replacement plans (rebaseTypoReplacementPlan) when the user appends characters while an async snapshot or LLM check is in flight. Since 2026-10-31 (v1.15.2), propose() checks !this._isRegionChecked(region) before calling _askModel, avoiding duplicate 336-token LLM checks when handleTypingContext already verified the same region. Both Stage 1 (deterministic layout/spelling) and Stage 2 (tail-anchored trailingWordOnly: true Gemma check) run seamlessly even when SuggestionService is enabled. |
UiaContextService |
Client for the sidecar UIA bridge (GET /uia/focus): focused text, caret rect/offset, IsPassword, IME composition state and hasSelection. The sidecar derives hasSelection only by comparing the TextPattern selection range Start/End endpoints: it does not call GetText on that range or add a separate selected-text payload (the existing focused-field text can still include a selection). Fail-closed on password/unavailable; transient failures back off for 1.5/3/6/12/30 s and permanent failures for 5 min. |
SuggestionService |
Next-sentence ghost text: 600 ms debounce, then buildSuggestionPrompt → LocalLLMService.streamGenerate with a caller-owned AbortSignal → candidate ranking and accept (via TextInsertService). Presentation-active includes candidates plus generating, warmingUp and partialText; clear/dismiss aborts the active request, invalidates its generation token, clears TTL state, then emits cleared/hide so a spinner-only popup closes. A successful final candidate payload resets generating=false and partialText=null. App DNA extends phrase ranking with appName context and a 1.75× same-app bonus; Memory Decay uses a 30-day half-life plus frequency. Why This Suggestion exposes only local-model/local-memory provenance and continuation/related/phrase/appPhrase counts to the overlay, never raw evidence text. Instant Recall uses local-memory provenance without another model-budget spend only when the model is unavailable, raises a non-cancellation error, truly times out, or returns empty; it never publishes after dismiss, new typing, token/context mismatch or staleness. Its runaway limits are a 5 s minimum interval, 6 requests/min default (hard-config maximum 12), 3 candidates, 64 output tokens, 12-character regeneration growth and an 8 s timeout; app startup does not warm the model and suggestion requests use keep_alive: 2m. |
global-input-hook.ts |
Ref-counted owner of the single process-wide uiohook hook so KeyBindingService and telemetry can both attach without one stopping the other. |
utils/win32-foreground.ts |
Foreground window title/pid/exe/bounds and active thread Korean IME state (isForegroundKoreanIme via GetGUIThreadInfo + ImmGetDefaultIMEWnd + SendMessageTimeoutW(WM_IME_CONTROL)) via koffi FFI into user32/kernel32/imm32 (chosen over get-windows, which needs an install script this repo does not run). |
sidecar/uia_bridge.py |
Windows UIA snapshot (uiautomation 2.0.29, comtypes) on a dedicated COM-initialised thread with a 1.5 s budget; never walks the UIA tree (Chrome/VS Code tree walks take 10-30 s). Uses GetVisibleRanges() / tail slicing [-MAX_TEXT_CHARS:] for large terminal scrollback buffers and ExpandToEnclosingUnit(..., waitTime=0) to avoid uiautomation.py's default 500 ms sleep on Windows Terminal. Warm calls measure 0-15 ms. |
2026-10-31..11-13 구조 경계: suggestion/SuggestionModelRequest.ts가 prefill/지시형 모델 요청 구성, 후보 파싱, 지어낸 사실·문맥 복사 검증을 소유한다. suggestion/FirstCandidateRunner.ts는 첫 후보의 최대 2회 요청, 빈 prefill·사실 검증 재시도와 내용을 노출하지 않는 진단 로그를 담당한다. suggestion/suggestion-policy-helpers.ts(2026-11-13 W5-11/G7)는 오타 교정 문맥 적용(applyCorrectionToContext), 중복 판정(sameCandidateText/isDuplicateCandidateText), 개인화 후보 재정렬(rerankFilledCandidates), 정책 정규화(normalizeSuggestionPolicyConfig), 결정 입력 시그니처(buildDecisionInputSignature), 터미널 커서 연속성 복원(resolveTerminalTypingContext) 등 순수 함수를 소유한다. _handleGenerationFailure는 오류·시간 초과·빈 후보의 개인 후보 유지 → 로컬 기억 폴백 → 실패 종료 순서를 한 곳에서 보장한다. _startGenerationSession은 취소기·세대 토큰·타임아웃을 포함한 GenerationSession을 반환하고, _prepareFirstCandidate는 기억 수집·오타 교정·모델 요청과 첫 응답을 FirstCandidatePreparation으로 반환하며 실패 시에도 폴백용 기억 문맥을 보존한다. _isGenerationResultCurrent는 세대 불일치·문맥 경과·접두 변경을 순서대로 검사하고, _publishFirstCandidate는 개인 후보 병합·성공 기록·후속 채우기 시작을 담당하고, _appendFilledCandidate는 후속 후보의 추가·개인화 재정렬·표시·세션 행 후보 수 갱신을 한 곳에서 처리한다. SuggestionService는 입력 정책, 세대/취소, 표시, 수락만 조율한다. 모델 요청과 로컬 기억 게시는 각각 SuggestionModelRequestInput·LocalMemoryPublication 객체를 사용하며, 동작 변경 없는 순수 리팩토링이다.
The existing input-telemetry-handlers and suggestion-handlers IPC extensions expose the receipt and the flow/suggestion summaries. Receipt reads or deletes report an IPC error when storage access fails; they do not return invented counts or claim a purge succeeded. The receipt is local-only: input_activity, typing_samples, and suggestions retain 30 days; personal_phrases has no age-based automatic expiry and is removed by individual deletion, delete-all, or consent withdrawal. Smart Exclusion never records password evidence and recommends only the current app after at least four observations, zero readable results, and a problematic ratio of at least 75%; its one-click action adds an exclusion and never auto-excludes. Shortcut Safety Audit is core auditKeyBindingMap applied by the Settings UI; it surfaces invalid/conflict issues while preserving the existing hold/double-press exceptions.
Account / infra / monetization
| Service | Purpose |
|---|---|
ConfigService |
electron-store AppConfig (configGet/Set, defaults) |
LicenseService |
Freemium tiers, quotas (daily_usage), activation, upgrade prompts |
CloudSyncService |
Supabase auth + lifecycle: per-user DB switching, local-mode import on first sign-in, Realtime (ws transport) + 5-min heartbeat, device check-in, debounced flush/pull triggers. The sync itself lives in services/sync/ |
services/sync/ |
SyncEngine (backfill once per user DB → push outbox → pull by server-clock keyset cursor → tombstones → memo-tag reconcile), sync-outbox (sync_outbox/sync_state tables), sync-adapters (history, dictionary, meetings, meeting memos/documents, custom commands, templates), memo-tag-sync, supabase-sync-remote, device-registration, realtime-transport, settings-sync (mobile user_settings ↔ language/theme/defaultLLMAction/activeInstructionId), audio-sync (recording upload to the audio bucket, remote cleanup on delete), knowledge adapter (source-text chunks), transcript-sync (meeting segments from the desktop transcript), builtin-instruction-sync (preset prompt edits ↔ server preset rows). Services record changes with getCloudSyncService().pushOne/pushDelete(entity, id); pulled rows are applied without re-queuing |
CloudSTTService |
Thin cloud STT wrapper over D3ROCloudDriver |
UpdateService |
electron-updater (canonical Forgejo feed, channels, mandatory/full-vs-delta policy, staged rollout). Native consent/restart dialogs resolve update.dialog.* through a fresh getPopupTranslator() at each prompt, so a runtime locale change is reflected without rebuilding the prompter; unit tests cover required/optional paths and a ko→en switch on the same prompter. update-dialog-preview.ts exposes only the explicit --preview-update-dialogs manual-QA entry described above. The current-source package was built and its native ABI verified; packaged dialog appearance/interaction remains unverified until that executable is run on the interactive desktop. |
AutoLaunchService |
OS login-item auto-start |
LoggerService |
electron-log wrapper + category loggers |
| upgrade / billing | No in-app checkout. license-handlers.ts LICENSE.OPEN_BILLING opens billingUrl({ tier }) (web Payple); tier returns via license:tierChanged. Stripe payment-handlers.ts·CheckoutModal·payment:* IPC removed 2026-09-26 |
Ads (services/ads/)
| File | Purpose |
|---|---|
AdMediationEngine |
Multi-ad mediation + header bidding |
AdSettlementService |
Revenue settlement, withholding, payout ledger |
BaseAdAdapter / UnavailableAdAdapter |
Adapter contract + fail-closed base + declarative UNAVAILABLE_AD_NETWORK_SPECS / createUnavailableAdAdapters() registry (2026-11-13: consolidated 9 empty subclass files into one table-driven registry) |
DirectHouseSponsorAdapter |
Real configurable adapter: bids/reports against an operator HTTPS endpointUrl (AdNetworkConfig.endpointUrl), validates creatives, fail-closed (adapter_not_configured) when unconfigured |
| 9 unavailable network specs (AppLovin, Carbon, EthicalAds, GoogleAdManager, InMobi, Mintegral, Playwire, PubMatic, Unity) | Instantiated via createUnavailableAdAdapters() — registered, no live bids (provider_not_integrated) |
3. IPC layer
Registry: src/main/ipc/index.ts calls 31 registerXHandlers() in fixed order. Channel SSOT: packages/core/src/ipc-channels.ts.
| Handler | Channel group(s) |
|---|---|
ads-handlers |
ADS |
audio-handlers |
AUDIO |
caption-handlers |
CAPTION + SYSTEM_AUDIO |
chain-handlers |
CHAIN |
cloud-sync-handlers |
CLOUD_SYNC; forwards engine data-changed as app:dataChanged { type: 'cloud-sync', entities } so History/Dashboard/Dictionary/Commands/Meetings/Templates reload |
config-handlers |
CONFIG |
context-handlers |
CONTEXT |
dictionary-handlers |
DICTIONARY |
file-transcription-handlers |
FILE_TRANSCRIPTION |
history-handlers |
HISTORY (incl. history:setFavorite, history:getAudio → local bytes or signed URL) + stats:getSummary |
unified-search-handlers |
UNIFIED_SEARCH (unifiedSearch:query), returning IPCResult<UnifiedSearchResponse> |
input-telemetry-handlers |
INPUT_TELEMETRY |
instruction-handlers |
INSTRUCTION |
keybinding-handlers |
KEYBINDING |
license-handlers |
LICENSE |
llm-handlers |
LLM + llm:premium:* + ONLINE_AUTH |
meeting-doc-template-handlers |
MEETING_DOC_TEMPLATE |
meeting-mode-handlers |
MEETING_MODE + MEETING_CHAT; meetingMode:importRecording opens the shared media picker and returns the completed meeting detail |
meeting-summary-handlers |
MEETING_SUMMARY |
memo-handlers |
MEMO |
rag-handlers |
RAG |
stt-handlers |
STT |
suggestion-handlers |
SUGGESTION + POPUP_SUGGESTION |
support-handlers |
SUPPORT (redactPII(..., 'mask') applied to diagnostic descriptions and logs before ticket creation, 2026-11-13 W5-01) |
system-handlers |
SYSTEM |
template-handlers |
DICTATION_TEMPLATE |
voice-action-handlers |
VOICE_ACTION |
voice-command-handlers |
VOICE_COMMAND |
voice-conversation-handlers |
VOICE_CONVERSATION |
voice-handlers |
VOICE |
window-handlers |
WINDOW + SYSTEM.OPEN_EXTERNAL |
The KEYBINDING group replaced the old per-action HOTKEY group. HOTKEY had 14 channels — a get/set pair per action plus three that were never implemented — so every new action meant new channels. KEYBINDING is 9 channels that take the action as a parameter: getMap, setBindings, resetAction, resetAll, validate, isEnabled, setEnabled, plus the triggered / changed events (packages/core/src/ipc-channels.ts:104). Adding an action now costs zero channels.
LLM.PROCESS normalizes at the IPC boundary. The handler runs buildInstructionInvocation itself when action === 'custom' with a customPrompt (llm-handlers.ts:94-108), so the renderer passes the raw instruction text and never duplicates the substitution or argument-placement rules. This is what makes VoiceModeService, ChainService, and LLM.PROCESS literally share one implementation. No channel or type changed for this; LLMProcessParams is unchanged.
Preload exposes window.electronAPI with 37 namespaces: platform, audio, config, voice, stt, runtime, keybinding, llm (incl. premium), history, unifiedSearch, dictionary, stats, window, system, instruction, app, memo, voiceCommand, context, chain, caption, license, fileTranscription, meetingSummary, dictationTemplate, rag, voiceAction, voiceConversation, meetingMode, meetingChat, meetingDocTemplate, cloudSync, onlineAuth, ads, support, inputTelemetry, suggestion. The keybinding bridge is 9 methods mirroring the channels above (src/preload/index.ts:323), replacing the 11-method hotkey bridge. Envelope: IPCResult<T> (success/error); app.onDataChanged is the global refresh channel.
4. Windows & popups
windows/WindowManager.ts manages 7 windows: main (borderless, custom TitleBar; macOS hiddenInset) created at boot, and 6 vanilla popups (recording-tip, result-popup, history-popup, command-popup, caption-overlay, suggestion-overlay) created lazily on first demand (peekHistoryPopupWindow / peekCommandPopupWindow inspect existing windows without instantiating hidden Chromium renderers at startup). Injects popup theme CSS + i18n strings via @d3ro/i18n/main; 2-phase resize. windows/TrayManager.ts — tray icon + menu + double-click show.
Popup invariants (each shipped broken once — do not regress):
- 팝업 HTML의 스크립트는 반드시
<script type="module">로 선언한다. Vite는 모듈 스크립트만 번들에 포함하므로 classic<script src="./script.js">는 dev에서만 로드되고 패키징 산출물에서는 파일 자체가 사라진다(오버레이가 정적 HTML로 멈춘 원인).scripts/ci/verify-desktop-renderer-bundles.mjs가 빌드 HTML이 참조하는 모든 로컬 asset의 존재를 검사한다. - 렌더러 로드 전의
webContents.send는 조용히 버려진다. 팝업 전송은sendToPopupWindow를 쓰고, 이 함수가did-finish-load까지 메시지를 보관했다가 전달한다.attachPopupLifecycle이 로드 상태 추적·테마 주입·팝업 렌더러 진단 로그를 한 곳에서 묶는다. - 팝업 표시는
presentPopup으로 통일한다(showInactive+ topmost 재선언 +moveTop+webContents.invalidate). 한 번hide()된 팝업이 두 번째 표시에서 z-order/repaint를 잃어 보이지 않던 문제를 막는다. suggestion-overlay는 비축소 UIA 선택이 생기면selection-active로 즉시 clear한다. 실제 표시 중일 때만 5초 이후에도 UIA를 주기 검증하고 mouse-up 뒤 120 ms에 다시 검증한다. 입력 focus가 사라져available/editable이 false가 되거나 선택이 생기면 요청을 abort하고 hide한다. X는 renderer panel을 먼저 즉시 숨긴 뒤 main IPC가BrowserWindow.hide()를 직접 호출하고 service dismiss를 수행한다. 늦은 결과는 generation token으로 재표시할 수 없다.
Vanilla popups (src/renderer/popups/):
| Popup | Purpose |
|---|---|
recording-tip |
9-bar waveform indicator, partial transcript |
result-popup |
Transcription result + copy, auto-close with hover pause |
history-popup |
Recent transcriptions; ↑↓/Enter/1-9/ESC. Opened by the history-popup action (default Ctrl+Shift+V, rebindable) |
command-popup |
Command selection. Opened by the command-popup action (default Ctrl+Shift+C, rebindable) |
caption-overlay |
Live caption overlay (font/opacity/maxLines) |
suggestion-overlay |
Next-sentence ghost text: caret-anchored (anchorFloatingPanel), non-focusable, click-through unless suggestionOverlayInteractive; accept/next/dismiss come from global key bindings (the window never owns focus). Its narrow layout shows a one-line local provenance/count summary only, never raw memory evidence. Presentation includes candidate, generating, warm-up and partial-text states; its X hides the panel before main-process dismissal. |
5. Renderer IA
Routing is state-based in renderer/navigation/app-navigation.tsx (AppRoute, params, history/back), no react-router. Ctrl+K opens the search route and Ctrl+1..3 opens Home/Records/Meetings. Since 2026-11-13, AppLayout.tsx keeps HomePage eager for instant first paint and code-splits the 7 secondary pages (UnifiedSearchPage, HistoryPage, MeetingModePage, VoiceConversationPage, CommandsPage, DictionaryPage, KnowledgeBasePage) plus 6 modals (SettingsModal, OnboardingModal, OllamaGuideModal, SupportModal, LicenseModal, RewardedQuotaModal) via React.lazy + conditional mounting, cutting the initial renderer JS chunk from 4,596.8 KB to 1,591.3 KB (-65.4%).
| Page | Route | Feature |
|---|---|---|
HomePage |
home | Today summary, recording/meeting/caption starts, current blockers, recent records/meetings, file transcription, bounded free usage |
UnifiedSearchPage |
search | Record+meeting search, newest-first merged results, source-aware detail navigation; source/tests complete, Electron manual proof pending |
HistoryPage |
records | History & memory timeline; search, tag/mode/favorite filters, pagination, export/delete |
DictionaryPage |
dictionary | Custom vocabulary editor |
CommandsPage |
commands | Custom instructions + voice keyword rules + LLM chains + dictation templates |
VoiceConversationPage |
conversation | Duplex voice assistant (local pipeline vs OpenAI Realtime) |
KnowledgeBasePage |
knowledge | Local RAG: add/index docs, semantic query, reindex/remove |
MeetingModePage |
meetings | Meeting studio: live or imported-media transcript with progress/cancel, linked audio, memos, minutes/doc generation/export, diarization |
Modals/components: SettingsModal (349 lines after 2026-11-13 W5-11/G7 extraction into components/settings/{GeneralSettingsTab,AudioSettingsTab,LlmSettingsTab,AboutSettingsTab,settings-model-helpers}.ts(x) alongside SttSettingsTab, InputInsightsPanel, LicenseTab, CloudSyncSection), LicenseModal, InputConsentPanel (receipt + current-app exclusion recommendation), InputInsightsView (flow, friction and app-quality summaries), OnboardingModal, UpgradePromptModal, TemplateSection, FileDropZone, OllamaGuideModal, CodexOAuthGuideModal, TitleBar, meeting components (TranscriptTab + EditableSegment include color-coded speaker badges and multi-speaker filter chips, 2026-11-13 GAP-MTG-02), voice-conversation, support (SupportModal), ads (AdBanner, RewardedQuotaModal), shared cards.
Key-binding UI lives in components/keybinding/ (Keycap, KeyBindingPicker, KeyBindingField, translation-key), embedded in the Settings General tab (SettingsModal.tsx:239) — one field per action plus a global on/off switch. The picker offers both key recording and a searchable grouped dropdown (MUI Autocomplete over KEY_CATALOG, KeyBindingPicker.tsx:536). It replaced HotkeyRecordModal. renderer/utils/format-hotkey.ts is now a 17-line platform adapter only; key names, modifier glyphs, and join rules come from @d3ro/core/keybinding.
Hooks: useRealtimeConversation (OpenAI Realtime WebRTC), useLicenseState, useProFeature, useKeyBindingMap (subscribes to keybinding:changed; the dashboard renders the live dictation binding through BindingKeycaps), useInputInsights (telemetry + suggestion state + weekly summary + phrases; the Dashboard shows a weekly input card when collection is on).
Test-facing surface (stable, Playwright E2E depends on these): AppSidebar nav-* buttons expose data-testid="nav-${route}" + aria-current="page"; SettingsModal tabs expose data-testid="settings-tab-${id}"; onboarding mode cards expose onboarding-mode-local|online; EmptyStateCard exposes empty-state-card. Add a testid before writing an E2E journey; do not select by MUI classes.
DB schema (src/main/db/schema.ts, drizzle SQLite): history, dictionary, stats, memo_tags, daily_usage, rag_documents, rag_chunks (incl. embedding_blob Float32 binary vector), meeting_sessions, meeting_memos (incl. immutable revision CAS counter, 2026-11-13 GAP-SYNC-08), meeting_documents, input_activity (hour × app counters), typing_samples, personal_phrases, suggestions, phrase_edges, sync_outbox, sync_state, plus FTS5 tokenize = 'trigram' virtual tables history_fts, meeting_sessions_fts, and meeting_memos_fts (db/index.ts: WAL + synchronous = NORMAL + temp_store = MEMORY + cache_size = -16000 + mmap_size = 67108864). PersonalGraphService.indexText and runMaintenance execute inside single SQLite transactions, and InputTelemetryService only attaches high-frequency mousemove/wheel hooks and _flushTimer when inputTelemetryEnabled && !inputTelemetryPaused.
6. Desktop status summary
-
2026-10-30 high-load typing protection (INPUT-04/19, GAP-INPUT-11): UIA snapshots are deduplicated and discarded on input/focus/native activity changes before updating text baselines.
TypoCorrectionServicepasses a replacement-time guard throughbootstraptoTextInsertService; the guard checks monotonic input revision, nativeGetLastInputInfotoken, and foreground after queue waits. Synthetic-key suppression starts only at actual injection. Automatic Windows corrections require a single native input batch; partial/unknownSendInputresults fail without replaying deletion. A successful batch enqueue does not prove that the target app/IME processed the intended text. The native token is session-scoped and only compared for equality (same-tick input remains limited by Windows timestamp resolution). -
Local inference worker headroom:
utils/inference-cpu-budget.tsandsidecar/cpu_budget.pyusemax(1, min(4, available CPUs - 2))for loopback Ollama generate/stream/chat and Whisper primary/auxiliary/CPU fallback. Remote Ollama defaults are unchanged. This bounds inference workers, not model-load IO, GPU usage, total CPU percentage, or concurrent engines; CPU-only inference may slow down. Existing per-letter terminal IME probing can still synchronously wait up to 30 ms and needs separate profiling before changing its correctness-sensitive behavior. -
Verification for this change: 14 focused desktop test files / 160 tests and Python CPU-budget + device-fallback tests / 23 tests pass; touched TypeScript ESLint and desktop production bundle build pass. Compiler-API comparison against tracked HEAD sources gives renderer strict diagnostics 0→0 and main strict diagnostics 16→14, with no introduced diagnostics. Remaining errors are in support handlers, KeyBindingService, PersonalGraphService and core typo-correction; full strict typecheck remains failing. Windows GetLastInputInfo read-only native probe succeeds. No interactive high-load typing run or packaged-sidecar rebuild is claimed.
-
2026-10-30 deterministic source A/B:
node scripts/verification/typing-race-proof.mjsoverlays onlyInputTelemetryService,TypoCorrectionServiceandTextInsertServicefrom HEADf49acf0without modifying the checkout; dependencies, test doubles and tests remain current in both runs. This is not a complete historical-release A/B. HEAD passed 1/17 (16 failures); current sources passed 17/17. Three delayed-queue cases independently isolate timestamp, monotonic revision and native-tick guards: the old services corrupted the next word to오늘 잘 됬어요 안녕됐어요while the fix preserved오늘 잘 됬어요 안녕하세요. The other 13 failures test snapshot invalidation/metadata/coalescing contracts, not 13 additional text-loss reproductions. The unchanged-input control still corrects in both runs. Source hashes remained unchanged during the run. Evidence:report.json. This proves the service-level race with an in-memory text sink and delayed UIA mocks; it does not reproduce the exact user symptom in Windows IME or under actual CPU/model load. -
Interactive verification infrastructure (prepared, not yet run):
scripts/verification/README.mddefines the matrix, evidence requirements and Windows contracts.typing-fixturebuilds a dedicated WinForms text sink with scan-code injection, mandatory Hangul/WM_IME_COMPOSITION calibration, 60/30/15 ms pacing, optional bounded CPU phases, focus cancellation and JSON result capture.typing-runtime/build.ps1buildstest-results/typing-load/runtime/D3ROTypingRuntime.exe; this interactive launcher refuses to start while the installedD3RO Voiceprocess is running. The runtime loads the actual product bundle with a separate local-only profile (automatic/model correction, suggestions, learning and telemetry off initially), blocks protocol/OS startup registration, and records product package version plus bundle SHA-256 inruntime.json. Syntax checking and launcher compilation passed; the runtime has not started. Auto-correction on/off, a positive correction control, actual model cold/warm overlap, CPU observations, Chromium/terminal surfaces and physical hardware typing remain GAP-INPUT-11 gates. -
Core dictation/LLM/history pipeline: implemented + tested. The vitest case count in
apps/desktopis 1465 after input intelligence and the Local Flow Intelligence extension; playwright e2e is separate. Read the pass numbers together with thebetter-sqlite3ABI the tree is built for (11GAP-INFRA-06) — they are not comparable across configurations:- Host Node ABI (2026-09-21, before the LLM fix): 1311 / 1314 passing. The three failures are environment-dependent rather than regressions — two need a local sidecar venv or embedding server, one pins an error message that has since changed (
11GAP-QA-02). This configuration has not been re-measured since the LLM fix. - Electron ABI (2026-09-21, after input intelligence):
366 failed | 1042 passed (1408), against a clean-tree baseline of366 failed | 994 passed (1360)in the same configuration — identical failure count, +48 passed, zero new failures. 365 of those 366 aretests/red/*.usecase.test.tsfiles dying at DB creation because of the ABI mismatch, not assertions. - Native-ABI mismatch run (2026-09-23,
1.5.0release verification):366 failed | 1099 passed (1465). Thebetter-sqlite3module is still built for Electron ABI 130 while the host Node is ABI 131, so this is the same mismatch configuration. The failure count is unchanged (365 ABI + 1 stale sidecar error-message assertion,11GAP-QA-02) and the passed count rose with the new tests, so there are zero new failures. Full strict/full suite and Electron GUI verification remain open gates, not evidence of green.
- Host Node ABI (2026-09-21, before the LLM fix): 1311 / 1314 passing. The three failures are environment-dependent rather than regressions — two need a local sidecar venv or embedding server, one pins an error message that has since changed (
-
Input intelligence (
2026-09-21): telemetry capture, weekly insights, next-sentence ghost text and phrase learning are implemented; real typing has verified capture → UIA snapshot → policy decisions, while overlay position/appearance, accept-insert, password blocking and weekly numbers remain manual gates (11GAP-INPUT-01). The 24.7 s cold / 4.9 s warm figures are a historical latency diagnosis, not current behaviour. 2026-09-22 operating verification: at 19:14:12 app startup warmup pluskeep_alive: 30mheldgemma4:e4bresident through 19:44:12 at VRAM 3,226,342,521 bytes and context 4096. Windows GPU Engine PID sampling found no active Ollama compute then, so this was forced residency rather than infinite inference. Separately, 19:11:17–19:11:58 logs show automatic suggestions repeatedly generated under the old 900 ms / 12 per min / 5-candidate / 128-token / one-character-growth policy: a permitted burst, not proof of a single stuck request. The replacement guard removes boot warmup, useskeep_alive: 2m, and sets 600 ms debounce, 5 s minimum interval, 6 per min (hard maximum 12), 3 candidates, 64 tokens, 12-character growth and 8 s timeout.LocalLLMServiceadds per-request cancellation, bounded generate/stream/chat calls, requireddoneframes and incomplete cleanup; voice cancellation is single-flight and reaches the request; server spawn/polling are deduplicated and disposed. Independent targeted verification passed 6 test files / 69 tests with 0 failures; changed code/tests ESLint andgit diff --checkexited 0. Raw Ollama proof: a cold bounded request hit the client hard timeout at 15.044 s then left/api/psempty and/api/versionrecovered in 80 ms; explicit warmup returned HTTP 200 in 16.639 s; the subsequentnum_predict=1,keep_alive='2m'request returned HTTP 200 in 553 ms withdone:true,eval_count:1,response=OK,done_reason:length, and an observed/api/psexpiry of about 119.9 s. At 19:48:59 +09:00, with no intervening generate/unload/kill/retry, a single/api/psreturned HTTP 200 in 45.8 ms with{models:[]}and/api/versionreturned HTTP 200 in 7.3 ms with0.32.13: raw API expiry/unload evidence only. This does not prove an app restart, GUI overlay or real automatic-typing path, so those remain[~]runtime gates (11GAP-LLM-04, GAP-INPUT-06). -
2026-09-23 overlay lifecycle / Windows child-process audit: focused five test files passed 80 tests; core had 131 passing tests in an earlier independent verification; desktop typecheck/lint, Python
py_compile, andgit diff --checkexited 0. Code and automated-test audit coverswindowsHide:trueon TTS PowerShell, VoiceAction cmd/PowerShell/general exec, audio-device and active-window PowerShell, plus all three ffmpeg paths; with existing SoX/STT/Ollama/sound-effect coverage, no Windows-capable desktop-main child-process call is known to be omitted. This is not Electron GUI runtime proof. Keep INPUT-07 and GAP-INPUT runtime gates[~]until an external-terminalrun-desktop.batrestart confirms dismissal under generation/selection/focus loss and no cmd/PowerShell window recurrence for TTS, voice action, audio enumeration, screen context and file transcription. -
Cross-platform packaging: Windows NSIS (signed,
forceCodeSigning), macOS DMG/ZIP arm64 (ad-hoc signing); auto-update via canonical Forgejo feed with update policy (release/update-policy.json). -
Local-first AI (SoX + faster-whisper sidecar + bundled Ollama) and cloud paths both present.
-
Local STT is packaged (
1.3.0):electron-builder.ymlextraResourcescopiessidecar-dist/sidecar→resources/sidecarandresources/ffmpeg→resources/ffmpeg;scripts/ci/verify-sidecar-bundle.mjsgates packaging. Build locally withnpm --prefix apps/desktop run sidecar:setup && npm --prefix apps/desktop run sidecar:build. The sidecar stays in console mode sostdout/stderrreach the app log (UTF-8, line-buffered); a packaged sidecar must exist or startup fails loudly instead of silently falling back to a system Python. -
All local engine URLs (
LocalSTTService,LocalLLMService,RAGService,OnlineLLMService,STTManager) pass throughsrc/main/utils/loopback.ts, which rewriteslocalhostto127.0.0.1, because some Windows hosts resolvelocalhostto IPv6 only and local engines bind IPv4. -
Meeting intelligence, RAG, voice conversation (local + Realtime), captions, file transcription: implemented.
-
Unified search (2026-09-28): source and automated tests only.
UnifiedSearchService+unifiedSearch:queryIPC + preload bridge +UnifiedSearchPagesearch records and meetings and route to their detail views. The focused service and IPC tests pass; the renderer is source/type-gated, not headful-tested. No external-terminal Electron run has yet provenCtrl+K, result rendering, or both detail transitions; treat the desktop capability as[~]until that run is retained. -
LLM instruction prompts: fixed 2026-09-21 (
9c2b4d4), not yet verified in a running app. Running a custom instruction inserted the instruction's own wording instead of the processed result. Two faults stacked: the instruction was passed in thetextargument with the system-prompt argument left empty, andBASE_SYSTEM_PROMPTShas nocustomkey so resolution fell back torefinesilently — the model polished the instruction and the transcript never reached it; separately, only{{text}}was substituted and none of the five built-in presets use it ({{targetLanguage}},{{userPrompt}}, or no placeholder), so the substitution was a no-op from the day it was written. Introduced infea923d(2026-04-05) and present in every releasev0.1.0-alpha..v1.4.0— the path never worked; this is not a regression. Plain actions (refine/summarize/grammar/expand) were unaffected and are now pinned by regression cases. The fix routes all three entry paths throughllm-prompts.ts(see §2) and additionally corrects two things found alongside it: a voice shortcut naming an instruction was nullified by thedefaultLLMAction === 'none'gate (VoiceModeService.ts:779), and the commands-page pipeline bench calledllm.generate, which preload does not expose, so every run threw and thecatchdisplayed the input as if it had succeeded — a fail-closed violation that is the reason the bug went unnoticed for five months (CommandsPage.tsx:180-205, now onllm.processwith failures rendered as failures).- Verification limits — do not read this as verified. Unit tests pass (
llm-prompts.test.ts21,llm-handlers.test.ts7,VoiceModeService.test.ts22,ChainService.test.ts5), and each of the four fixes was reverted individually to confirm the tests actually fail without it.npm run lint(apps/desktop scope) passes;tsconfig.check.jsonerrors went 36 → 35 (thellm.generateerror is gone) with no errors in the touched files. But there is no running-app run, andtests/red/{instruction,chain,voice,config}.usecase.test.ts— precisely the related paths — never executed because of thebetter-sqlite3ABI mismatch. That range is neither passing nor failing; it is untested (11GAP-LLM-02, GAP-INFRA-06).
- Verification limits — do not read this as verified. Unit tests pass (
-
Key bindings: implemented and verified on Windows. Every global shortcut now comes from one contract (
@d3ro/core/keybinding) with multiple bindings per action, mouse-button support, and no hardcoded accelerators left inbootstrap.ts. A manual run on 2026-09-21 confirmed legacy migration (custom values preserved), 6 actions loaded, the uiohook keyboard and mouse hook active with zero boot errors, and multi-binding working; contract side ispackages/core117 tests GREEN with no type errors in the key-binding files (11GAP-KEY-01[x]). Two things remain open:KeyBindingServicehas no unit test of its own, and macOS/Linux mouse behavior is unconfirmed (11GAP-KEY-02). The rewrite also fixed a dead hands-free double-press path, an order-dependent reserved-combo check, aglobalShortcut.unregisterAll()that wiped the popup accelerators, and asetEnabled(true)that re-enabled hooking with an empty binding set. -
The same pass fixed an unrelated pre-existing dashboard bug:
caption.onStateChangeddelivers{ state }, butDashboardPagepassed the whole object intosetCaptionState, so the caption status readout never showed the right value (DashboardPage.tsx:148). -
Ad mediation:
DirectHouseSponsorAdapterperforms real configurable REST bids; the other 9 adapters remain fail-closed stubs pending official SDKs (see11-gap-backlog.mdGAP-ADS-01/02). -
Tier resolution now routes through
@d3ro/core/entitlement(resolveEntitlement,normalizeEntitlementTier);useLicenseState.isProincludespro_plus. -
No
TODO/FIXMEmarkers found insrc(grep clean).src/main/types/is an empty directory.
7. Key file anchors
| Thing | Path |
|---|---|
| App entry / deep links | src/main/index.ts |
| Bootstrap order | src/main/bootstrap.ts |
| IPC registry | src/main/ipc/index.ts, src/main/ipc/renderer-broadcast.ts (safeSendToRenderer, broadcastToAllWindows SSOT across 9 IPC/bootstrap modules) |
| IPC channel SSOT | packages/core/src/ipc-channels.ts |
| Key-binding contract SSOT | packages/core/src/keybinding.ts (catalog, actions, validation, conflicts, formatting, parsing) |
| Input intelligence domain SSOT | packages/core/src/input-intelligence.ts (key classification, typed delta, suggestion policy, overlay anchoring, flow/friction aggregation, phrase decay/ranking, provenance and exclusion recommendation) |
| Input intelligence services | src/main/services/InputTelemetryService.ts, SuggestionService.ts, UiaContextService.ts, global-input-hook.ts |
| Foreground window FFI | src/main/utils/win32-foreground.ts (koffi → user32/kernel32) |
| UIA bridge (sidecar) | sidecar/uia_bridge.py + GET /uia/focus in sidecar/main.py |
| Input UI | src/renderer/components/input-insights/{InputConsentPanel,InputInsightsView}.tsx, src/renderer/hooks/useInputInsights.ts, src/renderer/popups/suggestion-overlay/ |
| Local Flow Intelligence tests | tests/main/services/input-flow-domain.test.ts (9), input-flow-services.test.ts (12); targeted suite 21/21 at the documented handoff point |
| Key-binding service / IPC / UI | src/main/services/KeyBindingService.ts, src/main/ipc/keybinding-handlers.ts, src/renderer/components/keybinding/ |
| Preload API | src/preload/index.ts |
| Windows | src/main/windows/WindowManager.ts |
| Voice orchestrator | src/main/services/VoiceModeService.ts |
| LLM prompt / placeholder SSOT | src/main/services/llm-prompts.ts (shared by VoiceModeService, ChainService, ipc/llm-handlers.ts) |
| DB schema | src/main/db/schema.ts |
| Renderer shell / routes | src/renderer/components/AppLayout.tsx |
| Unified search & hardening anchors | src/main/services/UnifiedSearchService.ts (FTS5→LIKE fallback + sort indexes), src/main/ipc/unified-search-handlers.ts, src/renderer/pages/UnifiedSearchPage.tsx, src/main/services/rag/chunk-store.ts, src/main/services/meeting/disk-wav-audio-sink.ts, src/renderer/utils/error-message.ts |
| Update feed SSOT | src/main/update-feed.ts |
| Update policy SSOT | release/update-policy.json + src/main/update-policy.ts |
| Path/loopback resolution | src/main/utils/paths.ts, src/main/utils/loopback.ts |
| Sidecar source / packaging | sidecar/main.py, scripts/setup-sidecar.mjs, scripts/build-sidecar.mjs, scripts/ci/verify-sidecar-bundle.mjs |
2026-11-01 verification: source A/B 2/17 before vs 17/17 current; real MP3/M4A conversion 2/2; account-switch regression 3/3 after reproducing all three failures; local packaged STT transcribed the synthetic 8.498 s fixture exactly. Windows 1.16.0 installer, portable and runtime are published; the rebuilt sidecar passed separate-process startup/health and the packaged Windows keyboard N-API binding loaded successfully without starting hooks. Required sidecar minimum 1.16.0 delivers CPU budgeting and terminal UIA changes. Recovery evidence.
2026-11-02 verification: installed 1.16.0 passed the actual Electron picker→timestamped transcript→local minutes/action-item journey. Native Korean IME passed 7/7 under bounded 100% CPU and 7/7 with a direct 9.389 s gemma4:e4b cold-load request overlapping two typing cases. The Latin-layout control exposed next-key/backspace interleaving; source now waits 140 ms at boundaries and batches the current trailing correction region after a 650 ms pause. Full desktop 2,773 tests and production build pass. The fix was published as Windows v1.16.1 on 2026-11-04; the installed-app Latin control still needs a new GUI run. Synthetic WinForms input is not physical-keyboard/Chromium/terminal proof. Verification evidence.
2026-11-14 verification (Wave 6 WS1/WS2 hygiene, optimization & bug hunt): unified safeSendToRenderer and broadcastToAllWindows across 9 IPC/bootstrap modules into src/main/ipc/renderer-broadcast.ts; wired OnlineLLMService stream abort (_abortController in chatStream), 401 token reset, empty data.text guard, and D3ROError wrapping; fixed UnifiedSearchService meetingTotal double-counting, startedAt/createdAt desc sort order, runtime FTS5-to-LIKE fallback (hasAllFtsTables), and DB indexes (idx_meeting_sessions_started_at, idx_meeting_memos_created_at); exported invalidateEmbeddedChunksCache across rag/chunk-store.ts, rag/retrieval.ts, sync/sync-adapters.ts, and db/index.ts, cleared redundant cached embedding JSON strings after Float32Array decode, and guarded malformed Float32 buffers; added MAX_WAV_DATA_BYTES = 0xffffffff - 36 uint32 guard and post-openSync fd cleanup in disk-wav-audio-sink.ts; closed timer/popup hide-show races across AdMediationEngine, AudioCaptureService, LocalLLMService, disposeAudioHandlers, bootstrap.ts, and WindowManager.ts; hardened sidecar/main.py (UploadFile.close(), <2/odd-byte PCM handling, _watch_disconnect, /shutdown _download_thread join). Renderer added src/renderer/utils/error-message.ts (toErrorMessage), unified meeting/pull models, guarded non-finite dates (formatters.ts, UnifiedSearchPage.tsx), cleaned unmount/sequence/timer/listener paths across pages/modals/hooks, and hardened all 6 vanilla JS popups (beforeunload listener/timer cleanup, textContent = '', result-popup prepareVersion/isVisible/rAF cancel, caption-overlay per-line fadeTimer, bounds clamps). @d3ro/desktop Vitest 210 files / 2,865 tests PASS, sidecar Python unittest 33/33 PASS, npm run build + npm run check:desktop-renderer (7 page(s)) PASS; interactive Electron GUI checks remain [~].
2026-11-14 verification (Wave 7 peer swarm input-core + desktop-shell, commits 2a009e4 / fc5d453 / 3f9e412 on main, post-v1.17.1): InputTelemetryService.ts adds TYPO_SETTLE_DELAY_MS (700 ms) / _typoSettleTimer, diffInsertionSpan() for _expectedInserts across punctuation trimming, expired _programmaticKeys cleanup, and _learnFlushTimer.unref(); TypoCorrectionService.ts expands punctuation boundaries (~…?!。), allows boundary-terminated tokens up to MAX_INPLACE_DELETE_CHARS (32) in trailingWordOnly, and skips permanent _markRegionChecked() lockout on trailingWordOnly passes (typo-live-punctuation.test.ts, TypoCorrectionService.test.ts). Main services harden popup/lifecycle/abort paths: WindowManager.hideResultPopup() cancels pendingResultShow and resets lastResultCursor = null (popup-hide-show-race.test.ts); CaptionService.ts records _captionModelId after ensureAuxModel, releases stale loadedAuxModelId, deduplicates concurrent stop()/dispose() via _stopPromise, and resets per-call AbortController + _refineEpoch/_refineRunId (caption-lifecycle-refine.test.ts); VoiceModeService.ts / LlmGateway.ts / LocalLLMService.ts propagate run.controller.signal, track /api/pull abort controllers, prevent polling from restoring availability post-dispose, and wrap dispose aborts into D3ROError(ErrorCode.LLM_ERROR); VoiceConversationService.ts / AudioCaptureService.ts / LocalSTTService.ts / caption-handlers.ts clear _responseAbortController, fix 'stopping' SoX resume & start() _cleanup(), wrap STT download dispose aborts into D3ROError(ErrorCode.STTModelDownloadCancelled), and fail closed with callback({}) in desktopCapturer.getSources. @d3ro/desktop Vitest 212 files / 2,890 tests PASS, sidecar Python unittest 33/33 PASS, npm run build + npm run check:desktop-renderer (7 page(s)) PASS; interactive Electron GUI verification (GAP-INPUT-08/09/10/11) remains [~].