ae7efb6acf
Popup pages loaded their scripts as classic <script src> tags, which the
renderer build never bundles, so an installed app rendered only the static
markup: the recording tip stayed at 0:00 with no wave bars and live captions
showed nothing.
- declare popup scripts as modules so the build emits them, and fail
packaging when a renderer page references an asset that was never produced
- hold popup IPC until the renderer has loaded and re-assert visibility on
every show, so a popup hidden once still appears next time
- surface popup renderer console and load failures in the main log
💘 Generated with Crush
Assisted-by: Crush:deepseek-v4.1-flash
218 lines
14 KiB
Markdown
218 lines
14 KiB
Markdown
# 04 — Desktop App (Electron) Map
|
||
|
||
> Surface: `apps/desktop`
|
||
> Stack: Electron 33 + React 19 + MUI 7 + Vite (electron-vite) + better-sqlite3/drizzle + uiohook-napi + nut-js
|
||
> Source root: `apps/desktop/src` (`main/`, `preload/`, `renderer/`)
|
||
|
||
---
|
||
|
||
## 1. Process architecture
|
||
|
||
| Layer | Path | Contents |
|
||
|---|---|---|
|
||
| Main | `src/main/` | Services, IPC handlers, windows, bootstrap/lifecycle, DB |
|
||
| Preload | `src/preload/` | `index.ts` exposes `window.electronAPI`; `popup.ts` exposes `window.popupAPI` |
|
||
| Renderer | `src/renderer/` | React app: `AppLayout` + 7 pages + modals + 5 vanilla popups |
|
||
|
||
**Main entry** `src/main/index.ts`: sets app name/AppUserModelId, disables GPU acceleration, EPIPE/uncaught handlers, registers `d3ro-voice://` deep-link protocol (Supabase OAuth implicit + PKCE), single-instance lock, then `bootstrap()` + `setupLifecycle()`.
|
||
|
||
**Bootstrap** `src/main/bootstrap.ts`: ordered `BootstrapStep[]` — logger, config, **database (critical)**, license, create-windows (critical), tray, **ipc-handlers (critical)**, custom-instructions, voice-commands, sound-effects, auto-launch, popup-preload, hotkey, voice-mode, llm-polling, meeting-summary-wiring, meeting-mode, cloud-sync, auto-update. Wires VoiceMode events to sound + history persistence.
|
||
|
||
---
|
||
|
||
## 2. Main services (`src/main/services/`)
|
||
|
||
Singleton + `EventEmitter` pattern (`getXService()` accessors).
|
||
|
||
### Core voice pipeline
|
||
| Service | Purpose |
|
||
|---|---|
|
||
| `VoiceModeService` | Orchestrator: 9-state `RecognitionState` + 4-state `AudioState`, dual-condition flush, action queue. Events: session-started/completed/cancelled, transcription-update, audio-level, recognition/audio-state-changed, premium-llm-fallback, error |
|
||
| `AudioCaptureService` | Mic PCM16 16kHz mono (bundled SoX on Windows, node-record-lpcm16 elsewhere). Spawns hidden (`windowsHide`); a missing SoX fails with the exact fix command |
|
||
| `LocalSTTService` | faster-whisper Python sidecar manager (state machine, dual-flush, model download/cancel, background warm-up, live partial transcription). Connects over IPv4 loopback (`getSidecarBaseUrl`) and fails fast with an actionable message when the bundled engine or virtualenv is missing |
|
||
| `HotkeyService` | uiohook-napi global hooking (dictation/hands-free/command/caption). Events: hotkey-pressed/released, double-press, error |
|
||
| `TextInsertService` | Clipboard save→set→Ctrl+V→restore via nut-js |
|
||
| `SoundEffectService` | Preloaded WAV feedback (start/stop/error/cancel/chime) |
|
||
|
||
### STT engine layer (`services/stt/`)
|
||
| File | Purpose |
|
||
|---|---|
|
||
| `STTManager` | Dispatcher across local + 6 cloud providers, auto-fallback (events provider-changed, config-changed, fallback-to-local). `transcribePartial`/`warmUpLocal` route to the local engine only |
|
||
| `types.ts` | `ISTTDriver` contract |
|
||
| `audio-utils.ts` | `pcmToWav`, `createProbeWav` |
|
||
| `drivers/OpenAI|Groq|Deepgram|AssemblyAI|Google|Custom|D3ROCloud` | Provider drivers; `D3ROCloudDriver` uses Supabase access token |
|
||
|
||
### LLM layer
|
||
| Service | Purpose |
|
||
|---|---|
|
||
| `LocalLLMService` | Ollama REST (models, pull w/ progress, server start, NDJSON streaming) |
|
||
| `PremiumLLMService` | Claude via Supabase `llm-proxy`, local fallback |
|
||
| `OnlineLLMService` | JWT-authenticated .NET backend client |
|
||
| `llm-prompts.ts` | `resolveSystemPrompt` SSOT for action prompts |
|
||
|
||
### Memory & knowledge
|
||
| Service | Purpose |
|
||
|---|---|
|
||
| `HistoryService` | SQLite history CRUD/search/stats |
|
||
| `DictionaryService` | Custom vocabulary CRUD/search + cloud sync hooks + JSON/CSV import/export (`dictionary:import`/`export`, save/open dialogs) |
|
||
| `MemoService` | Memo tags over history (`memo_tags`) |
|
||
| `RAGService` | Local RAG: `nomic-embed-text` embeddings, cosine search over `rag_chunks` |
|
||
| `CustomInstructionService` | User LLM commands (5 built-ins) |
|
||
| `VoiceCommandService` | Keyword → command rule matching |
|
||
| `ChainService` | Multi-step LLM pipelines (LLMChain) |
|
||
| `ScreenContextService` | Active-window + selected-text context |
|
||
|
||
### Phase 10+ features
|
||
| Service | Purpose |
|
||
|---|---|
|
||
| `CaptionService` | Live captions from system/loopback audio; caption overlay (events segment, state-changed, session-saved, error) |
|
||
| `FileTranscriptionService` | Audio/video file → ffmpeg → 30s chunks → STT merge (events progress, complete, error, state-changed) |
|
||
| `MeetingSummaryService` | Post-caption LLM summary |
|
||
| `DictationTemplateService` | Field-by-field voice form filling |
|
||
| `VoiceConversationService` | STT→LLM→TTS loop, 10-turn memory |
|
||
| `TTSPlaybackService` | Platform TTS (macOS `say`, Windows SAPI), sentence queue |
|
||
| `VoiceActionService` | Voice → LLM JSON action plan → OS execution (dangerous blocked) |
|
||
|
||
### Phase 12–15
|
||
| Service | Purpose |
|
||
|---|---|
|
||
| `MeetingModeService` | Meeting recording: live transcript, timestamp memos, doc generation/export, diarization |
|
||
| `MeetingDocTemplateService` | Meeting-doc templates (built-ins + CRUD) |
|
||
|
||
### Account / infra / monetization
|
||
| Service | Purpose |
|
||
|---|---|
|
||
| `ConfigService` | electron-store `AppConfig` (`configGet/Set`, defaults) |
|
||
| `LicenseService` | Freemium tiers, quotas (`daily_usage`), activation, upgrade prompts |
|
||
| `CloudSyncService` | Supabase sync, per-user DB switching, history/dictionary/meeting mirror |
|
||
| `CloudSTTService` | Thin cloud STT wrapper over `D3ROCloudDriver` |
|
||
| `UpdateService` | electron-updater (canonical Forgejo feed, channels, mandatory/full-vs-delta policy, staged rollout, restart dialog) |
|
||
| `AutoLaunchService` | OS login-item auto-start |
|
||
| `LoggerService` | electron-log wrapper + category loggers |
|
||
| `checkout`/payment | `payment-handlers.ts` — authenticated Edge-only Stripe/Payple checkout + server readback |
|
||
|
||
### Ads (`services/ads/`)
|
||
| File | Purpose |
|
||
|---|---|
|
||
| `AdMediationEngine` | Multi-ad mediation + header bidding |
|
||
| `AdSettlementService` | Revenue settlement, withholding, payout ledger |
|
||
| `BaseAdAdapter` / `UnavailableAdAdapter` | Adapter contract + fail-closed base |
|
||
| `DirectHouseSponsorAdapter` | **Real configurable adapter**: bids/reports against an operator HTTPS `endpointUrl` (`AdNetworkConfig.endpointUrl`), validates creatives, fail-closed (`adapter_not_configured`) when unconfigured |
|
||
| 9 placeholder adapters (AppLovin, Carbon, EthicalAds, GoogleAdManager, InMobi, Mintegral, Playwire, PubMatic, Unity) | Extend `UnavailableAdAdapter` — registered, no live bids (`provider_not_integrated`) |
|
||
|
||
---
|
||
|
||
## 3. IPC layer
|
||
|
||
Registry: `src/main/ipc/index.ts` calls 29 `registerXHandlers()` in fixed order. Channel SSOT: `packages/core/src/ipc-channels.ts`.
|
||
|
||
| Handler | Channel group(s) |
|
||
|---|---|
|
||
| `ads-handlers` | ADS |
|
||
| `audio-handlers` | AUDIO |
|
||
| `caption-handlers` | CAPTION + SYSTEM_AUDIO |
|
||
| `chain-handlers` | CHAIN |
|
||
| `cloud-sync-handlers` | CLOUD_SYNC |
|
||
| `config-handlers` | CONFIG |
|
||
| `context-handlers` | CONTEXT |
|
||
| `dictionary-handlers` | DICTIONARY |
|
||
| `file-transcription-handlers` | FILE_TRANSCRIPTION |
|
||
| `history-handlers` | HISTORY + `stats:getSummary` |
|
||
| `hotkey-handlers` | HOTKEY |
|
||
| `instruction-handlers` | INSTRUCTION |
|
||
| `license-handlers` | LICENSE |
|
||
| `llm-handlers` | LLM + `llm:premium:*` + ONLINE_AUTH |
|
||
| `meeting-doc-template-handlers` | MEETING_DOC_TEMPLATE |
|
||
| `meeting-mode-handlers` | MEETING_MODE + MEETING_CHAT |
|
||
| `meeting-summary-handlers` | MEETING_SUMMARY |
|
||
| `memo-handlers` | MEMO |
|
||
| `payment-handlers` | PAYMENT |
|
||
| `rag-handlers` | RAG |
|
||
| `stt-handlers` | STT |
|
||
| `support-handlers` | SUPPORT |
|
||
| `system-handlers` | SYSTEM |
|
||
| `template-handlers` | DICTATION_TEMPLATE |
|
||
| `voice-action-handlers` | VOICE_ACTION |
|
||
| `voice-command-handlers` | VOICE_COMMAND |
|
||
| `voice-conversation-handlers` | VOICE_CONVERSATION |
|
||
| `voice-handlers` | VOICE |
|
||
| `window-handlers` | WINDOW + `SYSTEM.OPEN_EXTERNAL` |
|
||
|
||
Preload exposes **`window.electronAPI`** with 33 namespaces: `platform, audio, config, voice, stt, hotkey, llm (incl. premium), history, dictionary, stats, window, system, instruction, app, memo, voiceCommand, context, chain, caption, license, fileTranscription, meetingSummary, dictationTemplate, rag, voiceAction, voiceConversation, meetingMode, meetingChat, meetingDocTemplate, cloudSync, onlineAuth, ads, support, payment`. Envelope: `IPCResult<T>` (success/error); `app.onDataChanged` is the global refresh channel.
|
||
|
||
---
|
||
|
||
## 4. Windows & popups
|
||
|
||
`windows/WindowManager.ts` creates 6 windows: main (borderless, custom TitleBar; macOS `hiddenInset`), recording-tip, result-popup, history-popup, command-popup, caption-overlay. Injects popup theme CSS + i18n strings; 2-phase resize. `windows/TrayManager.ts` — tray icon + menu + double-click show.
|
||
|
||
**Popup invariants** (each shipped broken once — do not regress):
|
||
|
||
- 팝업 HTML의 스크립트는 반드시 `<script type="module">`로 선언한다. Vite는 모듈 스크립트만 번들에 포함하므로 classic `<script src="./script.js">`는 dev에서만 로드되고 패키징 산출물에서는 파일 자체가 사라진다(오버레이가 정적 HTML로 멈춘 원인). `scripts/ci/verify-desktop-renderer-bundles.mjs`가 빌드 HTML이 참조하는 모든 로컬 asset의 존재를 검사한다.
|
||
- 렌더러 로드 전의 `webContents.send`는 조용히 버려진다. 팝업 전송은 `sendToPopupWindow`를 쓰고, 이 함수가 `did-finish-load`까지 메시지를 보관했다가 전달한다. `attachPopupLifecycle`이 로드 상태 추적·테마 주입·팝업 렌더러 진단 로그를 한 곳에서 묶는다.
|
||
- 팝업 표시는 `presentPopup`으로 통일한다(`showInactive` + topmost 재선언 + `moveTop` + `webContents.invalidate`). 한 번 `hide()`된 팝업이 두 번째 표시에서 z-order/repaint를 잃어 보이지 않던 문제를 막는다.
|
||
|
||
Vanilla popups (`src/renderer/popups/`):
|
||
| Popup | Purpose |
|
||
|---|---|
|
||
| `recording-tip` | 9-bar waveform indicator, partial transcript |
|
||
| `result-popup` | Transcription result + copy, auto-close with hover pause |
|
||
| `history-popup` | Recent transcriptions; ↑↓/Enter/1-9/ESC |
|
||
| `command-popup` | Command selection (Ctrl+Shift+C) |
|
||
| `caption-overlay` | Live caption overlay (font/opacity/maxLines) |
|
||
|
||
---
|
||
|
||
## 5. Renderer IA
|
||
|
||
Routing is state-based in `AppLayout.tsx` (`Route` union + `NAV_ITEMS`), no react-router.
|
||
|
||
| Page | Route | Feature |
|
||
|---|---|---|
|
||
| `DashboardPage` | dashboard | Voice cockpit: hero, bento tiles, multi-engine hub (STT/LLM), telemetry, recent history, file drop |
|
||
| `HistoryPage` | history | History & memory timeline; search, tag filter, pagination, export/delete |
|
||
| `DictionaryPage` | dictionary | Custom vocabulary editor |
|
||
| `CommandsPage` | commands | Custom instructions + voice keyword rules + LLM chains + dictation templates |
|
||
| `VoiceConversationPage` | conversation | Duplex voice assistant (local pipeline vs OpenAI Realtime) |
|
||
| `KnowledgeBasePage` | knowledge | Local RAG: add/index docs, semantic query, reindex/remove |
|
||
| `MeetingModePage` | meeting | Meeting studio: live transcript, memos, doc generation/export, diarization |
|
||
|
||
Modals/components: `SettingsModal` (tabs General/Audio/STT/LLM/License/Cloud/About), `LicenseModal`, `LicenseTab`, `CloudSyncSection`, `OnboardingModal`, `UpgradePromptModal`, `ProBadge`, `TemplateSection`, `FileDropZone`, `HotkeyRecordModal`, `OllamaGuideModal`, `CodexOAuthGuideModal`, `TitleBar`, `StatusBar`, meeting components (9), voice-conversation, payment (`CheckoutModal`, `checkout-flow.ts`), support (`SupportModal`), ads (`AdBanner`, `RewardedQuotaModal`), shared cards.
|
||
|
||
Hooks: `useRealtimeConversation` (OpenAI Realtime WebRTC), `useLicenseState`, `useProFeature`.
|
||
|
||
DB schema (`src/main/db/schema.ts`, drizzle SQLite): `history`, `dictionary`, `stats`, `memo_tags`, `daily_usage`, `rag_documents`, `rag_chunks`, `meeting_sessions`, `meeting_memos`, `meeting_documents`.
|
||
|
||
---
|
||
|
||
## 6. Desktop status summary
|
||
|
||
- Core dictation/LLM/history pipeline: **implemented + tested** (~590 desktop tests; vitest + playwright).
|
||
- Cross-platform packaging: Windows NSIS (signed, `forceCodeSigning`), macOS DMG/ZIP arm64 (ad-hoc signing); auto-update via canonical Forgejo feed with update policy (`release/update-policy.json`).
|
||
- Local-first AI (SoX + faster-whisper sidecar + bundled Ollama) and cloud paths both present.
|
||
- **Local STT is packaged** (`1.3.0`): `electron-builder.yml` `extraResources` copies `sidecar-dist/sidecar` → `resources/sidecar` and `resources/ffmpeg` → `resources/ffmpeg`; `scripts/ci/verify-sidecar-bundle.mjs` gates packaging. Build locally with `npm --prefix apps/desktop run sidecar:setup && npm --prefix apps/desktop run sidecar:build`. The sidecar stays in console mode so `stdout`/`stderr` reach the app log (UTF-8, line-buffered); a packaged sidecar **must** exist or startup fails loudly instead of silently falling back to a system Python.
|
||
- All local engine URLs (`LocalSTTService`, `LocalLLMService`, `RAGService`, `OnlineLLMService`, `STTManager`) pass through `src/main/utils/loopback.ts`, which rewrites `localhost` to `127.0.0.1`, because some Windows hosts resolve `localhost` to IPv6 only and local engines bind IPv4.
|
||
- Meeting intelligence, RAG, voice conversation (local + Realtime), captions, file transcription: implemented.
|
||
- **Ad mediation**: `DirectHouseSponsorAdapter` performs real configurable REST bids; the other 9 adapters remain fail-closed stubs pending official SDKs (see `11-gap-backlog.md` GAP-ADS-01/02).
|
||
- Tier resolution now routes through `@d3ro/core/entitlement` (`resolveEntitlement`, `normalizeEntitlementTier`); `useLicenseState.isPro` includes `pro_plus`.
|
||
- No `TODO`/`FIXME` markers found in `src` (grep clean). `src/main/types/` is an empty directory.
|
||
|
||
---
|
||
|
||
## 7. Key file anchors
|
||
|
||
| Thing | Path |
|
||
|---|---|
|
||
| App entry / deep links | `src/main/index.ts` |
|
||
| Bootstrap order | `src/main/bootstrap.ts` |
|
||
| IPC registry | `src/main/ipc/index.ts` |
|
||
| IPC channel SSOT | `packages/core/src/ipc-channels.ts` |
|
||
| Preload API | `src/preload/index.ts` |
|
||
| Windows | `src/main/windows/WindowManager.ts` |
|
||
| Voice orchestrator | `src/main/services/VoiceModeService.ts` |
|
||
| DB schema | `src/main/db/schema.ts` |
|
||
| Renderer shell / routes | `src/renderer/components/AppLayout.tsx` |
|
||
| Update feed SSOT | `src/main/update-feed.ts` |
|
||
| Update policy SSOT | `release/update-policy.json` + `src/main/update-policy.ts` |
|
||
| Path/loopback resolution | `src/main/utils/paths.ts`, `src/main/utils/loopback.ts` |
|
||
| Sidecar source / packaging | `sidecar/main.py`, `scripts/setup-sidecar.mjs`, `scripts/build-sidecar.mjs`, `scripts/ci/verify-sidecar-bundle.mjs` |
|