2d585bfc29
Local dictation had never produced a transcript on an installed build. The engine itself was healthy; every connection to it was broken. Installed builds shipped no speech engine at all: the packaging config had no entry for the faster-whisper sidecar and no pipeline step built one, so the app always fell back to a system Python without the runtime. Development was broken too, because the sidecar and SoX paths were resolved against the Vite output directory instead of the app root, which also meant recording failed with a SoX ENOENT. On hosts where localhost resolves only to IPv6, every local request was refused outright, which silently disabled both local transcription and the local LLM. The sidecar is now built and bundled (including the Silero VAD data it needs), gated by a packaging check that fails when the engine or its data is missing. Paths are discovered from the app root and fail loudly when the engine is absent. Local engine URLs are normalized to the IPv4 loopback, decoding is tuned so repeated hallucinations cannot compound (the same transcript now takes about a fifth of the time), the engine is warmed up at startup, and holding the hotkey now shows the text forming live in the recording tip.
212 lines
13 KiB
Markdown
212 lines
13 KiB
Markdown
# 04 — Desktop App (Electron) Map
|
||
|
||
> Surface: `apps/desktop`
|
||
> Stack: Electron 33 + React 19 + MUI 7 + Vite (electron-vite) + better-sqlite3/drizzle + uiohook-napi + nut-js
|
||
> Source root: `apps/desktop/src` (`main/`, `preload/`, `renderer/`)
|
||
|
||
---
|
||
|
||
## 1. Process architecture
|
||
|
||
| Layer | Path | Contents |
|
||
|---|---|---|
|
||
| Main | `src/main/` | Services, IPC handlers, windows, bootstrap/lifecycle, DB |
|
||
| Preload | `src/preload/` | `index.ts` exposes `window.electronAPI`; `popup.ts` exposes `window.popupAPI` |
|
||
| Renderer | `src/renderer/` | React app: `AppLayout` + 7 pages + modals + 5 vanilla popups |
|
||
|
||
**Main entry** `src/main/index.ts`: sets app name/AppUserModelId, disables GPU acceleration, EPIPE/uncaught handlers, registers `d3ro-voice://` deep-link protocol (Supabase OAuth implicit + PKCE), single-instance lock, then `bootstrap()` + `setupLifecycle()`.
|
||
|
||
**Bootstrap** `src/main/bootstrap.ts`: ordered `BootstrapStep[]` — logger, config, **database (critical)**, license, create-windows (critical), tray, **ipc-handlers (critical)**, custom-instructions, voice-commands, sound-effects, auto-launch, popup-preload, hotkey, voice-mode, llm-polling, meeting-summary-wiring, meeting-mode, cloud-sync, auto-update. Wires VoiceMode events to sound + history persistence.
|
||
|
||
---
|
||
|
||
## 2. Main services (`src/main/services/`)
|
||
|
||
Singleton + `EventEmitter` pattern (`getXService()` accessors).
|
||
|
||
### Core voice pipeline
|
||
| Service | Purpose |
|
||
|---|---|
|
||
| `VoiceModeService` | Orchestrator: 9-state `RecognitionState` + 4-state `AudioState`, dual-condition flush, action queue. Events: session-started/completed/cancelled, transcription-update, audio-level, recognition/audio-state-changed, premium-llm-fallback, error |
|
||
| `AudioCaptureService` | Mic PCM16 16kHz mono (bundled SoX on Windows, node-record-lpcm16 elsewhere). Spawns hidden (`windowsHide`); a missing SoX fails with the exact fix command |
|
||
| `LocalSTTService` | faster-whisper Python sidecar manager (state machine, dual-flush, model download/cancel, background warm-up, live partial transcription). Connects over IPv4 loopback (`getSidecarBaseUrl`) and fails fast with an actionable message when the bundled engine or virtualenv is missing |
|
||
| `HotkeyService` | uiohook-napi global hooking (dictation/hands-free/command/caption). Events: hotkey-pressed/released, double-press, error |
|
||
| `TextInsertService` | Clipboard save→set→Ctrl+V→restore via nut-js |
|
||
| `SoundEffectService` | Preloaded WAV feedback (start/stop/error/cancel/chime) |
|
||
|
||
### STT engine layer (`services/stt/`)
|
||
| File | Purpose |
|
||
|---|---|
|
||
| `STTManager` | Dispatcher across local + 6 cloud providers, auto-fallback (events provider-changed, config-changed, fallback-to-local). `transcribePartial`/`warmUpLocal` route to the local engine only |
|
||
| `types.ts` | `ISTTDriver` contract |
|
||
| `audio-utils.ts` | `pcmToWav`, `createProbeWav` |
|
||
| `drivers/OpenAI|Groq|Deepgram|AssemblyAI|Google|Custom|D3ROCloud` | Provider drivers; `D3ROCloudDriver` uses Supabase access token |
|
||
|
||
### LLM layer
|
||
| Service | Purpose |
|
||
|---|---|
|
||
| `LocalLLMService` | Ollama REST (models, pull w/ progress, server start, NDJSON streaming) |
|
||
| `PremiumLLMService` | Claude via Supabase `llm-proxy`, local fallback |
|
||
| `OnlineLLMService` | JWT-authenticated .NET backend client |
|
||
| `llm-prompts.ts` | `resolveSystemPrompt` SSOT for action prompts |
|
||
|
||
### Memory & knowledge
|
||
| Service | Purpose |
|
||
|---|---|
|
||
| `HistoryService` | SQLite history CRUD/search/stats |
|
||
| `DictionaryService` | Custom vocabulary CRUD/search + cloud sync hooks + JSON/CSV import/export (`dictionary:import`/`export`, save/open dialogs) |
|
||
| `MemoService` | Memo tags over history (`memo_tags`) |
|
||
| `RAGService` | Local RAG: `nomic-embed-text` embeddings, cosine search over `rag_chunks` |
|
||
| `CustomInstructionService` | User LLM commands (5 built-ins) |
|
||
| `VoiceCommandService` | Keyword → command rule matching |
|
||
| `ChainService` | Multi-step LLM pipelines (LLMChain) |
|
||
| `ScreenContextService` | Active-window + selected-text context |
|
||
|
||
### Phase 10+ features
|
||
| Service | Purpose |
|
||
|---|---|
|
||
| `CaptionService` | Live captions from system/loopback audio; caption overlay (events segment, state-changed, session-saved, error) |
|
||
| `FileTranscriptionService` | Audio/video file → ffmpeg → 30s chunks → STT merge (events progress, complete, error, state-changed) |
|
||
| `MeetingSummaryService` | Post-caption LLM summary |
|
||
| `DictationTemplateService` | Field-by-field voice form filling |
|
||
| `VoiceConversationService` | STT→LLM→TTS loop, 10-turn memory |
|
||
| `TTSPlaybackService` | Platform TTS (macOS `say`, Windows SAPI), sentence queue |
|
||
| `VoiceActionService` | Voice → LLM JSON action plan → OS execution (dangerous blocked) |
|
||
|
||
### Phase 12–15
|
||
| Service | Purpose |
|
||
|---|---|
|
||
| `MeetingModeService` | Meeting recording: live transcript, timestamp memos, doc generation/export, diarization |
|
||
| `MeetingDocTemplateService` | Meeting-doc templates (built-ins + CRUD) |
|
||
|
||
### Account / infra / monetization
|
||
| Service | Purpose |
|
||
|---|---|
|
||
| `ConfigService` | electron-store `AppConfig` (`configGet/Set`, defaults) |
|
||
| `LicenseService` | Freemium tiers, quotas (`daily_usage`), activation, upgrade prompts |
|
||
| `CloudSyncService` | Supabase sync, per-user DB switching, history/dictionary/meeting mirror |
|
||
| `CloudSTTService` | Thin cloud STT wrapper over `D3ROCloudDriver` |
|
||
| `UpdateService` | electron-updater (canonical Forgejo feed, channels, mandatory/full-vs-delta policy, staged rollout, restart dialog) |
|
||
| `AutoLaunchService` | OS login-item auto-start |
|
||
| `LoggerService` | electron-log wrapper + category loggers |
|
||
| `checkout`/payment | `payment-handlers.ts` — authenticated Edge-only Stripe/Payple checkout + server readback |
|
||
|
||
### Ads (`services/ads/`)
|
||
| File | Purpose |
|
||
|---|---|
|
||
| `AdMediationEngine` | Multi-ad mediation + header bidding |
|
||
| `AdSettlementService` | Revenue settlement, withholding, payout ledger |
|
||
| `BaseAdAdapter` / `UnavailableAdAdapter` | Adapter contract + fail-closed base |
|
||
| `DirectHouseSponsorAdapter` | **Real configurable adapter**: bids/reports against an operator HTTPS `endpointUrl` (`AdNetworkConfig.endpointUrl`), validates creatives, fail-closed (`adapter_not_configured`) when unconfigured |
|
||
| 9 placeholder adapters (AppLovin, Carbon, EthicalAds, GoogleAdManager, InMobi, Mintegral, Playwire, PubMatic, Unity) | Extend `UnavailableAdAdapter` — registered, no live bids (`provider_not_integrated`) |
|
||
|
||
---
|
||
|
||
## 3. IPC layer
|
||
|
||
Registry: `src/main/ipc/index.ts` calls 29 `registerXHandlers()` in fixed order. Channel SSOT: `packages/core/src/ipc-channels.ts`.
|
||
|
||
| Handler | Channel group(s) |
|
||
|---|---|
|
||
| `ads-handlers` | ADS |
|
||
| `audio-handlers` | AUDIO |
|
||
| `caption-handlers` | CAPTION + SYSTEM_AUDIO |
|
||
| `chain-handlers` | CHAIN |
|
||
| `cloud-sync-handlers` | CLOUD_SYNC |
|
||
| `config-handlers` | CONFIG |
|
||
| `context-handlers` | CONTEXT |
|
||
| `dictionary-handlers` | DICTIONARY |
|
||
| `file-transcription-handlers` | FILE_TRANSCRIPTION |
|
||
| `history-handlers` | HISTORY + `stats:getSummary` |
|
||
| `hotkey-handlers` | HOTKEY |
|
||
| `instruction-handlers` | INSTRUCTION |
|
||
| `license-handlers` | LICENSE |
|
||
| `llm-handlers` | LLM + `llm:premium:*` + ONLINE_AUTH |
|
||
| `meeting-doc-template-handlers` | MEETING_DOC_TEMPLATE |
|
||
| `meeting-mode-handlers` | MEETING_MODE + MEETING_CHAT |
|
||
| `meeting-summary-handlers` | MEETING_SUMMARY |
|
||
| `memo-handlers` | MEMO |
|
||
| `payment-handlers` | PAYMENT |
|
||
| `rag-handlers` | RAG |
|
||
| `stt-handlers` | STT |
|
||
| `support-handlers` | SUPPORT |
|
||
| `system-handlers` | SYSTEM |
|
||
| `template-handlers` | DICTATION_TEMPLATE |
|
||
| `voice-action-handlers` | VOICE_ACTION |
|
||
| `voice-command-handlers` | VOICE_COMMAND |
|
||
| `voice-conversation-handlers` | VOICE_CONVERSATION |
|
||
| `voice-handlers` | VOICE |
|
||
| `window-handlers` | WINDOW + `SYSTEM.OPEN_EXTERNAL` |
|
||
|
||
Preload exposes **`window.electronAPI`** with 33 namespaces: `platform, audio, config, voice, stt, hotkey, llm (incl. premium), history, dictionary, stats, window, system, instruction, app, memo, voiceCommand, context, chain, caption, license, fileTranscription, meetingSummary, dictationTemplate, rag, voiceAction, voiceConversation, meetingMode, meetingChat, meetingDocTemplate, cloudSync, onlineAuth, ads, support, payment`. Envelope: `IPCResult<T>` (success/error); `app.onDataChanged` is the global refresh channel.
|
||
|
||
---
|
||
|
||
## 4. Windows & popups
|
||
|
||
`windows/WindowManager.ts` creates 6 windows: main (borderless, custom TitleBar; macOS `hiddenInset`), recording-tip, result-popup, history-popup, command-popup, caption-overlay. Injects popup theme CSS + i18n strings; 2-phase resize. `windows/TrayManager.ts` — tray icon + menu + double-click show.
|
||
|
||
Vanilla popups (`src/renderer/popups/`):
|
||
| Popup | Purpose |
|
||
|---|---|
|
||
| `recording-tip` | 9-bar waveform indicator, partial transcript |
|
||
| `result-popup` | Transcription result + copy, auto-close with hover pause |
|
||
| `history-popup` | Recent transcriptions; ↑↓/Enter/1-9/ESC |
|
||
| `command-popup` | Command selection (Ctrl+Shift+C) |
|
||
| `caption-overlay` | Live caption overlay (font/opacity/maxLines) |
|
||
|
||
---
|
||
|
||
## 5. Renderer IA
|
||
|
||
Routing is state-based in `AppLayout.tsx` (`Route` union + `NAV_ITEMS`), no react-router.
|
||
|
||
| Page | Route | Feature |
|
||
|---|---|---|
|
||
| `DashboardPage` | dashboard | Voice cockpit: hero, bento tiles, multi-engine hub (STT/LLM), telemetry, recent history, file drop |
|
||
| `HistoryPage` | history | History & memory timeline; search, tag filter, pagination, export/delete |
|
||
| `DictionaryPage` | dictionary | Custom vocabulary editor |
|
||
| `CommandsPage` | commands | Custom instructions + voice keyword rules + LLM chains + dictation templates |
|
||
| `VoiceConversationPage` | conversation | Duplex voice assistant (local pipeline vs OpenAI Realtime) |
|
||
| `KnowledgeBasePage` | knowledge | Local RAG: add/index docs, semantic query, reindex/remove |
|
||
| `MeetingModePage` | meeting | Meeting studio: live transcript, memos, doc generation/export, diarization |
|
||
|
||
Modals/components: `SettingsModal` (tabs General/Audio/STT/LLM/License/Cloud/About), `LicenseModal`, `LicenseTab`, `CloudSyncSection`, `OnboardingModal`, `UpgradePromptModal`, `ProBadge`, `TemplateSection`, `FileDropZone`, `HotkeyRecordModal`, `OllamaGuideModal`, `CodexOAuthGuideModal`, `TitleBar`, `StatusBar`, meeting components (9), voice-conversation, payment (`CheckoutModal`, `checkout-flow.ts`), support (`SupportModal`), ads (`AdBanner`, `RewardedQuotaModal`), shared cards.
|
||
|
||
Hooks: `useRealtimeConversation` (OpenAI Realtime WebRTC), `useLicenseState`, `useProFeature`.
|
||
|
||
DB schema (`src/main/db/schema.ts`, drizzle SQLite): `history`, `dictionary`, `stats`, `memo_tags`, `daily_usage`, `rag_documents`, `rag_chunks`, `meeting_sessions`, `meeting_memos`, `meeting_documents`.
|
||
|
||
---
|
||
|
||
## 6. Desktop status summary
|
||
|
||
- Core dictation/LLM/history pipeline: **implemented + tested** (~590 desktop tests; vitest + playwright).
|
||
- Cross-platform packaging: Windows NSIS (signed, `forceCodeSigning`), macOS DMG/ZIP arm64 (ad-hoc signing); auto-update via canonical Forgejo feed with update policy (`release/update-policy.json`).
|
||
- Local-first AI (SoX + faster-whisper sidecar + bundled Ollama) and cloud paths both present.
|
||
- **Local STT is packaged** (`1.3.0`): `electron-builder.yml` `extraResources` copies `sidecar-dist/sidecar` → `resources/sidecar` and `resources/ffmpeg` → `resources/ffmpeg`; `scripts/ci/verify-sidecar-bundle.mjs` gates packaging. Build locally with `npm --prefix apps/desktop run sidecar:setup && npm --prefix apps/desktop run sidecar:build`. The sidecar stays in console mode so `stdout`/`stderr` reach the app log (UTF-8, line-buffered); a packaged sidecar **must** exist or startup fails loudly instead of silently falling back to a system Python.
|
||
- All local engine URLs (`LocalSTTService`, `LocalLLMService`, `RAGService`, `OnlineLLMService`, `STTManager`) pass through `src/main/utils/loopback.ts`, which rewrites `localhost` to `127.0.0.1`, because some Windows hosts resolve `localhost` to IPv6 only and local engines bind IPv4.
|
||
- Meeting intelligence, RAG, voice conversation (local + Realtime), captions, file transcription: implemented.
|
||
- **Ad mediation**: `DirectHouseSponsorAdapter` performs real configurable REST bids; the other 9 adapters remain fail-closed stubs pending official SDKs (see `11-gap-backlog.md` GAP-ADS-01/02).
|
||
- Tier resolution now routes through `@d3ro/core/entitlement` (`resolveEntitlement`, `normalizeEntitlementTier`); `useLicenseState.isPro` includes `pro_plus`.
|
||
- No `TODO`/`FIXME` markers found in `src` (grep clean). `src/main/types/` is an empty directory.
|
||
|
||
---
|
||
|
||
## 7. Key file anchors
|
||
|
||
| Thing | Path |
|
||
|---|---|
|
||
| App entry / deep links | `src/main/index.ts` |
|
||
| Bootstrap order | `src/main/bootstrap.ts` |
|
||
| IPC registry | `src/main/ipc/index.ts` |
|
||
| IPC channel SSOT | `packages/core/src/ipc-channels.ts` |
|
||
| Preload API | `src/preload/index.ts` |
|
||
| Windows | `src/main/windows/WindowManager.ts` |
|
||
| Voice orchestrator | `src/main/services/VoiceModeService.ts` |
|
||
| DB schema | `src/main/db/schema.ts` |
|
||
| Renderer shell / routes | `src/renderer/components/AppLayout.tsx` |
|
||
| Update feed SSOT | `src/main/update-feed.ts` |
|
||
| Update policy SSOT | `release/update-policy.json` + `src/main/update-policy.ts` |
|
||
| Path/loopback resolution | `src/main/utils/paths.ts`, `src/main/utils/loopback.ts` |
|
||
| Sidecar source / packaging | `sidecar/main.py`, `scripts/setup-sidecar.mjs`, `scripts/build-sidecar.mjs`, `scripts/ci/verify-sidecar-bundle.mjs` |
|