feat(meeting): correct transcripts with vocabulary learned from typing
Meeting and caption transcription never used what the app had already learned: speech recognition ran with a fixed Korean prompt, the refine prompt had no glossary, refined lines stopped inside the caption service, and the saved transcript was a raw join of segments. Names and jargon the user types every day were therefore re-mangled on every recording. The learned corpus (user dictionary plus personal phrases) now feeds one glossary that biases recognition, fixes mistyped words deterministically right after each line is finalized, and is handed to the local model when it polishes a line. Refined lines also reach the meeting transcript and screen through a new segment-updated event. Corrections never touch words the user actually wrote, refuse ties between equally likely candidates, and keep particles intact.
This commit is contained in:
parent
d631042836
commit
9c2b6a9782
21 changed files with 1134 additions and 61 deletions
|
|
@ -14,7 +14,7 @@
|
|||
| Preload | `src/preload/` | `index.ts` exposes `window.electronAPI`; `popup.ts` exposes `window.popupAPI` |
|
||||
| Renderer | `src/renderer/` | React app: `AppLayout` + 7 pages + modals + 5 vanilla popups |
|
||||
|
||||
**Main entry** `src/main/index.ts`: sets app name/AppUserModelId, disables GPU acceleration, EPIPE/uncaught handlers, registers `d3ro-voice://` deep-link protocol (Supabase OAuth implicit + PKCE), single-instance lock, then `bootstrap()` + `setupLifecycle()`.
|
||||
**Main entry** `src/main/index.ts`: sets app name/AppUserModelId, disables GPU acceleration, EPIPE/uncaught handlers, registers `d3ro-voice://` deep-link protocol (Supabase OAuth implicit + PKCE), single-instance lock, then `bootstrap()` + `setupLifecycle()`. The explicit manual-QA argument `--preview-update-dialogs` is isolated before protocol registration/bootstrap: it bypasses the single-instance lock, invokes the real update-dialog prompter for four consent/restart variants, touches no feed/download/install/settings state, then quits. This path is unit-tested, but its packaged native dialogs still require an external-terminal visual run.
|
||||
|
||||
**Bootstrap** `src/main/bootstrap.ts`: ordered `BootstrapStep[]` — logger, config, **database (critical)**, license, create-windows (critical), tray, **ipc-handlers (critical)**, custom-instructions, voice-commands, sound-effects, auto-launch, popup-preload, key-bindings, voice-mode, stt-warmup, llm-polling, meeting-summary-wiring, meeting-mode, cloud-sync, auto-update. Wires VoiceMode events to sound + history persistence, and subscribes to `KeyBindingService` `triggered` for the `history-popup` / `command-popup` actions (`bootstrap.ts:159`) — those two were hardcoded accelerators before and are now rebindable like everything else.
|
||||
|
||||
|
|
@ -66,7 +66,8 @@ Singleton + `EventEmitter` pattern (`getXService()` accessors).
|
|||
### Phase 10+ features
|
||||
| Service | Purpose |
|
||||
|---|---|
|
||||
| `CaptionService` | Live captions from system/loopback audio; caption overlay (events segment, state-changed, session-saved, error) |
|
||||
| `CaptionService` | Live captions from system/loopback audio; caption overlay (events segment, segment-updated, state-changed, session-saved, error). Since 2026-11-05 every recognition call carries the learned-vocabulary STT hint, each final line passes the deterministic glossary correction, and the LLM refine prompt receives the user-term/style block |
|
||||
| `TranscriptGlossaryService` | Learned-vocabulary gateway for transcription (`@d3ro/core/transcript-correction`): merges user dictionary words + pronunciations and personal-graph phrases into a 15 s cached glossary, and exposes the STT `initial_prompt` hint, deterministic word correction (dictionary variants + 1-jamo learned-term match that never touches words the user actually wrote), and the LLM 교열 reference block. Sources are injectable; DB failures fall back to an empty glossary so transcription never blocks |
|
||||
| `FileTranscriptionService` | Audio/video file → ffmpeg → 30s chunks → STT merge (events progress, complete, error, state-changed). Shared media-format SSOT is `file-transcription/media-formats.ts`; meeting import can skip duplicate history persistence and retain the normalized WAV |
|
||||
| `MeetingSummaryService` | Post-caption LLM summary |
|
||||
| `DictationTemplateService` | Field-by-field voice form filling |
|
||||
|
|
@ -77,7 +78,7 @@ Singleton + `EventEmitter` pattern (`getXService()` accessors).
|
|||
### Phase 12–15
|
||||
| Service | Purpose |
|
||||
|---|---|
|
||||
| `MeetingModeService` | Meeting recording and existing-media import: live/file transcript, normalized meeting audio, timestamp memos, doc generation/export, diarization. `meeting/imported-recording.ts` maps file-transcription segments to the canonical meeting transcript. Account revision checks before/after WAV copy and after transcription reject even A→B→A changes; unsuccessful imports remove only their own generated audio |
|
||||
| `MeetingModeService` | Meeting recording and existing-media import: live/file transcript, normalized meeting audio, timestamp memos, doc generation/export, diarization. `meeting/imported-recording.ts` maps file-transcription segments to the canonical meeting transcript. Account revision checks before/after WAV copy and after transcription reject even A→B→A changes; unsuccessful imports remove only their own generated audio. Since 2026-11-05 refined caption lines reach the meeting transcript (`meetingMode:segmentUpdated`) instead of the raw text, and auto-polish receives the learned-vocabulary block |
|
||||
| `MeetingDocTemplateService` | Meeting-doc templates (built-ins + CRUD) |
|
||||
|
||||
### Input intelligence (2026-09-21; Local Flow Intelligence extension 2026-09-22)
|
||||
|
|
@ -103,7 +104,7 @@ The existing `input-telemetry-handlers` and `suggestion-handlers` IPC extensions
|
|||
| `CloudSyncService` | Supabase auth + lifecycle: per-user DB switching, local-mode import on first sign-in, Realtime (`ws` transport) + 5-min heartbeat, device check-in, debounced flush/pull triggers. The sync itself lives in `services/sync/` |
|
||||
| `services/sync/` | `SyncEngine` (backfill once per user DB → push outbox → pull by server-clock keyset cursor → tombstones → memo-tag reconcile), `sync-outbox` (`sync_outbox`/`sync_state` tables), `sync-adapters` (history, dictionary, meetings, meeting memos/documents, custom commands, templates), `memo-tag-sync`, `supabase-sync-remote`, `device-registration`, `realtime-transport`, `settings-sync` (mobile `user_settings` ↔ language/theme/defaultLLMAction/activeInstructionId), `audio-sync` (recording upload to the `audio` bucket, remote cleanup on delete), knowledge adapter (source-text chunks), `transcript-sync` (meeting segments from the desktop transcript), `builtin-instruction-sync` (preset prompt edits ↔ server preset rows). Services record changes with `getCloudSyncService().pushOne/pushDelete(entity, id)`; pulled rows are applied without re-queuing |
|
||||
| `CloudSTTService` | Thin cloud STT wrapper over `D3ROCloudDriver` |
|
||||
| `UpdateService` | electron-updater (canonical Forgejo feed, channels, mandatory/full-vs-delta policy, staged rollout, restart dialog) |
|
||||
| `UpdateService` | electron-updater (canonical Forgejo feed, channels, mandatory/full-vs-delta policy, staged rollout). Native consent/restart dialogs resolve `update.dialog.*` through a fresh `getPopupTranslator()` at each prompt, so a runtime locale change is reflected without rebuilding the prompter; unit tests cover required/optional paths and a ko→en switch on the same prompter. `update-dialog-preview.ts` exposes only the explicit `--preview-update-dialogs` manual-QA entry described above. The current-source package was built and its native ABI verified; packaged dialog appearance/interaction remains unverified until that executable is run on the interactive desktop. |
|
||||
| `AutoLaunchService` | OS login-item auto-start |
|
||||
| `LoggerService` | electron-log wrapper + category loggers |
|
||||
| upgrade / billing | No in-app checkout. `license-handlers.ts` `LICENSE.OPEN_BILLING` opens `billingUrl({ tier })` (web Payple); tier returns via `license:tierChanged`. Stripe `payment-handlers.ts`·`CheckoutModal`·`payment:*` IPC removed 2026-09-26 |
|
||||
|
|
@ -209,6 +210,8 @@ Key-binding UI lives in `components/keybinding/` (`Keycap`, `KeyBindingPicker`,
|
|||
|
||||
Hooks: `useRealtimeConversation` (OpenAI Realtime WebRTC), `useLicenseState`, `useProFeature`, `useKeyBindingMap` (subscribes to `keybinding:changed`; the dashboard renders the live `dictation` binding through `BindingKeycaps`), `useInputInsights` (telemetry + suggestion state + weekly summary + phrases; the Dashboard shows a weekly input card when collection is on).
|
||||
|
||||
Test-facing surface (stable, Playwright E2E depends on these): `AppSidebar` `nav-*` buttons expose `data-testid="nav-${route}"` + `aria-current="page"`; `SettingsModal` tabs expose `data-testid="settings-tab-${id}"`; onboarding mode cards expose `onboarding-mode-local|online`; `EmptyStateCard` exposes `empty-state-card`. Add a testid before writing an E2E journey; do not select by MUI classes.
|
||||
|
||||
DB schema (`src/main/db/schema.ts`, drizzle SQLite): `history`, `dictionary`, `stats`, `memo_tags`, `daily_usage`, `rag_documents`, `rag_chunks`, `meeting_sessions`, `meeting_memos`, `meeting_documents`, `input_activity` (hour × app counters), `typing_samples`, `personal_phrases`, `suggestions`.
|
||||
|
||||
---
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue