refactor(meeting): share the transcript polish prompt with tests
Move the polish instructions into their own module so the prompt the app sends, the unit tests and the local-model probe use the same text. Tests now pin the two rules that matter: fix a misheard name or in-house term to the learned spelling, and never swap an ordinary word on similarity alone. Verified against the local model with the real learned corpus: 하람록에 was rewritten to HaramLog에 and the dictionary pronunciation 디에로 to D3RO, while a natural sentence with no learned term came back with no word substitutions. Recorded the trap that this model returns empty content unless thinking is disabled.
This commit is contained in:
parent
63e741b232
commit
648c7e4f4d
4 changed files with 65 additions and 22 deletions
|
|
@ -58,6 +58,7 @@ import {
|
|||
import { markdownToDocx } from '@d3ro/core/utils/markdown-to-docx'
|
||||
import { getFileTranscriptionService } from './FileTranscriptionService'
|
||||
import { getTranscriptGlossaryService } from './TranscriptGlossaryService'
|
||||
import { buildPolishTranscriptPrompt } from './meeting/polish-prompt'
|
||||
import { buildImportedMeetingTranscript } from './meeting/imported-recording'
|
||||
import type {
|
||||
MeetingModeState,
|
||||
|
|
@ -95,26 +96,6 @@ const TRANSCRIPTION_FAILED_MESSAGE = 'Transcription failed during recording'
|
|||
/** 자막 인식 실패를 렌더러에 다시 알리는 최소 간격 */
|
||||
const CAPTION_FAILURE_NOTICE_INTERVAL_MS = 30_000
|
||||
|
||||
/**
|
||||
* 전사 AI 교열(다듬기) 지시문.
|
||||
*
|
||||
* 사용자가 학습시킨 용어로 「잘못 들은 단어」를 반드시 고치게 하되, 근거 없는 일반 단어 교체는 금지한다.
|
||||
* 한글 어절을 1자모 거리로 기계적으로 바꾸면 정상 문장이 망가지는 것을 실측했으므로(MTG-14),
|
||||
* 단어 교체 판단은 문맥을 보는 모델에게 맡기고 근거(용어·표현 목록)를 함께 준다.
|
||||
*/
|
||||
const POLISH_TRANSCRIPT_PROMPT = [
|
||||
'다음 음성 전사 텍스트를 회의록으로 다듬습니다.',
|
||||
'',
|
||||
'## 반드시 지킬 것',
|
||||
'- 필러(음, 어, 그, 아, 뭐 등)·말 바꾸기(A 아니 A가 아니라 B → B)·불필요한 어절·의미 중복을 제거하세요.',
|
||||
'- 아래 [사용자 용어]·[사용자가 자주 쓰는 용어]와 같은 이름·제품명·사내 용어가 전사에서 비슷하게 다르게 적혀 있고 문맥상 그것이 확실하면, **반드시 그 표기로 고치세요.**',
|
||||
'- 이미 맞게 적힌 용어는 그대로 두세요.',
|
||||
'- 반대로, 근거가 없는 일반 명사·부사·조사·어미는 절대 다른 단어로 바꾸지 마세요. 비슷하게 들린다는 이유만으로 단어를 바꾸면 문장이 망가집니다.',
|
||||
'- 확실하지 않으면 원문을 그대로 두세요.',
|
||||
'- 화자의 말투(반말/존댓말)와 의미·내용은 바꾸지 마세요. 타임스탬프 [MM:SS]와 화자 표기([화자 N] 등)는 그대로 유지하세요.',
|
||||
'- [사용자가 실제로 쓰는 표현]은 말투·용어 참고용입니다. 그 문장을 그대로 옮겨 쓰지 마세요.',
|
||||
].join('\n')
|
||||
|
||||
/**
|
||||
* 회의 기능이 쓰는 LLM 포트 — 백엔드 선택(local/online, 폴백)은 LlmGateway 한 곳이 정한다.
|
||||
* 문서 생성·요약 입력은 이 한도(프록시 메시지 한도) 안으로 맞춘다.
|
||||
|
|
@ -1462,7 +1443,7 @@ class MeetingModeService extends EventEmitter {
|
|||
const polished = await this._rewriteTranscriptInChunks(
|
||||
llm,
|
||||
preCorrected.text,
|
||||
POLISH_TRANSCRIPT_PROMPT + (glossary ? `\n\n${glossary}` : ''),
|
||||
buildPolishTranscriptPrompt({ glossary }),
|
||||
ErrorCode.MeetingPolishFailed,
|
||||
)
|
||||
|
||||
|
|
|
|||
26
apps/desktop/src/main/services/meeting/polish-prompt.ts
Normal file
26
apps/desktop/src/main/services/meeting/polish-prompt.ts
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
// src/main/services/meeting/polish-prompt.ts
|
||||
// 전사 AI 교열(다듬기) 지시문의 정본.
|
||||
//
|
||||
// 사용자가 학습시킨 용어로 「잘못 들은 단어」를 반드시 고치게 하되, 근거 없는 일반 단어 교체는 금지한다.
|
||||
// 한글 어절을 1자모 거리로 기계적으로 바꾸면 정상 문장이 망가지는 것을 실측했으므로(MTG-14),
|
||||
// 단어 교체 판단은 문맥을 보는 모델에게 맡기고 근거(용어·표현 목록)를 함께 준다.
|
||||
|
||||
/** 교열 규칙 본문 — 글로서리 블록과 분리해 테스트·프로브가 같은 문장을 쓴다. */
|
||||
export const POLISH_TRANSCRIPT_RULES = [
|
||||
'다음 음성 전사 텍스트를 회의록으로 다듬습니다.',
|
||||
'',
|
||||
'## 반드시 지킬 것',
|
||||
'- 필러(음, 어, 그, 아, 뭐 등)·말 바꾸기(A 아니 A가 아니라 B → B)·불필요한 어절·의미 중복을 제거하세요.',
|
||||
'- 아래 [사용자 용어]·[사용자가 자주 쓰는 용어]와 같은 이름·제품명·사내 용어가 전사에서 비슷하게 다르게 적혀 있고 문맥상 그것이 확실하면, **반드시 그 표기로 고치세요.**',
|
||||
'- 이미 맞게 적힌 용어는 그대로 두세요.',
|
||||
'- 반대로, 근거가 없는 일반 명사·부사·조사·어미는 절대 다른 단어로 바꾸지 마세요. 비슷하게 들린다는 이유만으로 단어를 바꾸면 문장이 망가집니다.',
|
||||
'- 확실하지 않으면 원문을 그대로 두세요.',
|
||||
'- 화자의 말투(반말/존댓말)와 의미·내용은 바꾸지 마세요. 타임스탬프 [MM:SS]와 화자 표기([화자 N] 등)는 그대로 유지하세요.',
|
||||
'- [사용자가 실제로 쓰는 표현]은 말투·용어 참고용입니다. 그 문장을 그대로 옮겨 쓰지 마세요.',
|
||||
].join('\n')
|
||||
|
||||
/** 글로서리 블록(`buildGlossaryPromptSection`)을 규칙 뒤에 붙인 최종 시스템 프롬프트. */
|
||||
export function buildPolishTranscriptPrompt(input: { glossary?: string } = {}): string {
|
||||
const glossary = input.glossary?.trim()
|
||||
return glossary ? `${POLISH_TRANSCRIPT_RULES}\n\n${glossary}` : POLISH_TRANSCRIPT_RULES
|
||||
}
|
||||
36
apps/desktop/tests/main/services/polish-prompt.test.ts
Normal file
36
apps/desktop/tests/main/services/polish-prompt.test.ts
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
// tests/main/services/polish-prompt.test.ts
|
||||
// 전사 AI 교열 프롬프트가 학습 용어를 근거로 쓰도록 지시하는지 (MTG-14).
|
||||
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import {
|
||||
buildPolishTranscriptPrompt,
|
||||
POLISH_TRANSCRIPT_RULES,
|
||||
} from '../../../src/main/services/meeting/polish-prompt'
|
||||
|
||||
describe('전사 교열 프롬프트', () => {
|
||||
it('학습 용어 오인식은 반드시 고치고, 근거 없는 단어 교체는 금지한다', () => {
|
||||
expect(POLISH_TRANSCRIPT_RULES).toContain('반드시 그 표기로 고치세요')
|
||||
expect(POLISH_TRANSCRIPT_RULES).toContain('일반 명사·부사·조사·어미는 절대 다른 단어로 바꾸지 마세요')
|
||||
expect(POLISH_TRANSCRIPT_RULES).toContain('[사용자 용어]')
|
||||
expect(POLISH_TRANSCRIPT_RULES).toContain('[사용자가 자주 쓰는 용어]')
|
||||
})
|
||||
|
||||
it('타임스탬프·화자 표기와 말투를 유지하라고 지시한다', () => {
|
||||
expect(POLISH_TRANSCRIPT_RULES).toContain('[MM:SS]')
|
||||
expect(POLISH_TRANSCRIPT_RULES).toContain('화자 표기')
|
||||
expect(POLISH_TRANSCRIPT_RULES).toContain('말투')
|
||||
})
|
||||
|
||||
it('글로서리 블록은 규칙 뒤에 붙는다', () => {
|
||||
const prompt = buildPolishTranscriptPrompt({ glossary: '[사용자 용어 — 이 표기를 그대로 쓰세요]\n- D3RO' })
|
||||
|
||||
expect(prompt.startsWith(POLISH_TRANSCRIPT_RULES)).toBe(true)
|
||||
expect(prompt).toContain('D3RO')
|
||||
expect(prompt.endsWith('- D3RO')).toBe(true)
|
||||
})
|
||||
|
||||
it('글로서리가 없으면 규칙만 돌려준다', () => {
|
||||
expect(buildPolishTranscriptPrompt()).toBe(POLISH_TRANSCRIPT_RULES)
|
||||
expect(buildPolishTranscriptPrompt({ glossary: ' ' })).toBe(POLISH_TRANSCRIPT_RULES)
|
||||
})
|
||||
})
|
||||
|
|
@ -89,7 +89,7 @@ Status quick-reference: `[x]` done+verified · `[~]` partial/unverified · `[ ]`
|
|||
| MTG-11 | Meeting AI chat over transcript | [x] | [ ] | [ ] | [x] | Desktop `MeetingChatPanel` + `MEETING_CHAT` |
|
||||
| MTG-12 | Content reporting for generated docs | [-] | [ ] | [x] | [x] | `content-report` Edge + generation receipts |
|
||||
| MTG-13 | Existing audio/video import into a meeting | [x] | [ ] | [~] | [x] | **2026-11-01:** Desktop `meetingMode:importRecording` reuses `FileTranscriptionService` without duplicate history, saves canonical timestamped transcript + normalized WAV, then opens existing minutes generation. Mobile `MeetingsScreen` creates the workspace then routes `meetingId` + `importRequest` to the existing durable Record meeting pipeline. **2026-11-02:** installed Windows 1.16.0 passed the actual Electron picker→timestamped transcript→local summary/action-item journey, so desktop is `[x]`. Fold6 `.3` import entry/editor rendered, but authenticated device file-to-minutes E2E remains (`GAP-MTG-03`). [Evidence](../evidence/2026-11-02-verification.md). |
|
||||
| MTG-14 | Learning-informed transcript correction (user dictionary + typed vocabulary) | [~] | [ ] | [ ] | [ ] | **2026-11-05 (root cause + rebuild of the transcript-correction seam):** the learned corpus was only wired to input suggestions. Meeting/live captions ran STT with a hardcoded `다음은 한국어 대화입니다.` prompt (no dictionary, no learned terms), the refine prompt had no glossary, the refine result updated only `CaptionService`'s own copy (meetings and the renderer kept the raw line), and the persisted `rawTranscript` was a plain join of raw segments. Now: core SSOT `@d3ro/core/transcript-correction` (glossary build, dictionary-variant replacement, learned-identifier correction with a unique-best rule, LLM reference block) + desktop `TranscriptGlossaryService` (dictionary + `personal_phrases`, 15 s cache invalidated on dictionary writes, fail-closed) are wired into `CaptionService` (hint on every recognition, deterministic correction per final line, glossary-aware refine, `segment-updated` event), `MeetingRecordingSession`/`MeetingModeService` (refined lines reach the meeting transcript and renderer via `meetingMode:segmentUpdated`; auto-polish gets the glossary block), `FileTranscriptionService` (per-chunk hint + correction) and `VoiceModeService` (learned-term pass). **2026-11-05 (v1.16.3 correction policy):** the first cut also rewrote ordinary Hangul words by 1-jamo distance. Measured against the real local corpus (users/_local/d3ro.db, 3 stored meeting transcripts) it changed **286 of 3,005 words** (하면 to 라면, 이제 to 이게, 매일 to 메일, 정보 to 정도). Deterministic correction is now limited to dictionary pronunciations and typed Latin identifiers (canonical casing preserved); Korean terms reach recognition through the STT hint and the polish prompt only. Re-measured on the same corpus: **0 of 3,005**. Real-engine check (sidecar large-v3-turbo, Korean TTS fixture containing the user term): baseline 하람로그에 to hinted transcript haramlog에, and a dictionary pronunciation entry still maps 디에로 to D3RO. Hangul-layout typing artifacts (dkssudgktpdy) are dropped from the hint. **2026-11-06 (v1.16.4 AI correction grounded in learning):** the polish pass no longer runs as a generic rewrite. Deterministic dictionary/identifier correction runs before and after the model, and the model receives a learned term list plus the user expressions with explicit rules (fix a misheard name/product/in-house term to the learned spelling; never swap ordinary nouns or endings on similarity alone). Re-measured on the real corpus: 0 of 3,005 deterministic changes, and the generated term block now shows canonical identifiers (HaramLog, Android, AI) with Hangul-layout artifacts removed from terms, hints and style examples. Dictation output is re-checked against the learned vocabulary after the LLM finishes. Tests: core transcript-correction.test.ts (17), desktop transcript-glossary.test.ts (10), red meeting-llm usecase (+1); live meeting with real jargon still open (GAP-MTG-04). |
|
||||
| MTG-14 | Learning-informed transcript correction (user dictionary + typed vocabulary) | [~] | [ ] | [ ] | [ ] | **2026-11-05 (root cause + rebuild of the transcript-correction seam):** the learned corpus was only wired to input suggestions. Meeting/live captions ran STT with a hardcoded `다음은 한국어 대화입니다.` prompt (no dictionary, no learned terms), the refine prompt had no glossary, the refine result updated only `CaptionService`'s own copy (meetings and the renderer kept the raw line), and the persisted `rawTranscript` was a plain join of raw segments. Now: core SSOT `@d3ro/core/transcript-correction` (glossary build, dictionary-variant replacement, learned-identifier correction with a unique-best rule, LLM reference block) + desktop `TranscriptGlossaryService` (dictionary + `personal_phrases`, 15 s cache invalidated on dictionary writes, fail-closed) are wired into `CaptionService` (hint on every recognition, deterministic correction per final line, glossary-aware refine, `segment-updated` event), `MeetingRecordingSession`/`MeetingModeService` (refined lines reach the meeting transcript and renderer via `meetingMode:segmentUpdated`; auto-polish gets the glossary block), `FileTranscriptionService` (per-chunk hint + correction) and `VoiceModeService` (learned-term pass). **2026-11-05 (v1.16.3 correction policy):** the first cut also rewrote ordinary Hangul words by 1-jamo distance. Measured against the real local corpus (users/_local/d3ro.db, 3 stored meeting transcripts) it changed **286 of 3,005 words** (하면 to 라면, 이제 to 이게, 매일 to 메일, 정보 to 정도). Deterministic correction is now limited to dictionary pronunciations and typed Latin identifiers (canonical casing preserved); Korean terms reach recognition through the STT hint and the polish prompt only. Re-measured on the same corpus: **0 of 3,005**. Real-engine check (sidecar large-v3-turbo, Korean TTS fixture containing the user term): baseline 하람로그에 to hinted transcript haramlog에, and a dictionary pronunciation entry still maps 디에로 to D3RO. Hangul-layout typing artifacts (dkssudgktpdy) are dropped from the hint. **2026-11-06 (v1.16.4 AI correction grounded in learning):** the polish pass no longer runs as a generic rewrite. Deterministic dictionary/identifier correction runs before and after the model, and the model receives a learned term list plus the user expressions with explicit rules (fix a misheard name/product/in-house term to the learned spelling; never swap ordinary nouns or endings on similarity alone). Re-measured on the real corpus: 0 of 3,005 deterministic changes, and the generated term block now shows canonical identifiers (HaramLog, Android, AI) with Hangul-layout artifacts removed from terms, hints and style examples. Dictation output is re-checked against the learned vocabulary after the LLM finishes. Tests: core transcript-correction.test.ts (17), desktop transcript-glossary.test.ts (10), red meeting-llm usecase (+1), polish-prompt.test.ts (4). **Real-model probe (2026-11-06, local gemma4:e4b with think:false):** with the production prompt and the real learned corpus, the misheard 하람록에 was rewritten to HaramLog에 and the dictionary pronunciation 디에로 to D3RO (2 word changes, no collateral edits); a natural Korean sentence without a learned term came back with 0 word substitutions (only a spacing merge); filler and self-correction were removed. Trap: without think:false (or /no_think) this model spends the whole output budget on its thinking block and returns empty content, which the chunk guard rejects. Live meeting with real jargon still open (GAP-MTG-04). |
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue