대시보드 폴드아웃/드릴다운 정리 + 페르소나 역린·misconduct 반응 + 게이트웨이 격리·RAG 비차단 수정
SSOT 대시보드:
- 한신대 기술분석 PDF(19쪽) 정합성 분석 + 이번 세션 발견 섹션 추가
- 섹션 폴드아웃(접기)·상단 목차(드릴다운)·모두 펼치기/접기 — 내용 보존, 레이아웃만 정리
페르소나 반응 강화('저항·반응 조절' 핵심 차별):
- PersonaCard.triggers(역린) 필드 + CCD 핵심상처 파생 역린 블록
- L0에 무례·모욕·조롱 시 현실적 동맹 균열 반응 지침
버그·성능 수정(라이브/E2E로 포착):
- 게이트웨이 페르소나 격리: --append-system-prompt를 --system-prompt(교체)로 + --exclude-dynamic-system-prompt-sections (내담자 캐릭터 붕괴·개발맥락 누출 차단)
- RAG: 임베더 동기 로드(약 7-13초)를 _warm_rag_caches 백그라운드 warm으로(세션 생성 블로킹 회귀 수정)
- voice TTS RMS 데드힌트 제거, init_state OpennessParams 파라미터객체화
- 한국어 PII(날짜·금액·주소) 마스킹 보강
- 레이아웃 시각 게이트: 폼 컨트롤 값 스크롤 오탐 제외(7/7)
검증: 백엔드 84/84, E2E 42(데스크톱 27·모바일 11·아바타 4), 시각 게이트 7/7
This commit is contained in:
Yun Chan
parent
cb2aebd76c
commit
085460b5e0
327 changed files with 31226 additions and 1829 deletions
165
docs/phase3/kpi-measurement-plan.md
Normal file
165
docs/phase3/kpi-measurement-plan.md
Normal file
|
|
@ -0,0 +1,165 @@
|
|||
# Phase 3 KPI Measurement Plan
|
||||
|
||||
## Purpose
|
||||
|
||||
Define the Phase 3 KPI report so the dashboard gates are measured the same way each
|
||||
time. These metrics are pilot evidence, not claims of validated clinical effectiveness.
|
||||
|
||||
Primary evidence file: `02-measures/kpi_report.json`
|
||||
|
||||
## Report Shape
|
||||
|
||||
```json
|
||||
{
|
||||
"pilot_id": "phase3-pilot-YYYYMMDD",
|
||||
"generated_at": "2026-06-26T00:00:00Z",
|
||||
"source_window": {
|
||||
"started_at": "2026-00-00T00:00:00Z",
|
||||
"ended_at": "2026-00-00T00:00:00Z"
|
||||
},
|
||||
"cohort_size": 20,
|
||||
"metrics": {
|
||||
"pilot_completion": {},
|
||||
"session_completion": {},
|
||||
"embedding_consistency": {},
|
||||
"hallucination_rate": {},
|
||||
"top1": {},
|
||||
"sus": {},
|
||||
"inter_rater_kappa": {},
|
||||
"icc": {},
|
||||
"self_efficacy_prepost": {}
|
||||
},
|
||||
"exclusions": [],
|
||||
"open_schema_gaps": [],
|
||||
"review": {
|
||||
"operator": "",
|
||||
"reviewed_at": "",
|
||||
"decision": "pending"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Each metric object must include:
|
||||
|
||||
```json
|
||||
{
|
||||
"value": 0,
|
||||
"threshold": 0,
|
||||
"pass": false,
|
||||
"numerator": null,
|
||||
"denominator": null,
|
||||
"method": "",
|
||||
"source_files": []
|
||||
}
|
||||
```
|
||||
|
||||
## KPI Definitions
|
||||
|
||||
| KPI | Gate | Definition | Evidence |
|
||||
| --- | --- | --- | --- |
|
||||
| Pilot completion | >=20 participants | Count active participants with consent, required sessions, pre/post, SUS, and supervisor review. | `pilot_roster.csv`, `session_completion.csv`, `prepost_measures.csv`, `sus_responses.csv` |
|
||||
| Session completion | >=20 participants | Count participants with at least two `completed` sessions and non-empty `supervisor_reviewed_at`. | `session_completion.csv` |
|
||||
| Embedding consistency | >=0.79 | Mean cosine similarity for persona invariant checks across masked session turns. Report model, embedding version, mean, p10, and count. | DB query export, `kpi_report.json` |
|
||||
| Hallucination rate | <=0.05, target <=0.01 | Unsupported or contradicted factual claims divided by reviewed eligible evaluator/client-response claims. | supervisor review export, `kpi_report.json` |
|
||||
| Top-1 retrieval | >=0.80 | Top-1 evidence match rate for golden feedback or RAG evaluation queries. | golden evaluation run, `kpi_report.json` |
|
||||
| SUS | >=80 | Standard 10-item SUS score transformed to a 0-100 score; report mean, median, and response count. | `sus_responses.csv` |
|
||||
| Cohen kappa | >=0.60 | Agreement between human reviewers or human-vs-AI labels for categorical tags used in dataset export. | annotation export, `kpi_report.json` |
|
||||
| ICC | >=0.75 | Intraclass correlation for numeric ratings such as feedback score dimensions. | annotation export, `kpi_report.json` |
|
||||
| Self-efficacy pre/post | Report and review | Paired pre/post change for the approved self-efficacy measure. Report effect direction, test used, p-value, and missingness. | `prepost_measures.csv` |
|
||||
|
||||
## Metric Methods
|
||||
|
||||
### Pilot Completion
|
||||
|
||||
Computation:
|
||||
|
||||
```text
|
||||
participant passes =
|
||||
active consent receipt exists
|
||||
AND >=2 completed sessions
|
||||
AND every counted session has supervisor_reviewed_at
|
||||
AND pre measure exists
|
||||
AND post measure exists
|
||||
AND SUS response set exists or missingness is explicitly reviewed
|
||||
```
|
||||
|
||||
The dashboard gate passes when at least 20 participants pass. Withdrawn participants are
|
||||
reported in exclusions and must not be silently removed from denominator notes.
|
||||
|
||||
### Embedding Consistency
|
||||
|
||||
Operator command placeholder:
|
||||
|
||||
```powershell
|
||||
# Use the deployed reporting job when available. The output is copied into kpi_report.json.
|
||||
python scripts\check-phase3-artifacts.py --check --evidence-root $root
|
||||
```
|
||||
|
||||
Minimum report fields:
|
||||
|
||||
```json
|
||||
{
|
||||
"value": 0.79,
|
||||
"threshold": 0.79,
|
||||
"pass": true,
|
||||
"method": "masked persona invariant cosine similarity",
|
||||
"model": "embedding-model-name-and-version",
|
||||
"p10": 0.0,
|
||||
"count": 0,
|
||||
"source_files": ["02-measures/persona_embedding_eval.csv"]
|
||||
}
|
||||
```
|
||||
|
||||
### Hallucination Rate
|
||||
|
||||
Count only reviewed claims that should be grounded in the persona card, session state,
|
||||
approved knowledge base, or rubric. Unsupported clinical, safety, or personal-history
|
||||
claims are high severity and must be listed in `review.exceptions`.
|
||||
|
||||
Formula:
|
||||
|
||||
```text
|
||||
hallucination_rate = unsupported_or_contradicted_claims / reviewed_eligible_claims
|
||||
```
|
||||
|
||||
### SUS
|
||||
|
||||
Report the standard SUS transformed 0-100 score. Do not copy item text into the
|
||||
repository. Store only item number, numeric response, participant id, and timestamp.
|
||||
|
||||
Minimum report fields:
|
||||
|
||||
```json
|
||||
{
|
||||
"value": 80.0,
|
||||
"threshold": 80.0,
|
||||
"pass": true,
|
||||
"mean": 80.0,
|
||||
"median": 80.0,
|
||||
"respondents": 20,
|
||||
"source_files": ["02-measures/sus_responses.csv"]
|
||||
}
|
||||
```
|
||||
|
||||
### Inter-Rater Agreement
|
||||
|
||||
Use the same item universe for kappa and ICC that is later referenced by the dataset
|
||||
export manifest. If agreement fails, the dataset can still be exported as a technical
|
||||
artifact only when the manifest marks it `not_gold`.
|
||||
|
||||
## Operator Checklist
|
||||
|
||||
- [ ] Evidence root exists outside the source tree.
|
||||
- [ ] Roster and consent receipt counts match.
|
||||
- [ ] Session completion rows are exported after supervisor review.
|
||||
- [ ] Pre/post and SUS files are present.
|
||||
- [ ] `kpi_report.json` contains all required metric keys.
|
||||
- [ ] Every failed metric has an owner, disposition, and decision date.
|
||||
- [ ] Export manifest references the same cohort and exclusions as the KPI report.
|
||||
|
||||
Validation command:
|
||||
|
||||
```powershell
|
||||
python scripts\check-phase3-artifacts.py --check --evidence-root $root
|
||||
```
|
||||
|
||||
Loading…
Add table
Add a link
Reference in a new issue