4e6b0045e3
- 관리자(role=admin)가 학습자·교수자·관리자 워크스페이스를 모두 접근하도록 can_access_role/require_role와 프론트 auth 헬퍼·Sidebar 내비를 정리하고 admin 워크스페이스 내비 E2E를 추가. - 소유자 결정 7건 전건 확정(2026-06-30)을 SSOT 대시보드·백로그에 반영하고 결정 필요 7→0으로 동기화. SSOT drift 게이트 기대 카운트도 갱신. - 확정된 H1 평가설계(κ≥0.70·ICC≥0.75·환각률≤0.03·t-검정 α=0.05·무작위 배정)를 approved-export κ 게이트(checker·dataset_export·recursive export)와 KPI 측정계획·export manifest 문서에 반영. 검증: npm run typecheck, npm run check:api-types, 백엔드 pytest 290 passed, admin 내비 E2E 1 passed, 레이아웃 시각게이트 9/9, session-layout 4 passed, SSOT drift 게이트 PASS.
187 lines
6.6 KiB
Markdown
187 lines
6.6 KiB
Markdown
# Phase 3 KPI Measurement Plan
|
||
|
||
## Purpose
|
||
|
||
Define the Phase 3 KPI report so the dashboard gates are measured the same way each
|
||
time. These metrics are pilot evidence, not claims of validated clinical effectiveness.
|
||
|
||
Primary evidence file: `02-measures/kpi_report.json`
|
||
|
||
## Evaluation Design (owner-confirmed 2026-06-30)
|
||
|
||
- **표본·배정**: 20명을 실험군/통제군으로 **무작위 배정(randomized)**. 단회기 50분, 3척도(self-efficacy / skill proficiency / training satisfaction) pre-post.
|
||
- **통계검정**: 군 간 비교는 **독립표본 t-검정**, 유의수준 **α=0.05**. 정규성 위반 시 Mann-Whitney U로 대체 가능하나 기본은 t-검정.
|
||
- **신뢰도 목표**: 평가자 간 **κ≥0.70**(Cohen's kappa), 수치 평정 **ICC≥0.75**.
|
||
- **환각률 목표**: 상담 응답 환각률 **≤0.03**.
|
||
- 위 파라미터는 설계 확정값이며, 실제 효과성 판정·통계해석은 20명 파일럿 evidence 수집 후 수행한다(파일럿 게이트).
|
||
|
||
## Report Shape
|
||
|
||
```json
|
||
{
|
||
"pilot_id": "phase3-pilot-YYYYMMDD",
|
||
"generated_at": "2026-06-26T00:00:00Z",
|
||
"source_window": {
|
||
"started_at": "2026-00-00T00:00:00Z",
|
||
"ended_at": "2026-00-00T00:00:00Z"
|
||
},
|
||
"cohort_size": 20,
|
||
"metrics": {
|
||
"pilot_completion": {},
|
||
"session_completion": {},
|
||
"embedding_consistency": {},
|
||
"hallucination_rate": {},
|
||
"top1": {},
|
||
"sus": {},
|
||
"inter_rater_kappa": {},
|
||
"icc": {},
|
||
"self_efficacy_prepost": {}
|
||
},
|
||
"exclusions": [],
|
||
"open_schema_gaps": [],
|
||
"review": {
|
||
"operator": "",
|
||
"reviewed_at": "",
|
||
"decision": "pending"
|
||
}
|
||
}
|
||
```
|
||
|
||
Each metric object must include:
|
||
|
||
```json
|
||
{
|
||
"value": 0,
|
||
"threshold": 0,
|
||
"pass": false,
|
||
"numerator": null,
|
||
"denominator": null,
|
||
"method": "",
|
||
"source_files": []
|
||
}
|
||
```
|
||
|
||
## KPI Definitions
|
||
|
||
| KPI | Gate | Definition | Evidence |
|
||
| --- | --- | --- | --- |
|
||
| Pilot completion | >=20 participants | Count active participants with consent, required sessions, pre/post, SUS, and supervisor review. | `pilot_roster.csv`, `session_completion.csv`, `prepost_measures.csv`, `sus_responses.csv` |
|
||
| Session completion | >=20 participants | Count participants with at least two `completed` sessions and non-empty `supervisor_reviewed_at`. | `session_completion.csv` |
|
||
| Embedding consistency | >=0.79 | Mean cosine similarity for persona invariant checks across masked session turns. Report model, embedding version, mean, p10, and count. | DB query export, `kpi_report.json` |
|
||
| Hallucination rate | <=0.03 (owner 2026-06-30) | Unsupported or contradicted factual claims divided by reviewed eligible evaluator/client-response claims. | supervisor review export, `kpi_report.json` |
|
||
| Top-1 retrieval | >=0.80 | Top-1 evidence match rate for golden feedback or RAG evaluation queries. | golden evaluation run, `kpi_report.json` |
|
||
| SUS | >=80 | Standard 10-item SUS score transformed to a 0-100 score; report mean, median, and response count. | `sus_responses.csv` |
|
||
| Cohen kappa | >=0.70 (owner 2026-06-30) | Agreement between human reviewers or human-vs-AI labels for categorical tags used in dataset export. | annotation export, `kpi_report.json` |
|
||
| ICC | >=0.75 | Intraclass correlation for numeric ratings such as feedback score dimensions. | annotation export, `kpi_report.json` |
|
||
| Self-efficacy pre/post | Report and review | Paired pre/post change for the approved self-efficacy measure. Report effect direction, test used, p-value, and missingness. | `prepost_measures.csv` |
|
||
|
||
## Metric Methods
|
||
|
||
### Pilot Completion
|
||
|
||
Computation:
|
||
|
||
```text
|
||
participant passes =
|
||
active consent receipt exists
|
||
AND >=2 completed sessions
|
||
AND every counted session has supervisor_reviewed_at
|
||
AND pre measure exists
|
||
AND post measure exists
|
||
AND SUS response set exists or missingness is explicitly reviewed
|
||
```
|
||
|
||
The dashboard gate passes when at least 20 participants pass. Withdrawn participants are
|
||
reported in exclusions and must not be silently removed from denominator notes.
|
||
|
||
### Embedding Consistency
|
||
|
||
Operator command placeholder:
|
||
|
||
```powershell
|
||
# Use the deployed reporting job when available. The output is copied into kpi_report.json.
|
||
python scripts\check-phase3-artifacts.py --check --evidence-root $root
|
||
```
|
||
|
||
Minimum report fields:
|
||
|
||
```json
|
||
{
|
||
"value": 0.79,
|
||
"threshold": 0.79,
|
||
"pass": true,
|
||
"method": "masked persona invariant cosine similarity",
|
||
"model": "embedding-model-name-and-version",
|
||
"p10": 0.0,
|
||
"count": 0,
|
||
"source_files": ["02-measures/persona_embedding_eval.csv"]
|
||
}
|
||
```
|
||
|
||
### Hallucination Rate
|
||
|
||
Count only reviewed claims that should be grounded in the persona card, session state,
|
||
approved knowledge base, or rubric. Unsupported clinical, safety, or personal-history
|
||
claims are high severity and must be listed in `review.exceptions`.
|
||
|
||
Formula:
|
||
|
||
```text
|
||
hallucination_rate = unsupported_or_contradicted_claims / reviewed_eligible_claims
|
||
```
|
||
|
||
### SUS
|
||
|
||
Report the standard SUS transformed 0-100 score. Do not copy item text into the
|
||
repository. Store only item number, numeric response, participant id, and timestamp.
|
||
|
||
Minimum report fields:
|
||
|
||
```json
|
||
{
|
||
"value": 80.0,
|
||
"threshold": 80.0,
|
||
"pass": true,
|
||
"mean": 80.0,
|
||
"median": 80.0,
|
||
"respondents": 20,
|
||
"source_files": ["02-measures/sus_responses.csv"]
|
||
}
|
||
```
|
||
|
||
### Inter-Rater Agreement
|
||
|
||
Use the same item universe for kappa and ICC that is later referenced by the dataset
|
||
export manifest. If agreement fails, the dataset can still be exported as a technical
|
||
artifact only when the manifest marks it `not_gold`.
|
||
|
||
### Pre/Post Export Scaffold
|
||
|
||
The current app stores aggregate pre/post values in `app.learner_prepost_measure`.
|
||
Generate the Phase 3 scaffold files with:
|
||
|
||
```powershell
|
||
python scripts\export-phase3-kpi.py --database-url $env:DATABASE_URL --output-root $root --pilot-id phase3-pilot-draft
|
||
```
|
||
|
||
This writes `02-measures/prepost_measures.csv` and `02-measures/kpi_report.json`.
|
||
Participant identifiers are pseudonymous `P3-###` keys. The report fills paired
|
||
normalized mean pre/post/delta and complete/missing pair counts for the three pilot
|
||
measures. It does not encode validated item scoring, experimental/control assignment,
|
||
statistical tests, or official pass/fail interpretation.
|
||
|
||
## Operator Checklist
|
||
|
||
- [ ] Evidence root exists outside the source tree.
|
||
- [ ] Roster and consent receipt counts match.
|
||
- [ ] Session completion rows are exported after supervisor review.
|
||
- [ ] Pre/post and SUS files are present.
|
||
- [ ] `kpi_report.json` contains all required metric keys.
|
||
- [ ] Every failed metric has an owner, disposition, and decision date.
|
||
- [ ] Export manifest references the same cohort and exclusions as the KPI report.
|
||
|
||
Validation command:
|
||
|
||
```powershell
|
||
python scripts\check-phase3-artifacts.py --check --evidence-root $root
|
||
```
|