vignette/docs/phase3/kpi-measurement-plan.md

Phase 3 KPI Measurement Plan

Purpose

Define the Phase 3 KPI report so the dashboard gates are measured the same way each time. These metrics are pilot evidence, not claims of validated clinical effectiveness.

Primary evidence file: 02-measures/kpi_report.json

Evaluation Design (owner-confirmed 2026-06-30)

• 표본·배정: 20명을 실험군/통제군으로 무작위 배정(randomized). 단회기 50분, 3척도(self-efficacy / skill proficiency / training satisfaction) pre-post.
• 통계검정: 군 간 비교는 독립표본 t-검정, 유의수준 α=0.05. 정규성 위반 시 Mann-Whitney U로 대체 가능하나 기본은 t-검정.
• 신뢰도 목표: 평가자 간 κ≥0.70(Cohen's kappa), 수치 평정 ICC≥0.75.
• 환각률 목표: 상담 응답 환각률 ≤0.03.
• 위 파라미터는 설계 확정값이며, 실제 효과성 판정·통계해석은 20명 파일럿 evidence 수집 후 수행한다(파일럿 게이트).

Report Shape

{
  "pilot_id": "phase3-pilot-YYYYMMDD",
  "generated_at": "2026-06-26T00:00:00Z",
  "source_window": {
    "started_at": "2026-00-00T00:00:00Z",
    "ended_at": "2026-00-00T00:00:00Z"
  },
  "cohort_size": 20,
  "metrics": {
    "pilot_completion": {},
    "session_completion": {},
    "embedding_consistency": {},
    "hallucination_rate": {},
    "top1": {},
    "sus": {},
    "inter_rater_kappa": {},
    "icc": {},
    "self_efficacy_prepost": {}
  },
  "exclusions": [],
  "open_schema_gaps": [],
  "review": {
    "operator": "",
    "reviewed_at": "",
    "decision": "pending"
  }
}

Each metric object must include:

{
  "value": 0,
  "threshold": 0,
  "pass": false,
  "numerator": null,
  "denominator": null,
  "method": "",
  "source_files": []
}

KPI Definitions

KPI	Gate	Definition	Evidence
Pilot completion	>=20 participants	Count active participants with consent, required sessions, pre/post, SUS, and supervisor review.	pilot_roster.csv, session_completion.csv, prepost_measures.csv, sus_responses.csv
Session completion	>=20 participants	Count participants with at least two completed sessions and non-empty supervisor_reviewed_at.	session_completion.csv
Embedding consistency	>=0.79	Mean cosine similarity for persona invariant checks across masked session turns. Report model, embedding version, mean, p10, and count.	DB query export, kpi_report.json
Hallucination rate	<=0.03 (owner 2026-06-30)	Unsupported or contradicted factual claims divided by reviewed eligible evaluator/client-response claims.	supervisor review export, kpi_report.json
Top-1 retrieval	>=0.80	Top-1 evidence match rate for golden feedback or RAG evaluation queries.	golden evaluation run, kpi_report.json
SUS	>=80	Standard 10-item SUS score transformed to a 0-100 score; report mean, median, and response count.	sus_responses.csv
Cohen kappa	>=0.70 (owner 2026-06-30)	Agreement between human reviewers or human-vs-AI labels for categorical tags used in dataset export.	annotation export, kpi_report.json
ICC	>=0.75	Intraclass correlation for numeric ratings such as feedback score dimensions.	annotation export, kpi_report.json
Self-efficacy pre/post	Report and review	Paired pre/post change for the approved self-efficacy measure. Report effect direction, test used, p-value, and missingness.	prepost_measures.csv

Metric Methods

Pilot Completion

Computation:

participant passes =
  active consent receipt exists
  AND >=2 completed sessions
  AND every counted session has supervisor_reviewed_at
  AND pre measure exists
  AND post measure exists
  AND SUS response set exists or missingness is explicitly reviewed

The dashboard gate passes when at least 20 participants pass. Withdrawn participants are reported in exclusions and must not be silently removed from denominator notes.

Embedding Consistency

Operator command placeholder:

# Use the deployed reporting job when available. The output is copied into kpi_report.json.
python scripts\check-phase3-artifacts.py --check --evidence-root $root

Minimum report fields:

{
  "value": 0.79,
  "threshold": 0.79,
  "pass": true,
  "method": "masked persona invariant cosine similarity",
  "model": "embedding-model-name-and-version",
  "p10": 0.0,
  "count": 0,
  "source_files": ["02-measures/persona_embedding_eval.csv"]
}

Hallucination Rate

Count only reviewed claims that should be grounded in the persona card, session state, approved knowledge base, or rubric. Unsupported clinical, safety, or personal-history claims are high severity and must be listed in review.exceptions.

Formula:

hallucination_rate = unsupported_or_contradicted_claims / reviewed_eligible_claims

SUS

Report the standard SUS transformed 0-100 score. Do not copy item text into the repository. Store only item number, numeric response, participant id, and timestamp.

Minimum report fields:

{
  "value": 80.0,
  "threshold": 80.0,
  "pass": true,
  "mean": 80.0,
  "median": 80.0,
  "respondents": 20,
  "source_files": ["02-measures/sus_responses.csv"]
}

Inter-Rater Agreement

Use the same item universe for kappa and ICC that is later referenced by the dataset export manifest. If agreement fails, the dataset can still be exported as a technical artifact only when the manifest marks it not_gold.

Pre/Post Export Scaffold

The current app stores aggregate pre/post values in app.learner_prepost_measure. Generate the Phase 3 scaffold files with:

python scripts\export-phase3-kpi.py --database-url $env:DATABASE_URL --output-root $root --pilot-id phase3-pilot-draft

This writes 02-measures/prepost_measures.csv and 02-measures/kpi_report.json. Participant identifiers are pseudonymous P3-### keys. The report fills paired normalized mean pre/post/delta and complete/missing pair counts for the three pilot measures. It does not encode validated item scoring, experimental/control assignment, statistical tests, or official pass/fail interpretation.

Operator Checklist

• Evidence root exists outside the source tree.
• Roster and consent receipt counts match.
• Session completion rows are exported after supervisor review.
• Pre/post and SUS files are present.
• kpi_report.json contains all required metric keys.
• Every failed metric has an owner, disposition, and decision date.
• Export manifest references the same cohort and exclusions as the KPI report.

Validation command:

python scripts\check-phase3-artifacts.py --check --evidence-root $root