vignette/docs/phase3/kpi-measurement-plan.md

Phase 3 KPI Measurement Plan

Purpose

Define the Phase 3 KPI report so the dashboard gates are measured the same way each time. These metrics are pilot evidence, not claims of validated clinical effectiveness.

Primary evidence file: 02-measures/kpi_report.json

Report Shape

{
  "pilot_id": "phase3-pilot-YYYYMMDD",
  "generated_at": "2026-06-26T00:00:00Z",
  "source_window": {
    "started_at": "2026-00-00T00:00:00Z",
    "ended_at": "2026-00-00T00:00:00Z"
  },
  "cohort_size": 20,
  "metrics": {
    "pilot_completion": {},
    "session_completion": {},
    "embedding_consistency": {},
    "hallucination_rate": {},
    "top1": {},
    "sus": {},
    "inter_rater_kappa": {},
    "icc": {},
    "self_efficacy_prepost": {}
  },
  "exclusions": [],
  "open_schema_gaps": [],
  "review": {
    "operator": "",
    "reviewed_at": "",
    "decision": "pending"
  }
}

Each metric object must include:

{
  "value": 0,
  "threshold": 0,
  "pass": false,
  "numerator": null,
  "denominator": null,
  "method": "",
  "source_files": []
}

KPI Definitions

KPI	Gate	Definition	Evidence
Pilot completion	>=20 participants	Count active participants with consent, required sessions, pre/post, SUS, and supervisor review.	pilot_roster.csv, session_completion.csv, prepost_measures.csv, sus_responses.csv
Session completion	>=20 participants	Count participants with at least two completed sessions and non-empty supervisor_reviewed_at.	session_completion.csv
Embedding consistency	>=0.79	Mean cosine similarity for persona invariant checks across masked session turns. Report model, embedding version, mean, p10, and count.	DB query export, kpi_report.json
Hallucination rate	<=0.05, target <=0.01	Unsupported or contradicted factual claims divided by reviewed eligible evaluator/client-response claims.	supervisor review export, kpi_report.json
Top-1 retrieval	>=0.80	Top-1 evidence match rate for golden feedback or RAG evaluation queries.	golden evaluation run, kpi_report.json
SUS	>=80	Standard 10-item SUS score transformed to a 0-100 score; report mean, median, and response count.	sus_responses.csv
Cohen kappa	>=0.60	Agreement between human reviewers or human-vs-AI labels for categorical tags used in dataset export.	annotation export, kpi_report.json
ICC	>=0.75	Intraclass correlation for numeric ratings such as feedback score dimensions.	annotation export, kpi_report.json
Self-efficacy pre/post	Report and review	Paired pre/post change for the approved self-efficacy measure. Report effect direction, test used, p-value, and missingness.	prepost_measures.csv

Metric Methods

Pilot Completion

Computation:

participant passes =
  active consent receipt exists
  AND >=2 completed sessions
  AND every counted session has supervisor_reviewed_at
  AND pre measure exists
  AND post measure exists
  AND SUS response set exists or missingness is explicitly reviewed

The dashboard gate passes when at least 20 participants pass. Withdrawn participants are reported in exclusions and must not be silently removed from denominator notes.

Embedding Consistency

Operator command placeholder:

# Use the deployed reporting job when available. The output is copied into kpi_report.json.
python scripts\check-phase3-artifacts.py --check --evidence-root $root

Minimum report fields:

{
  "value": 0.79,
  "threshold": 0.79,
  "pass": true,
  "method": "masked persona invariant cosine similarity",
  "model": "embedding-model-name-and-version",
  "p10": 0.0,
  "count": 0,
  "source_files": ["02-measures/persona_embedding_eval.csv"]
}

Hallucination Rate

Count only reviewed claims that should be grounded in the persona card, session state, approved knowledge base, or rubric. Unsupported clinical, safety, or personal-history claims are high severity and must be listed in review.exceptions.

Formula:

hallucination_rate = unsupported_or_contradicted_claims / reviewed_eligible_claims

SUS

Report the standard SUS transformed 0-100 score. Do not copy item text into the repository. Store only item number, numeric response, participant id, and timestamp.

Minimum report fields:

{
  "value": 80.0,
  "threshold": 80.0,
  "pass": true,
  "mean": 80.0,
  "median": 80.0,
  "respondents": 20,
  "source_files": ["02-measures/sus_responses.csv"]
}

Inter-Rater Agreement

Use the same item universe for kappa and ICC that is later referenced by the dataset export manifest. If agreement fails, the dataset can still be exported as a technical artifact only when the manifest marks it not_gold.

Operator Checklist

• Evidence root exists outside the source tree.
• Roster and consent receipt counts match.
• Session completion rows are exported after supervisor review.
• Pre/post and SUS files are present.
• kpi_report.json contains all required metric keys.
• Every failed metric has an owner, disposition, and decision date.
• Export manifest references the same cohort and exclusions as the KPI report.

Validation command:

python scripts\check-phase3-artifacts.py --check --evidence-root $root