Phase 3 KPI Measurement Plan

Purpose

Define the Phase 3 KPI report so the dashboard gates are measured the same way each time. These metrics are pilot evidence, not claims of validated clinical effectiveness.

Primary evidence file: 02-measures/kpi_report.json

Report Shape

{
  "pilot_id": "phase3-pilot-YYYYMMDD",
  "generated_at": "2026-06-26T00:00:00Z",
  "source_window": {
    "started_at": "2026-00-00T00:00:00Z",
    "ended_at": "2026-00-00T00:00:00Z"
  },
  "cohort_size": 20,
  "metrics": {
    "pilot_completion": {},
    "session_completion": {},
    "embedding_consistency": {},
    "hallucination_rate": {},
    "top1": {},
    "sus": {},
    "inter_rater_kappa": {},
    "icc": {},
    "self_efficacy_prepost": {}
  },
  "exclusions": [],
  "open_schema_gaps": [],
  "review": {
    "operator": "",
    "reviewed_at": "",
    "decision": "pending"
  }
}

Each metric object must include:

{
  "value": 0,
  "threshold": 0,
  "pass": false,
  "numerator": null,
  "denominator": null,
  "method": "",
  "source_files": []
}

KPI Definitions

KPI	Gate	Definition	Evidence
Pilot completion	>=20 participants	Count active participants with consent, required sessions, pre/post, SUS, and supervisor review.	`pilot_roster.csv`, `session_completion.csv`, `prepost_measures.csv`, `sus_responses.csv`
Session completion	>=20 participants	Count participants with at least two `completed` sessions and non-empty `supervisor_reviewed_at`.	`session_completion.csv`
Embedding consistency	>=0.79	Mean cosine similarity for persona invariant checks across masked session turns. Report model, embedding version, mean, p10, and count.	DB query export, `kpi_report.json`
Hallucination rate	<=0.05, target <=0.01	Unsupported or contradicted factual claims divided by reviewed eligible evaluator/client-response claims.	supervisor review export, `kpi_report.json`
Top-1 retrieval	>=0.80	Top-1 evidence match rate for golden feedback or RAG evaluation queries.	golden evaluation run, `kpi_report.json`
SUS	>=80	Standard 10-item SUS score transformed to a 0-100 score; report mean, median, and response count.	`sus_responses.csv`
Cohen kappa	>=0.60	Agreement between human reviewers or human-vs-AI labels for categorical tags used in dataset export.	annotation export, `kpi_report.json`
ICC	>=0.75	Intraclass correlation for numeric ratings such as feedback score dimensions.	annotation export, `kpi_report.json`
Self-efficacy pre/post	Report and review	Paired pre/post change for the approved self-efficacy measure. Report effect direction, test used, p-value, and missingness.	`prepost_measures.csv`

Metric Methods

Pilot Completion

Computation:

participant passes =
  active consent receipt exists
  AND >=2 completed sessions
  AND every counted session has supervisor_reviewed_at
  AND pre measure exists
  AND post measure exists
  AND SUS response set exists or missingness is explicitly reviewed

The dashboard gate passes when at least 20 participants pass. Withdrawn participants are reported in exclusions and must not be silently removed from denominator notes.

Embedding Consistency

Operator command placeholder:

# Use the deployed reporting job when available. The output is copied into kpi_report.json.
python scripts\check-phase3-artifacts.py --check --evidence-root $root

Minimum report fields:

{
  "value": 0.79,
  "threshold": 0.79,
  "pass": true,
  "method": "masked persona invariant cosine similarity",
  "model": "embedding-model-name-and-version",
  "p10": 0.0,
  "count": 0,
  "source_files": ["02-measures/persona_embedding_eval.csv"]
}

Hallucination Rate

Count only reviewed claims that should be grounded in the persona card, session state, approved knowledge base, or rubric. Unsupported clinical, safety, or personal-history claims are high severity and must be listed in review.exceptions.

Formula:

hallucination_rate = unsupported_or_contradicted_claims / reviewed_eligible_claims

SUS

Report the standard SUS transformed 0-100 score. Do not copy item text into the repository. Store only item number, numeric response, participant id, and timestamp.

Minimum report fields:

{
  "value": 80.0,
  "threshold": 80.0,
  "pass": true,
  "mean": 80.0,
  "median": 80.0,
  "respondents": 20,
  "source_files": ["02-measures/sus_responses.csv"]
}

Inter-Rater Agreement

Use the same item universe for kappa and ICC that is later referenced by the dataset export manifest. If agreement fails, the dataset can still be exported as a technical artifact only when the manifest marks it not_gold.

Pre/Post Export Scaffold

The current app stores aggregate pre/post values in app.learner_prepost_measure. Generate the Phase 3 scaffold files with:

python scripts\export-phase3-kpi.py --database-url $env:DATABASE_URL --output-root $root --pilot-id phase3-pilot-draft

This writes 02-measures/prepost_measures.csv and 02-measures/kpi_report.json. Participant identifiers are pseudonymous P3-### keys. The report fills paired normalized mean pre/post/delta and complete/missing pair counts for the three pilot measures. It does not encode validated item scoring, experimental/control assignment, statistical tests, or official pass/fail interpretation.

Operator Checklist

Evidence root exists outside the source tree.
Roster and consent receipt counts match.
Session completion rows are exported after supervisor review.
Pre/post and SUS files are present.
kpi_report.json contains all required metric keys.
Every failed metric has an owner, disposition, and decision date.
Export manifest references the same cohort and exclusions as the KPI report.

Validation command:

python scripts\check-phase3-artifacts.py --check --evidence-root $root

5.9 KiB Raw Blame History