vignette/docs/phase3/kpi-measurement-plan.md

# Phase 3 KPI Measurement Plan

## Purpose

Define the Phase 3 KPI report so the dashboard gates are measured the same way each
time. These metrics are pilot evidence, not claims of validated clinical effectiveness.

Primary evidence file: `02-measures/kpi_report.json`

## Report Shape

```json
{
  "pilot_id": "phase3-pilot-YYYYMMDD",
  "generated_at": "2026-06-26T00:00:00Z",
  "source_window": {
    "started_at": "2026-00-00T00:00:00Z",
    "ended_at": "2026-00-00T00:00:00Z"
  },
  "cohort_size": 20,
  "metrics": {
    "pilot_completion": {},
    "session_completion": {},
    "embedding_consistency": {},
    "hallucination_rate": {},
    "top1": {},
    "sus": {},
    "inter_rater_kappa": {},
    "icc": {},
    "self_efficacy_prepost": {}
  },
  "exclusions": [],
  "open_schema_gaps": [],
  "review": {
    "operator": "",
    "reviewed_at": "",
    "decision": "pending"
  }
}
```

Each metric object must include:

```json
{
  "value": 0,
  "threshold": 0,
  "pass": false,
  "numerator": null,
  "denominator": null,
  "method": "",
  "source_files": []
}
```

## KPI Definitions

| KPI | Gate | Definition | Evidence |
| --- | --- | --- | --- |
| Pilot completion | >=20 participants | Count active participants with consent, required sessions, pre/post, SUS, and supervisor review. | `pilot_roster.csv`, `session_completion.csv`, `prepost_measures.csv`, `sus_responses.csv` |
| Session completion | >=20 participants | Count participants with at least two `completed` sessions and non-empty `supervisor_reviewed_at`. | `session_completion.csv` |
| Embedding consistency | >=0.79 | Mean cosine similarity for persona invariant checks across masked session turns. Report model, embedding version, mean, p10, and count. | DB query export, `kpi_report.json` |
| Hallucination rate | <=0.05, target <=0.01 | Unsupported or contradicted factual claims divided by reviewed eligible evaluator/client-response claims. | supervisor review export, `kpi_report.json` |
| Top-1 retrieval | >=0.80 | Top-1 evidence match rate for golden feedback or RAG evaluation queries. | golden evaluation run, `kpi_report.json` |
| SUS | >=80 | Standard 10-item SUS score transformed to a 0-100 score; report mean, median, and response count. | `sus_responses.csv` |
| Cohen kappa | >=0.60 | Agreement between human reviewers or human-vs-AI labels for categorical tags used in dataset export. | annotation export, `kpi_report.json` |
| ICC | >=0.75 | Intraclass correlation for numeric ratings such as feedback score dimensions. | annotation export, `kpi_report.json` |
| Self-efficacy pre/post | Report and review | Paired pre/post change for the approved self-efficacy measure. Report effect direction, test used, p-value, and missingness. | `prepost_measures.csv` |

## Metric Methods

### Pilot Completion

Computation:

```text
participant passes =
  active consent receipt exists
  AND >=2 completed sessions
  AND every counted session has supervisor_reviewed_at
  AND pre measure exists
  AND post measure exists
  AND SUS response set exists or missingness is explicitly reviewed
```

The dashboard gate passes when at least 20 participants pass. Withdrawn participants are
reported in exclusions and must not be silently removed from denominator notes.

### Embedding Consistency

Operator command placeholder:

```powershell
# Use the deployed reporting job when available. The output is copied into kpi_report.json.
python scripts\check-phase3-artifacts.py --check --evidence-root $root
```

Minimum report fields:

```json
{
  "value": 0.79,
  "threshold": 0.79,
  "pass": true,
  "method": "masked persona invariant cosine similarity",
  "model": "embedding-model-name-and-version",
  "p10": 0.0,
  "count": 0,
  "source_files": ["02-measures/persona_embedding_eval.csv"]
}
```

### Hallucination Rate

Count only reviewed claims that should be grounded in the persona card, session state,
approved knowledge base, or rubric. Unsupported clinical, safety, or personal-history
claims are high severity and must be listed in `review.exceptions`.

Formula:

```text
hallucination_rate = unsupported_or_contradicted_claims / reviewed_eligible_claims
```

### SUS

Report the standard SUS transformed 0-100 score. Do not copy item text into the
repository. Store only item number, numeric response, participant id, and timestamp.

Minimum report fields:

```json
{
  "value": 80.0,
  "threshold": 80.0,
  "pass": true,
  "mean": 80.0,
  "median": 80.0,
  "respondents": 20,
  "source_files": ["02-measures/sus_responses.csv"]
}
```

### Inter-Rater Agreement

Use the same item universe for kappa and ICC that is later referenced by the dataset
export manifest. If agreement fails, the dataset can still be exported as a technical
artifact only when the manifest marks it `not_gold`.

### Pre/Post Export Scaffold

The current app stores aggregate pre/post values in `app.learner_prepost_measure`.
Generate the Phase 3 scaffold files with:

```powershell
python scripts\export-phase3-kpi.py --database-url $env:DATABASE_URL --output-root $root --pilot-id phase3-pilot-draft
```

This writes `02-measures/prepost_measures.csv` and `02-measures/kpi_report.json`.
Participant identifiers are pseudonymous `P3-###` keys. The report fills paired
normalized mean pre/post/delta and complete/missing pair counts for the three pilot
measures. It does not encode validated item scoring, experimental/control assignment,
statistical tests, or official pass/fail interpretation.

## Operator Checklist

- [ ] Evidence root exists outside the source tree.
- [ ] Roster and consent receipt counts match.
- [ ] Session completion rows are exported after supervisor review.
- [ ] Pre/post and SUS files are present.
- [ ] `kpi_report.json` contains all required metric keys.
- [ ] Every failed metric has an owner, disposition, and decision date.
- [ ] Export manifest references the same cohort and exclusions as the KPI report.

Validation command:

```powershell
python scripts\check-phase3-artifacts.py --check --evidence-root $root
```