vignette/docs/phase3/pilot-runbook.md

Phase 3 Education Pilot Runbook

Purpose

Run a 20-person education pilot with auditable evidence that each included participant was consented, completed the required learning sessions, produced pre/post measures, and was included or excluded from KPI and export outputs correctly.

This runbook assumes Phase 2 role separation, session persistence, supervisor review, and safety event logging are already deployed. It does not authorize real participant collection until the privacy and consent checklist is signed.

Entry Gates

• G-P3-01 Product scope is locked as education/training, not diagnosis or treatment.
• G-P3-02 Current consent form, privacy notice, retention rule, withdrawal path, and third-party/inference transfer disclosure are approved by the project owner and legal reviewer.
• G-P3-03 The pilot uses pseudonymous participant_id values only. Names, emails, phone numbers, student numbers, and direct identifiers are not stored in evidence files.
• G-P3-04 Safety policy is active: self-harm scenarios remain excluded unless a later approved protocol explicitly adds them.
• G-P3-05 Instructor/admin access has RBAC and audit logging enabled before any pilot session starts.
• G-P3-06 Evidence root has been created outside the source tree and the checker runs:

$root="D:\vignette-evidence\phase3-pilot-YYYYMMDD"
python scripts\check-phase3-artifacts.py --check --evidence-root $root

The first run is expected to fail on missing files. The run output becomes the setup gap list.

Participant Roster Gate

Target: 20 active, consented trainees.

Evidence file: 00-intake/pilot_roster.csv

Required header:

participant_id,cohort_id,consent_version,consent_signed_at,withdrawal_state,enrolled_at

Allowed withdrawal_state values:

• active
• withdrawn_before_data_use
• withdrawn_after_data_use
• excluded_by_operator

Rules:

• participant_id must be a pseudonymous value such as P3-001.
• The roster must not include names, emails, phone numbers, student IDs, addresses, or date of birth.
• A separate identity map may exist only in the approved restricted location owned by the data steward.
• The pilot can start when 20 participants are active with a non-empty consent_signed_at.

Evidence file: 00-intake/consent_receipts.csv

Required header:

participant_id,consent_version,signed_at,signer_role,receipt_id

Operator command:

python scripts\check-phase3-artifacts.py --check --evidence-root $root

Session Completion Gate

Minimum completion definition for dashboard Phase 3:

• Consent receipt exists before the first session.
• Participant has at least two completed simulation sessions.
• Each completed session has an end timestamp, non-zero learner turns, and supervisor review evidence.
• Participant has both pre and post measures.
• Participant has SUS responses or an explicit missing-response reason in the KPI report.

Evidence file: 01-sessions/session_completion.csv

Required header:

participant_id,session_id,persona_id,started_at,ended_at,completion_state,turns_count,supervisor_reviewed_at

Allowed completion_state values:

• completed
• abandoned
• operator_cancelled
• excluded_from_analysis

Operational cadence:

1. Before each session, confirm the participant remains active.
2. After each session, export one row to session_completion.csv.
3. Supervisor reviews the persisted session and records supervisor_reviewed_at.
4. Any safety event or access override is linked in the KPI report, not copied into the roster.

Session evidence query placeholder:

-- Run against the approved read-only reporting connection.
-- Replace table/column names with the deployed schema names if they differ.
SELECT
  s.learner_id AS participant_id,
  s.id AS session_id,
  s.persona_id,
  s.started_at,
  s.ended_at,
  CASE WHEN s.ended_at IS NOT NULL THEN 'completed' ELSE 'abandoned' END AS completion_state,
  COUNT(t.id) FILTER (WHERE t.speaker = 'counselor') AS turns_count,
  MAX(r.reviewed_at) AS supervisor_reviewed_at
FROM app.sessions s
LEFT JOIN app.turns t ON t.session_id = s.id
LEFT JOIN app.session_reviews r ON r.session_id = s.id
GROUP BY s.learner_id, s.id, s.persona_id, s.started_at, s.ended_at;

If the deployed schema does not yet have app.session_reviews, the operator must attach a signed reviewer export and note the schema gap in 02-measures/kpi_report.json.

Measure Collection Gate

Evidence file: 02-measures/prepost_measures.csv

Required header:

participant_id,measure_name,timepoint,score,collected_at

Allowed timepoint values:

• pre
• post

Evidence file: 02-measures/sus_responses.csv

Required header:

participant_id,item,response,collected_at

Rules:

• measure_name must be stable across pre and post rows.
• SUS rows use item numbers and numeric responses only. Do not store free-text comments in sus_responses.csv.
• Free-text participant feedback, if collected, must go through the privacy review path before being added to any export.

Exit Gate

Phase 3 pilot completion evidence is sufficient when:

• pilot_roster.csv contains at least 20 active, consented participants.
• At least 20 participants meet the minimum completion definition.
• kpi_report.json is generated and reviewed.
• export_manifest.json is generated, PII scan is pass or reviewed, and the export is approved.
• privacy_audit.md is signed with no unresolved high severity item.
• withdrawal_log.csv exists, even if it only contains the header.

Archive command:

python scripts\check-phase3-artifacts.py --check --evidence-root $root --output "$root\phase3_artifact_check.json"

The checker writes only the report path given by --output; otherwise it is read-only.