대시보드 폴드아웃/드릴다운 정리 + 페르소나 역린·misconduct 반응 + 게이트웨이 격리·RAG 비차단 수정
SSOT 대시보드:
- 한신대 기술분석 PDF(19쪽) 정합성 분석 + 이번 세션 발견 섹션 추가
- 섹션 폴드아웃(접기)·상단 목차(드릴다운)·모두 펼치기/접기 — 내용 보존, 레이아웃만 정리
페르소나 반응 강화('저항·반응 조절' 핵심 차별):
- PersonaCard.triggers(역린) 필드 + CCD 핵심상처 파생 역린 블록
- L0에 무례·모욕·조롱 시 현실적 동맹 균열 반응 지침
버그·성능 수정(라이브/E2E로 포착):
- 게이트웨이 페르소나 격리: --append-system-prompt를 --system-prompt(교체)로 + --exclude-dynamic-system-prompt-sections (내담자 캐릭터 붕괴·개발맥락 누출 차단)
- RAG: 임베더 동기 로드(약 7-13초)를 _warm_rag_caches 백그라운드 warm으로(세션 생성 블로킹 회귀 수정)
- voice TTS RMS 데드힌트 제거, init_state OpennessParams 파라미터객체화
- 한국어 PII(날짜·금액·주소) 마스킹 보강
- 레이아웃 시각 게이트: 폼 컨트롤 값 스크롤 오탐 제외(7/7)
검증: 백엔드 84/84, E2E 42(데스크톱 27·모바일 11·아바타 4), 시각 게이트 7/7
This commit is contained in:
parent
cb2aebd76c
commit
085460b5e0
327 changed files with 31226 additions and 1829 deletions
58
docs/phase3/README.md
Normal file
58
docs/phase3/README.md
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
# Phase 3 Operational Gates
|
||||
|
||||
This folder turns the Phase 3 dashboard items into executable operator gates for the
|
||||
20-person education pilot. It is not a legal opinion; the privacy and consent gates
|
||||
must be reviewed by the project data steward and counsel before real participant data
|
||||
is collected.
|
||||
|
||||
## Artifacts
|
||||
|
||||
- [pilot-runbook.md](pilot-runbook.md) - roster, session completion, operator cadence,
|
||||
and evidence files for the 20-person education pilot.
|
||||
- [kpi-measurement-plan.md](kpi-measurement-plan.md) - KPI definitions, thresholds,
|
||||
data sources, and report shape.
|
||||
- [dataset-export-manifest.md](dataset-export-manifest.md) - anonymized JSONL export
|
||||
and manifest expectations for recursive-learning dataset output.
|
||||
- [privacy-consent-audit.md](privacy-consent-audit.md) - consent, withdrawal, privacy,
|
||||
and audit checklist.
|
||||
|
||||
## Evidence Root
|
||||
|
||||
Operators should keep real pilot evidence outside the source tree by default, for
|
||||
example:
|
||||
|
||||
```powershell
|
||||
$env:PHASE3_EVIDENCE_ROOT="D:\vignette-evidence\phase3-pilot-YYYYMMDD"
|
||||
python scripts\check-phase3-artifacts.py --check --evidence-root $env:PHASE3_EVIDENCE_ROOT
|
||||
```
|
||||
|
||||
The expected evidence root layout is:
|
||||
|
||||
```text
|
||||
00-intake/
|
||||
pilot_roster.csv
|
||||
consent_receipts.csv
|
||||
01-sessions/
|
||||
session_completion.csv
|
||||
02-measures/
|
||||
prepost_measures.csv
|
||||
sus_responses.csv
|
||||
kpi_report.json
|
||||
03-export/
|
||||
export_manifest.json
|
||||
04-privacy/
|
||||
privacy_audit.md
|
||||
withdrawal_log.csv
|
||||
```
|
||||
|
||||
No evidence file should contain names, email addresses, phone numbers, raw student IDs,
|
||||
raw source case records, raw audio, or third-party API secrets. Participant identity
|
||||
mapping, if required, must live in the approved restricted location owned by the data
|
||||
steward, not in the export or this repository.
|
||||
|
||||
## Phase 3 Dashboard Coverage
|
||||
|
||||
- 20-person education pilot: `pilot-runbook.md`
|
||||
- KPI measurement: `kpi-measurement-plan.md`
|
||||
- Recursive-learning dataset export: `dataset-export-manifest.md`
|
||||
- Privacy, consent, withdrawal audit: `privacy-consent-audit.md`
|
||||
175
docs/phase3/dataset-export-manifest.md
Normal file
175
docs/phase3/dataset-export-manifest.md
Normal file
|
|
@ -0,0 +1,175 @@
|
|||
# Phase 3 Anonymized Dataset Export Manifest
|
||||
|
||||
## Purpose
|
||||
|
||||
Define the manifest and JSONL expectations for the first recursive-learning dataset
|
||||
artifact. The export is allowed only after consent scope, withdrawal state, PII masking,
|
||||
and reviewer agreement are documented.
|
||||
|
||||
Primary evidence file: `03-export/export_manifest.json`
|
||||
|
||||
## Export Gate
|
||||
|
||||
The export may be marked `approved_for_recursive_learning_seed` only when:
|
||||
|
||||
- Source participants are active or otherwise allowed by consent scope at export time.
|
||||
- Withdrawn participants are excluded according to the approved withdrawal policy.
|
||||
- Direct identifiers and raw identity mapping are absent from exported files.
|
||||
- Free text is masked or transformed according to the manifest.
|
||||
- PII scan is pass, or every finding has a reviewer disposition.
|
||||
- IAA gates pass for gold exports: kappa >=0.60 and ICC >=0.75.
|
||||
- Legal/data-steward review is complete.
|
||||
|
||||
If any gate is missing, set `export_status` to `technical_dry_run` or `blocked`, not
|
||||
`approved_for_recursive_learning_seed`.
|
||||
|
||||
## Manifest Shape
|
||||
|
||||
```json
|
||||
{
|
||||
"export_id": "phase3-rl-seed-YYYYMMDD-001",
|
||||
"dataset_name": "vignette_phase3_recursive_learning_seed",
|
||||
"export_status": "technical_dry_run",
|
||||
"created_at": "2026-06-26T00:00:00Z",
|
||||
"purpose": "recursive-learning seed dataset for education simulator improvement",
|
||||
"source_window": {
|
||||
"started_at": "2026-00-00T00:00:00Z",
|
||||
"ended_at": "2026-00-00T00:00:00Z"
|
||||
},
|
||||
"source_tables": [
|
||||
"app.sessions",
|
||||
"app.turns",
|
||||
"app.feedback_scores",
|
||||
"ds.annotation",
|
||||
"audit.persona_drift_log"
|
||||
],
|
||||
"selection_criteria": {
|
||||
"cohort_id": "phase3",
|
||||
"min_completed_sessions": 2,
|
||||
"include_withdrawn": false,
|
||||
"excluded_safety_scope": ["self_harm_scenario_primary"]
|
||||
},
|
||||
"consent_scope": {
|
||||
"consent_version": "",
|
||||
"allowed_uses": ["education_quality_review", "recursive_learning_seed"],
|
||||
"withdrawal_cutoff_applied_at": "",
|
||||
"participants_included": 0,
|
||||
"participants_excluded": 0
|
||||
},
|
||||
"anonymization": {
|
||||
"participant_key": "pseudonymous export key; no identity map included",
|
||||
"text_transform": "masked_text_only",
|
||||
"direct_identifier_policy": "blocked",
|
||||
"salt_or_identity_map_location": "not in export"
|
||||
},
|
||||
"pii_scan": {
|
||||
"tool": "",
|
||||
"version": "",
|
||||
"ran_at": "",
|
||||
"status": "pending",
|
||||
"findings": []
|
||||
},
|
||||
"agreement": {
|
||||
"kappa": null,
|
||||
"icc": null,
|
||||
"gold_status": "not_gold"
|
||||
},
|
||||
"files": [
|
||||
{
|
||||
"path": "03-export/anonymized_dataset.jsonl",
|
||||
"rows": 0,
|
||||
"sha256": "",
|
||||
"schema": "phase3_dataset_item_v1"
|
||||
}
|
||||
],
|
||||
"approvals": {
|
||||
"data_steward": "",
|
||||
"legal_or_privacy_reviewer": "",
|
||||
"technical_operator": "",
|
||||
"approved_at": ""
|
||||
},
|
||||
"known_limitations": []
|
||||
}
|
||||
```
|
||||
|
||||
## JSONL Record Shape
|
||||
|
||||
Each JSONL line must be one dataset item:
|
||||
|
||||
```json
|
||||
{
|
||||
"schema": "phase3_dataset_item_v1",
|
||||
"item_id": "export-local-id",
|
||||
"participant_key": "PX-0001",
|
||||
"session_key": "SX-0001",
|
||||
"turn_key": "TX-0001",
|
||||
"persona_id": "P1",
|
||||
"stage": "rapport",
|
||||
"speaker": "counselor",
|
||||
"text_masked": "masked or transformed utterance text",
|
||||
"techniques": [],
|
||||
"client_states": [],
|
||||
"feedback_scores": [],
|
||||
"supervisor_comments": [],
|
||||
"source_refs": {
|
||||
"session_started_at": "2026-00-00T00:00:00Z",
|
||||
"export_manifest_id": "phase3-rl-seed-YYYYMMDD-001"
|
||||
},
|
||||
"privacy": {
|
||||
"direct_identifiers_removed": true,
|
||||
"pii_scan_status": "pass",
|
||||
"consent_scope": "recursive_learning_seed"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Blocked fields:
|
||||
|
||||
- name
|
||||
- email
|
||||
- phone
|
||||
- student_id
|
||||
- national_id
|
||||
- address
|
||||
- date_of_birth
|
||||
- raw_audio_path
|
||||
- raw_voice
|
||||
- raw_source_case
|
||||
- identity_map
|
||||
- API keys, tokens, cookies, or credentials
|
||||
|
||||
## Operator Commands
|
||||
|
||||
Generate the export only through an approved read-only reporting job. Until that job is
|
||||
implemented, use the manifest as a contract and mark exports `technical_dry_run`.
|
||||
|
||||
Validate artifact presence and headers:
|
||||
|
||||
```powershell
|
||||
python scripts\check-phase3-artifacts.py --check --evidence-root $root
|
||||
```
|
||||
|
||||
Write a validation report:
|
||||
|
||||
```powershell
|
||||
python scripts\check-phase3-artifacts.py --check --evidence-root $root --output "$root\phase3_artifact_check.json"
|
||||
```
|
||||
|
||||
Compute a file hash for the manifest:
|
||||
|
||||
```powershell
|
||||
Get-FileHash "$root\03-export\anonymized_dataset.jsonl" -Algorithm SHA256
|
||||
```
|
||||
|
||||
## Export Review Checklist
|
||||
|
||||
- [ ] Manifest `export_status` matches the actual approval state.
|
||||
- [ ] Files listed in `files[]` exist and have row counts and SHA256 hashes.
|
||||
- [ ] `participants_included` plus `participants_excluded` matches the consent roster
|
||||
decision log.
|
||||
- [ ] `include_withdrawn` is false unless legal review explicitly approved otherwise.
|
||||
- [ ] Text fields use masked/transformed text only.
|
||||
- [ ] PII scan report is attached or referenced.
|
||||
- [ ] Agreement metrics match `02-measures/kpi_report.json`.
|
||||
- [ ] Export limitations state whether the file is gold data or a dry-run seed.
|
||||
|
||||
165
docs/phase3/kpi-measurement-plan.md
Normal file
165
docs/phase3/kpi-measurement-plan.md
Normal file
|
|
@ -0,0 +1,165 @@
|
|||
# Phase 3 KPI Measurement Plan
|
||||
|
||||
## Purpose
|
||||
|
||||
Define the Phase 3 KPI report so the dashboard gates are measured the same way each
|
||||
time. These metrics are pilot evidence, not claims of validated clinical effectiveness.
|
||||
|
||||
Primary evidence file: `02-measures/kpi_report.json`
|
||||
|
||||
## Report Shape
|
||||
|
||||
```json
|
||||
{
|
||||
"pilot_id": "phase3-pilot-YYYYMMDD",
|
||||
"generated_at": "2026-06-26T00:00:00Z",
|
||||
"source_window": {
|
||||
"started_at": "2026-00-00T00:00:00Z",
|
||||
"ended_at": "2026-00-00T00:00:00Z"
|
||||
},
|
||||
"cohort_size": 20,
|
||||
"metrics": {
|
||||
"pilot_completion": {},
|
||||
"session_completion": {},
|
||||
"embedding_consistency": {},
|
||||
"hallucination_rate": {},
|
||||
"top1": {},
|
||||
"sus": {},
|
||||
"inter_rater_kappa": {},
|
||||
"icc": {},
|
||||
"self_efficacy_prepost": {}
|
||||
},
|
||||
"exclusions": [],
|
||||
"open_schema_gaps": [],
|
||||
"review": {
|
||||
"operator": "",
|
||||
"reviewed_at": "",
|
||||
"decision": "pending"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Each metric object must include:
|
||||
|
||||
```json
|
||||
{
|
||||
"value": 0,
|
||||
"threshold": 0,
|
||||
"pass": false,
|
||||
"numerator": null,
|
||||
"denominator": null,
|
||||
"method": "",
|
||||
"source_files": []
|
||||
}
|
||||
```
|
||||
|
||||
## KPI Definitions
|
||||
|
||||
| KPI | Gate | Definition | Evidence |
|
||||
| --- | --- | --- | --- |
|
||||
| Pilot completion | >=20 participants | Count active participants with consent, required sessions, pre/post, SUS, and supervisor review. | `pilot_roster.csv`, `session_completion.csv`, `prepost_measures.csv`, `sus_responses.csv` |
|
||||
| Session completion | >=20 participants | Count participants with at least two `completed` sessions and non-empty `supervisor_reviewed_at`. | `session_completion.csv` |
|
||||
| Embedding consistency | >=0.79 | Mean cosine similarity for persona invariant checks across masked session turns. Report model, embedding version, mean, p10, and count. | DB query export, `kpi_report.json` |
|
||||
| Hallucination rate | <=0.05, target <=0.01 | Unsupported or contradicted factual claims divided by reviewed eligible evaluator/client-response claims. | supervisor review export, `kpi_report.json` |
|
||||
| Top-1 retrieval | >=0.80 | Top-1 evidence match rate for golden feedback or RAG evaluation queries. | golden evaluation run, `kpi_report.json` |
|
||||
| SUS | >=80 | Standard 10-item SUS score transformed to a 0-100 score; report mean, median, and response count. | `sus_responses.csv` |
|
||||
| Cohen kappa | >=0.60 | Agreement between human reviewers or human-vs-AI labels for categorical tags used in dataset export. | annotation export, `kpi_report.json` |
|
||||
| ICC | >=0.75 | Intraclass correlation for numeric ratings such as feedback score dimensions. | annotation export, `kpi_report.json` |
|
||||
| Self-efficacy pre/post | Report and review | Paired pre/post change for the approved self-efficacy measure. Report effect direction, test used, p-value, and missingness. | `prepost_measures.csv` |
|
||||
|
||||
## Metric Methods
|
||||
|
||||
### Pilot Completion
|
||||
|
||||
Computation:
|
||||
|
||||
```text
|
||||
participant passes =
|
||||
active consent receipt exists
|
||||
AND >=2 completed sessions
|
||||
AND every counted session has supervisor_reviewed_at
|
||||
AND pre measure exists
|
||||
AND post measure exists
|
||||
AND SUS response set exists or missingness is explicitly reviewed
|
||||
```
|
||||
|
||||
The dashboard gate passes when at least 20 participants pass. Withdrawn participants are
|
||||
reported in exclusions and must not be silently removed from denominator notes.
|
||||
|
||||
### Embedding Consistency
|
||||
|
||||
Operator command placeholder:
|
||||
|
||||
```powershell
|
||||
# Use the deployed reporting job when available. The output is copied into kpi_report.json.
|
||||
python scripts\check-phase3-artifacts.py --check --evidence-root $root
|
||||
```
|
||||
|
||||
Minimum report fields:
|
||||
|
||||
```json
|
||||
{
|
||||
"value": 0.79,
|
||||
"threshold": 0.79,
|
||||
"pass": true,
|
||||
"method": "masked persona invariant cosine similarity",
|
||||
"model": "embedding-model-name-and-version",
|
||||
"p10": 0.0,
|
||||
"count": 0,
|
||||
"source_files": ["02-measures/persona_embedding_eval.csv"]
|
||||
}
|
||||
```
|
||||
|
||||
### Hallucination Rate
|
||||
|
||||
Count only reviewed claims that should be grounded in the persona card, session state,
|
||||
approved knowledge base, or rubric. Unsupported clinical, safety, or personal-history
|
||||
claims are high severity and must be listed in `review.exceptions`.
|
||||
|
||||
Formula:
|
||||
|
||||
```text
|
||||
hallucination_rate = unsupported_or_contradicted_claims / reviewed_eligible_claims
|
||||
```
|
||||
|
||||
### SUS
|
||||
|
||||
Report the standard SUS transformed 0-100 score. Do not copy item text into the
|
||||
repository. Store only item number, numeric response, participant id, and timestamp.
|
||||
|
||||
Minimum report fields:
|
||||
|
||||
```json
|
||||
{
|
||||
"value": 80.0,
|
||||
"threshold": 80.0,
|
||||
"pass": true,
|
||||
"mean": 80.0,
|
||||
"median": 80.0,
|
||||
"respondents": 20,
|
||||
"source_files": ["02-measures/sus_responses.csv"]
|
||||
}
|
||||
```
|
||||
|
||||
### Inter-Rater Agreement
|
||||
|
||||
Use the same item universe for kappa and ICC that is later referenced by the dataset
|
||||
export manifest. If agreement fails, the dataset can still be exported as a technical
|
||||
artifact only when the manifest marks it `not_gold`.
|
||||
|
||||
## Operator Checklist
|
||||
|
||||
- [ ] Evidence root exists outside the source tree.
|
||||
- [ ] Roster and consent receipt counts match.
|
||||
- [ ] Session completion rows are exported after supervisor review.
|
||||
- [ ] Pre/post and SUS files are present.
|
||||
- [ ] `kpi_report.json` contains all required metric keys.
|
||||
- [ ] Every failed metric has an owner, disposition, and decision date.
|
||||
- [ ] Export manifest references the same cohort and exclusions as the KPI report.
|
||||
|
||||
Validation command:
|
||||
|
||||
```powershell
|
||||
python scripts\check-phase3-artifacts.py --check --evidence-root $root
|
||||
```
|
||||
|
||||
187
docs/phase3/pilot-runbook.md
Normal file
187
docs/phase3/pilot-runbook.md
Normal file
|
|
@ -0,0 +1,187 @@
|
|||
# Phase 3 Education Pilot Runbook
|
||||
|
||||
## Purpose
|
||||
|
||||
Run a 20-person education pilot with auditable evidence that each included participant
|
||||
was consented, completed the required learning sessions, produced pre/post measures,
|
||||
and was included or excluded from KPI and export outputs correctly.
|
||||
|
||||
This runbook assumes Phase 2 role separation, session persistence, supervisor review,
|
||||
and safety event logging are already deployed. It does not authorize real participant
|
||||
collection until the privacy and consent checklist is signed.
|
||||
|
||||
## Entry Gates
|
||||
|
||||
- `G-P3-01` Product scope is locked as education/training, not diagnosis or treatment.
|
||||
- `G-P3-02` Current consent form, privacy notice, retention rule, withdrawal path, and
|
||||
third-party/inference transfer disclosure are approved by the project owner and legal
|
||||
reviewer.
|
||||
- `G-P3-03` The pilot uses pseudonymous `participant_id` values only. Names, emails,
|
||||
phone numbers, student numbers, and direct identifiers are not stored in evidence
|
||||
files.
|
||||
- `G-P3-04` Safety policy is active: self-harm scenarios remain excluded unless a later
|
||||
approved protocol explicitly adds them.
|
||||
- `G-P3-05` Instructor/admin access has RBAC and audit logging enabled before any pilot
|
||||
session starts.
|
||||
- `G-P3-06` Evidence root has been created outside the source tree and the checker runs:
|
||||
|
||||
```powershell
|
||||
$root="D:\vignette-evidence\phase3-pilot-YYYYMMDD"
|
||||
python scripts\check-phase3-artifacts.py --check --evidence-root $root
|
||||
```
|
||||
|
||||
The first run is expected to fail on missing files. The run output becomes the setup
|
||||
gap list.
|
||||
|
||||
## Participant Roster Gate
|
||||
|
||||
Target: 20 active, consented trainees.
|
||||
|
||||
Evidence file: `00-intake/pilot_roster.csv`
|
||||
|
||||
Required header:
|
||||
|
||||
```csv
|
||||
participant_id,cohort_id,consent_version,consent_signed_at,withdrawal_state,enrolled_at
|
||||
```
|
||||
|
||||
Allowed `withdrawal_state` values:
|
||||
|
||||
- `active`
|
||||
- `withdrawn_before_data_use`
|
||||
- `withdrawn_after_data_use`
|
||||
- `excluded_by_operator`
|
||||
|
||||
Rules:
|
||||
|
||||
- `participant_id` must be a pseudonymous value such as `P3-001`.
|
||||
- The roster must not include names, emails, phone numbers, student IDs, addresses, or
|
||||
date of birth.
|
||||
- A separate identity map may exist only in the approved restricted location owned by
|
||||
the data steward.
|
||||
- The pilot can start when 20 participants are `active` with a non-empty
|
||||
`consent_signed_at`.
|
||||
|
||||
Evidence file: `00-intake/consent_receipts.csv`
|
||||
|
||||
Required header:
|
||||
|
||||
```csv
|
||||
participant_id,consent_version,signed_at,signer_role,receipt_id
|
||||
```
|
||||
|
||||
Operator command:
|
||||
|
||||
```powershell
|
||||
python scripts\check-phase3-artifacts.py --check --evidence-root $root
|
||||
```
|
||||
|
||||
## Session Completion Gate
|
||||
|
||||
Minimum completion definition for dashboard Phase 3:
|
||||
|
||||
- Consent receipt exists before the first session.
|
||||
- Participant has at least two completed simulation sessions.
|
||||
- Each completed session has an end timestamp, non-zero learner turns, and supervisor
|
||||
review evidence.
|
||||
- Participant has both pre and post measures.
|
||||
- Participant has SUS responses or an explicit missing-response reason in the KPI
|
||||
report.
|
||||
|
||||
Evidence file: `01-sessions/session_completion.csv`
|
||||
|
||||
Required header:
|
||||
|
||||
```csv
|
||||
participant_id,session_id,persona_id,started_at,ended_at,completion_state,turns_count,supervisor_reviewed_at
|
||||
```
|
||||
|
||||
Allowed `completion_state` values:
|
||||
|
||||
- `completed`
|
||||
- `abandoned`
|
||||
- `operator_cancelled`
|
||||
- `excluded_from_analysis`
|
||||
|
||||
Operational cadence:
|
||||
|
||||
1. Before each session, confirm the participant remains `active`.
|
||||
2. After each session, export one row to `session_completion.csv`.
|
||||
3. Supervisor reviews the persisted session and records `supervisor_reviewed_at`.
|
||||
4. Any safety event or access override is linked in the KPI report, not copied into the
|
||||
roster.
|
||||
|
||||
Session evidence query placeholder:
|
||||
|
||||
```sql
|
||||
-- Run against the approved read-only reporting connection.
|
||||
-- Replace table/column names with the deployed schema names if they differ.
|
||||
SELECT
|
||||
s.learner_id AS participant_id,
|
||||
s.id AS session_id,
|
||||
s.persona_id,
|
||||
s.started_at,
|
||||
s.ended_at,
|
||||
CASE WHEN s.ended_at IS NOT NULL THEN 'completed' ELSE 'abandoned' END AS completion_state,
|
||||
COUNT(t.id) FILTER (WHERE t.speaker = 'counselor') AS turns_count,
|
||||
MAX(r.reviewed_at) AS supervisor_reviewed_at
|
||||
FROM app.sessions s
|
||||
LEFT JOIN app.turns t ON t.session_id = s.id
|
||||
LEFT JOIN app.session_reviews r ON r.session_id = s.id
|
||||
GROUP BY s.learner_id, s.id, s.persona_id, s.started_at, s.ended_at;
|
||||
```
|
||||
|
||||
If the deployed schema does not yet have `app.session_reviews`, the operator must
|
||||
attach a signed reviewer export and note the schema gap in `02-measures/kpi_report.json`.
|
||||
|
||||
## Measure Collection Gate
|
||||
|
||||
Evidence file: `02-measures/prepost_measures.csv`
|
||||
|
||||
Required header:
|
||||
|
||||
```csv
|
||||
participant_id,measure_name,timepoint,score,collected_at
|
||||
```
|
||||
|
||||
Allowed `timepoint` values:
|
||||
|
||||
- `pre`
|
||||
- `post`
|
||||
|
||||
Evidence file: `02-measures/sus_responses.csv`
|
||||
|
||||
Required header:
|
||||
|
||||
```csv
|
||||
participant_id,item,response,collected_at
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- `measure_name` must be stable across pre and post rows.
|
||||
- SUS rows use item numbers and numeric responses only. Do not store free-text comments
|
||||
in `sus_responses.csv`.
|
||||
- Free-text participant feedback, if collected, must go through the privacy review path
|
||||
before being added to any export.
|
||||
|
||||
## Exit Gate
|
||||
|
||||
Phase 3 pilot completion evidence is sufficient when:
|
||||
|
||||
- `pilot_roster.csv` contains at least 20 active, consented participants.
|
||||
- At least 20 participants meet the minimum completion definition.
|
||||
- `kpi_report.json` is generated and reviewed.
|
||||
- `export_manifest.json` is generated, PII scan is pass or reviewed, and the export is
|
||||
approved.
|
||||
- `privacy_audit.md` is signed with no unresolved high severity item.
|
||||
- `withdrawal_log.csv` exists, even if it only contains the header.
|
||||
|
||||
Archive command:
|
||||
|
||||
```powershell
|
||||
python scripts\check-phase3-artifacts.py --check --evidence-root $root --output "$root\phase3_artifact_check.json"
|
||||
```
|
||||
|
||||
The checker writes only the report path given by `--output`; otherwise it is read-only.
|
||||
|
||||
152
docs/phase3/privacy-consent-audit.md
Normal file
152
docs/phase3/privacy-consent-audit.md
Normal file
|
|
@ -0,0 +1,152 @@
|
|||
# Phase 3 Privacy, Consent, and Withdrawal Audit
|
||||
|
||||
## Purpose
|
||||
|
||||
Provide an operator checklist for consent, withdrawal, privacy review, and evidence
|
||||
collection before and during the 20-person education pilot. This is an operational
|
||||
artifact, not legal advice.
|
||||
|
||||
Primary evidence files:
|
||||
|
||||
- `04-privacy/privacy_audit.md`
|
||||
- `04-privacy/withdrawal_log.csv`
|
||||
|
||||
## Pre-Pilot Privacy Gate
|
||||
|
||||
Copy this checklist into `04-privacy/privacy_audit.md` and complete it with reviewer
|
||||
names, dates, and evidence links.
|
||||
|
||||
```markdown
|
||||
# Phase 3 Privacy Audit
|
||||
|
||||
Pilot ID:
|
||||
Evidence root:
|
||||
Review date:
|
||||
Operator:
|
||||
Data steward:
|
||||
Legal/privacy reviewer:
|
||||
|
||||
## Scope
|
||||
- [ ] Intended use is education/training, not diagnosis, treatment, or emergency care.
|
||||
- [ ] Participant population and inclusion/exclusion criteria are approved.
|
||||
- [ ] Self-harm primary scenarios remain excluded unless separately approved.
|
||||
- [ ] Minor/source-case data use, if any, has documented lawful consent or approval.
|
||||
|
||||
## Consent
|
||||
- [ ] Consent form version is recorded.
|
||||
- [ ] Consent covers session storage, review, KPI reporting, and any dataset export use.
|
||||
- [ ] Consent explains third-party model or inference transfer, if applicable.
|
||||
- [ ] Consent explains withdrawal process and practical limits after anonymized export.
|
||||
- [ ] Consent receipt is captured before first session.
|
||||
|
||||
## Privacy Controls
|
||||
- [ ] Evidence files use pseudonymous participant IDs only.
|
||||
- [ ] Identity map location is restricted and not in the repository or export.
|
||||
- [ ] PII masking is active before external model calls.
|
||||
- [ ] Raw audio, raw source cases, and direct identifiers are excluded from Phase 3
|
||||
artifacts unless separately approved.
|
||||
- [ ] Retention and deletion schedule is documented.
|
||||
- [ ] RBAC and audit logging are enabled for instructor/admin views.
|
||||
- [ ] Export PII scan process is documented.
|
||||
|
||||
## Vendor and Transfer Review
|
||||
- [ ] Model providers and regions are listed.
|
||||
- [ ] Data sent to each provider is described at field level.
|
||||
- [ ] Secrets are not stored in evidence files.
|
||||
- [ ] Any overseas transfer, subprocessors, or institutional requirements have been
|
||||
reviewed by counsel or the designated privacy reviewer.
|
||||
|
||||
## Decision
|
||||
- [ ] Approved to start pilot.
|
||||
- [ ] Approved to generate KPI report.
|
||||
- [ ] Approved to generate anonymized export.
|
||||
- [ ] Blocked pending legal/privacy review.
|
||||
|
||||
Open issues:
|
||||
```
|
||||
|
||||
## Withdrawal Log
|
||||
|
||||
Evidence file: `04-privacy/withdrawal_log.csv`
|
||||
|
||||
Required header:
|
||||
|
||||
```csv
|
||||
participant_id,requested_at,effective_at,scope,status,attestation_path
|
||||
```
|
||||
|
||||
Allowed `scope` values:
|
||||
|
||||
- `future_sessions_only`
|
||||
- `exclude_from_analysis`
|
||||
- `exclude_from_export`
|
||||
- `delete_where_policy_allows`
|
||||
|
||||
Allowed `status` values:
|
||||
|
||||
- `received`
|
||||
- `in_progress`
|
||||
- `completed`
|
||||
- `rejected_by_policy`
|
||||
- `needs_legal_review`
|
||||
|
||||
## Withdrawal Runbook
|
||||
|
||||
1. Record the request in `withdrawal_log.csv` using only `participant_id`.
|
||||
2. Stop new sessions for that participant.
|
||||
3. Mark the roster `withdrawal_state`.
|
||||
4. Identify sessions, turns, measures, review records, and export rows in scope.
|
||||
5. Exclude the participant from future exports unless legal review states otherwise.
|
||||
6. Remove or anonymize records where the approved retention/deletion policy allows.
|
||||
7. Create an attestation note at the path listed in `attestation_path`.
|
||||
8. Re-run the artifact checker and KPI report.
|
||||
9. If an export was already produced, add a manifest limitation and create a replacement
|
||||
export if required by policy.
|
||||
|
||||
Attestation template:
|
||||
|
||||
```markdown
|
||||
# Withdrawal Attestation
|
||||
|
||||
Participant ID:
|
||||
Request received:
|
||||
Effective date:
|
||||
Scope:
|
||||
Operator:
|
||||
Reviewer:
|
||||
|
||||
Actions completed:
|
||||
- [ ] Account/session access stopped.
|
||||
- [ ] Roster updated.
|
||||
- [ ] KPI denominator/exclusion notes updated.
|
||||
- [ ] Export manifest updated or export blocked.
|
||||
- [ ] Deletion/anonymization completed where policy allows.
|
||||
|
||||
Records not changed and reason:
|
||||
|
||||
Reviewer decision:
|
||||
```
|
||||
|
||||
## Daily Operator Checklist During Pilot
|
||||
|
||||
- [ ] New sessions have consent receipts before start time.
|
||||
- [ ] No direct identifiers were added to evidence CSVs.
|
||||
- [ ] Instructor/admin access review entries are present for any manual review.
|
||||
- [ ] Withdrawal requests are processed before KPI/export generation.
|
||||
- [ ] Safety events are reviewed without copying sensitive free text into the roster.
|
||||
- [ ] Evidence root checker is run after each pilot day:
|
||||
|
||||
```powershell
|
||||
python scripts\check-phase3-artifacts.py --check --evidence-root $root
|
||||
```
|
||||
|
||||
## Exit Gate
|
||||
|
||||
The privacy gate is complete only when:
|
||||
|
||||
- `privacy_audit.md` is signed and has no unresolved high severity issue.
|
||||
- `withdrawal_log.csv` exists and all rows are terminal or explicitly marked
|
||||
`needs_legal_review`.
|
||||
- KPI and export artifacts use the same withdrawal/exclusion decisions.
|
||||
- Legal/privacy reviewer has approved real participant evidence handling.
|
||||
|
||||
Loading…
Add table
Add a link
Reference in a new issue