대시보드 폴드아웃/드릴다운 정리 + 페르소나 역린·misconduct 반응 + 게이트웨이 격리·RAG 비차단 수정

SSOT 대시보드:
- 한신대 기술분석 PDF(19쪽) 정합성 분석 + 이번 세션 발견 섹션 추가
- 섹션 폴드아웃(접기)·상단 목차(드릴다운)·모두 펼치기/접기 — 내용 보존, 레이아웃만 정리

페르소나 반응 강화('저항·반응 조절' 핵심 차별):
- PersonaCard.triggers(역린) 필드 + CCD 핵심상처 파생 역린 블록
- L0에 무례·모욕·조롱 시 현실적 동맹 균열 반응 지침

버그·성능 수정(라이브/E2E로 포착):
- 게이트웨이 페르소나 격리: --append-system-prompt를 --system-prompt(교체)로 + --exclude-dynamic-system-prompt-sections (내담자 캐릭터 붕괴·개발맥락 누출 차단)
- RAG: 임베더 동기 로드(약 7-13초)를 _warm_rag_caches 백그라운드 warm으로(세션 생성 블로킹 회귀 수정)
- voice TTS RMS 데드힌트 제거, init_state OpennessParams 파라미터객체화
- 한국어 PII(날짜·금액·주소) 마스킹 보강
- 레이아웃 시각 게이트: 폼 컨트롤 값 스크롤 오탐 제외(7/7)

검증: 백엔드 84/84, E2E 42(데스크톱 27·모바일 11·아바타 4), 시각 게이트 7/7
This commit is contained in:
Yun Chan 2026-06-27 02:30:46 +09:00
parent cb2aebd76c
commit 085460b5e0
327 changed files with 31226 additions and 1829 deletions

58
docs/phase3/README.md Normal file
View file

@ -0,0 +1,58 @@
# Phase 3 Operational Gates
This folder turns the Phase 3 dashboard items into executable operator gates for the
20-person education pilot. It is not a legal opinion; the privacy and consent gates
must be reviewed by the project data steward and counsel before real participant data
is collected.
## Artifacts
- [pilot-runbook.md](pilot-runbook.md) - roster, session completion, operator cadence,
and evidence files for the 20-person education pilot.
- [kpi-measurement-plan.md](kpi-measurement-plan.md) - KPI definitions, thresholds,
data sources, and report shape.
- [dataset-export-manifest.md](dataset-export-manifest.md) - anonymized JSONL export
and manifest expectations for recursive-learning dataset output.
- [privacy-consent-audit.md](privacy-consent-audit.md) - consent, withdrawal, privacy,
and audit checklist.
## Evidence Root
Operators should keep real pilot evidence outside the source tree by default, for
example:
```powershell
$env:PHASE3_EVIDENCE_ROOT="D:\vignette-evidence\phase3-pilot-YYYYMMDD"
python scripts\check-phase3-artifacts.py --check --evidence-root $env:PHASE3_EVIDENCE_ROOT
```
The expected evidence root layout is:
```text
00-intake/
pilot_roster.csv
consent_receipts.csv
01-sessions/
session_completion.csv
02-measures/
prepost_measures.csv
sus_responses.csv
kpi_report.json
03-export/
export_manifest.json
04-privacy/
privacy_audit.md
withdrawal_log.csv
```
No evidence file should contain names, email addresses, phone numbers, raw student IDs,
raw source case records, raw audio, or third-party API secrets. Participant identity
mapping, if required, must live in the approved restricted location owned by the data
steward, not in the export or this repository.
## Phase 3 Dashboard Coverage
- 20-person education pilot: `pilot-runbook.md`
- KPI measurement: `kpi-measurement-plan.md`
- Recursive-learning dataset export: `dataset-export-manifest.md`
- Privacy, consent, withdrawal audit: `privacy-consent-audit.md`

View file

@ -0,0 +1,175 @@
# Phase 3 Anonymized Dataset Export Manifest
## Purpose
Define the manifest and JSONL expectations for the first recursive-learning dataset
artifact. The export is allowed only after consent scope, withdrawal state, PII masking,
and reviewer agreement are documented.
Primary evidence file: `03-export/export_manifest.json`
## Export Gate
The export may be marked `approved_for_recursive_learning_seed` only when:
- Source participants are active or otherwise allowed by consent scope at export time.
- Withdrawn participants are excluded according to the approved withdrawal policy.
- Direct identifiers and raw identity mapping are absent from exported files.
- Free text is masked or transformed according to the manifest.
- PII scan is pass, or every finding has a reviewer disposition.
- IAA gates pass for gold exports: kappa >=0.60 and ICC >=0.75.
- Legal/data-steward review is complete.
If any gate is missing, set `export_status` to `technical_dry_run` or `blocked`, not
`approved_for_recursive_learning_seed`.
## Manifest Shape
```json
{
"export_id": "phase3-rl-seed-YYYYMMDD-001",
"dataset_name": "vignette_phase3_recursive_learning_seed",
"export_status": "technical_dry_run",
"created_at": "2026-06-26T00:00:00Z",
"purpose": "recursive-learning seed dataset for education simulator improvement",
"source_window": {
"started_at": "2026-00-00T00:00:00Z",
"ended_at": "2026-00-00T00:00:00Z"
},
"source_tables": [
"app.sessions",
"app.turns",
"app.feedback_scores",
"ds.annotation",
"audit.persona_drift_log"
],
"selection_criteria": {
"cohort_id": "phase3",
"min_completed_sessions": 2,
"include_withdrawn": false,
"excluded_safety_scope": ["self_harm_scenario_primary"]
},
"consent_scope": {
"consent_version": "",
"allowed_uses": ["education_quality_review", "recursive_learning_seed"],
"withdrawal_cutoff_applied_at": "",
"participants_included": 0,
"participants_excluded": 0
},
"anonymization": {
"participant_key": "pseudonymous export key; no identity map included",
"text_transform": "masked_text_only",
"direct_identifier_policy": "blocked",
"salt_or_identity_map_location": "not in export"
},
"pii_scan": {
"tool": "",
"version": "",
"ran_at": "",
"status": "pending",
"findings": []
},
"agreement": {
"kappa": null,
"icc": null,
"gold_status": "not_gold"
},
"files": [
{
"path": "03-export/anonymized_dataset.jsonl",
"rows": 0,
"sha256": "",
"schema": "phase3_dataset_item_v1"
}
],
"approvals": {
"data_steward": "",
"legal_or_privacy_reviewer": "",
"technical_operator": "",
"approved_at": ""
},
"known_limitations": []
}
```
## JSONL Record Shape
Each JSONL line must be one dataset item:
```json
{
"schema": "phase3_dataset_item_v1",
"item_id": "export-local-id",
"participant_key": "PX-0001",
"session_key": "SX-0001",
"turn_key": "TX-0001",
"persona_id": "P1",
"stage": "rapport",
"speaker": "counselor",
"text_masked": "masked or transformed utterance text",
"techniques": [],
"client_states": [],
"feedback_scores": [],
"supervisor_comments": [],
"source_refs": {
"session_started_at": "2026-00-00T00:00:00Z",
"export_manifest_id": "phase3-rl-seed-YYYYMMDD-001"
},
"privacy": {
"direct_identifiers_removed": true,
"pii_scan_status": "pass",
"consent_scope": "recursive_learning_seed"
}
}
```
Blocked fields:
- name
- email
- phone
- student_id
- national_id
- address
- date_of_birth
- raw_audio_path
- raw_voice
- raw_source_case
- identity_map
- API keys, tokens, cookies, or credentials
## Operator Commands
Generate the export only through an approved read-only reporting job. Until that job is
implemented, use the manifest as a contract and mark exports `technical_dry_run`.
Validate artifact presence and headers:
```powershell
python scripts\check-phase3-artifacts.py --check --evidence-root $root
```
Write a validation report:
```powershell
python scripts\check-phase3-artifacts.py --check --evidence-root $root --output "$root\phase3_artifact_check.json"
```
Compute a file hash for the manifest:
```powershell
Get-FileHash "$root\03-export\anonymized_dataset.jsonl" -Algorithm SHA256
```
## Export Review Checklist
- [ ] Manifest `export_status` matches the actual approval state.
- [ ] Files listed in `files[]` exist and have row counts and SHA256 hashes.
- [ ] `participants_included` plus `participants_excluded` matches the consent roster
decision log.
- [ ] `include_withdrawn` is false unless legal review explicitly approved otherwise.
- [ ] Text fields use masked/transformed text only.
- [ ] PII scan report is attached or referenced.
- [ ] Agreement metrics match `02-measures/kpi_report.json`.
- [ ] Export limitations state whether the file is gold data or a dry-run seed.

View file

@ -0,0 +1,165 @@
# Phase 3 KPI Measurement Plan
## Purpose
Define the Phase 3 KPI report so the dashboard gates are measured the same way each
time. These metrics are pilot evidence, not claims of validated clinical effectiveness.
Primary evidence file: `02-measures/kpi_report.json`
## Report Shape
```json
{
"pilot_id": "phase3-pilot-YYYYMMDD",
"generated_at": "2026-06-26T00:00:00Z",
"source_window": {
"started_at": "2026-00-00T00:00:00Z",
"ended_at": "2026-00-00T00:00:00Z"
},
"cohort_size": 20,
"metrics": {
"pilot_completion": {},
"session_completion": {},
"embedding_consistency": {},
"hallucination_rate": {},
"top1": {},
"sus": {},
"inter_rater_kappa": {},
"icc": {},
"self_efficacy_prepost": {}
},
"exclusions": [],
"open_schema_gaps": [],
"review": {
"operator": "",
"reviewed_at": "",
"decision": "pending"
}
}
```
Each metric object must include:
```json
{
"value": 0,
"threshold": 0,
"pass": false,
"numerator": null,
"denominator": null,
"method": "",
"source_files": []
}
```
## KPI Definitions
| KPI | Gate | Definition | Evidence |
| --- | --- | --- | --- |
| Pilot completion | >=20 participants | Count active participants with consent, required sessions, pre/post, SUS, and supervisor review. | `pilot_roster.csv`, `session_completion.csv`, `prepost_measures.csv`, `sus_responses.csv` |
| Session completion | >=20 participants | Count participants with at least two `completed` sessions and non-empty `supervisor_reviewed_at`. | `session_completion.csv` |
| Embedding consistency | >=0.79 | Mean cosine similarity for persona invariant checks across masked session turns. Report model, embedding version, mean, p10, and count. | DB query export, `kpi_report.json` |
| Hallucination rate | <=0.05, target <=0.01 | Unsupported or contradicted factual claims divided by reviewed eligible evaluator/client-response claims. | supervisor review export, `kpi_report.json` |
| Top-1 retrieval | >=0.80 | Top-1 evidence match rate for golden feedback or RAG evaluation queries. | golden evaluation run, `kpi_report.json` |
| SUS | >=80 | Standard 10-item SUS score transformed to a 0-100 score; report mean, median, and response count. | `sus_responses.csv` |
| Cohen kappa | >=0.60 | Agreement between human reviewers or human-vs-AI labels for categorical tags used in dataset export. | annotation export, `kpi_report.json` |
| ICC | >=0.75 | Intraclass correlation for numeric ratings such as feedback score dimensions. | annotation export, `kpi_report.json` |
| Self-efficacy pre/post | Report and review | Paired pre/post change for the approved self-efficacy measure. Report effect direction, test used, p-value, and missingness. | `prepost_measures.csv` |
## Metric Methods
### Pilot Completion
Computation:
```text
participant passes =
active consent receipt exists
AND >=2 completed sessions
AND every counted session has supervisor_reviewed_at
AND pre measure exists
AND post measure exists
AND SUS response set exists or missingness is explicitly reviewed
```
The dashboard gate passes when at least 20 participants pass. Withdrawn participants are
reported in exclusions and must not be silently removed from denominator notes.
### Embedding Consistency
Operator command placeholder:
```powershell
# Use the deployed reporting job when available. The output is copied into kpi_report.json.
python scripts\check-phase3-artifacts.py --check --evidence-root $root
```
Minimum report fields:
```json
{
"value": 0.79,
"threshold": 0.79,
"pass": true,
"method": "masked persona invariant cosine similarity",
"model": "embedding-model-name-and-version",
"p10": 0.0,
"count": 0,
"source_files": ["02-measures/persona_embedding_eval.csv"]
}
```
### Hallucination Rate
Count only reviewed claims that should be grounded in the persona card, session state,
approved knowledge base, or rubric. Unsupported clinical, safety, or personal-history
claims are high severity and must be listed in `review.exceptions`.
Formula:
```text
hallucination_rate = unsupported_or_contradicted_claims / reviewed_eligible_claims
```
### SUS
Report the standard SUS transformed 0-100 score. Do not copy item text into the
repository. Store only item number, numeric response, participant id, and timestamp.
Minimum report fields:
```json
{
"value": 80.0,
"threshold": 80.0,
"pass": true,
"mean": 80.0,
"median": 80.0,
"respondents": 20,
"source_files": ["02-measures/sus_responses.csv"]
}
```
### Inter-Rater Agreement
Use the same item universe for kappa and ICC that is later referenced by the dataset
export manifest. If agreement fails, the dataset can still be exported as a technical
artifact only when the manifest marks it `not_gold`.
## Operator Checklist
- [ ] Evidence root exists outside the source tree.
- [ ] Roster and consent receipt counts match.
- [ ] Session completion rows are exported after supervisor review.
- [ ] Pre/post and SUS files are present.
- [ ] `kpi_report.json` contains all required metric keys.
- [ ] Every failed metric has an owner, disposition, and decision date.
- [ ] Export manifest references the same cohort and exclusions as the KPI report.
Validation command:
```powershell
python scripts\check-phase3-artifacts.py --check --evidence-root $root
```

View file

@ -0,0 +1,187 @@
# Phase 3 Education Pilot Runbook
## Purpose
Run a 20-person education pilot with auditable evidence that each included participant
was consented, completed the required learning sessions, produced pre/post measures,
and was included or excluded from KPI and export outputs correctly.
This runbook assumes Phase 2 role separation, session persistence, supervisor review,
and safety event logging are already deployed. It does not authorize real participant
collection until the privacy and consent checklist is signed.
## Entry Gates
- `G-P3-01` Product scope is locked as education/training, not diagnosis or treatment.
- `G-P3-02` Current consent form, privacy notice, retention rule, withdrawal path, and
third-party/inference transfer disclosure are approved by the project owner and legal
reviewer.
- `G-P3-03` The pilot uses pseudonymous `participant_id` values only. Names, emails,
phone numbers, student numbers, and direct identifiers are not stored in evidence
files.
- `G-P3-04` Safety policy is active: self-harm scenarios remain excluded unless a later
approved protocol explicitly adds them.
- `G-P3-05` Instructor/admin access has RBAC and audit logging enabled before any pilot
session starts.
- `G-P3-06` Evidence root has been created outside the source tree and the checker runs:
```powershell
$root="D:\vignette-evidence\phase3-pilot-YYYYMMDD"
python scripts\check-phase3-artifacts.py --check --evidence-root $root
```
The first run is expected to fail on missing files. The run output becomes the setup
gap list.
## Participant Roster Gate
Target: 20 active, consented trainees.
Evidence file: `00-intake/pilot_roster.csv`
Required header:
```csv
participant_id,cohort_id,consent_version,consent_signed_at,withdrawal_state,enrolled_at
```
Allowed `withdrawal_state` values:
- `active`
- `withdrawn_before_data_use`
- `withdrawn_after_data_use`
- `excluded_by_operator`
Rules:
- `participant_id` must be a pseudonymous value such as `P3-001`.
- The roster must not include names, emails, phone numbers, student IDs, addresses, or
date of birth.
- A separate identity map may exist only in the approved restricted location owned by
the data steward.
- The pilot can start when 20 participants are `active` with a non-empty
`consent_signed_at`.
Evidence file: `00-intake/consent_receipts.csv`
Required header:
```csv
participant_id,consent_version,signed_at,signer_role,receipt_id
```
Operator command:
```powershell
python scripts\check-phase3-artifacts.py --check --evidence-root $root
```
## Session Completion Gate
Minimum completion definition for dashboard Phase 3:
- Consent receipt exists before the first session.
- Participant has at least two completed simulation sessions.
- Each completed session has an end timestamp, non-zero learner turns, and supervisor
review evidence.
- Participant has both pre and post measures.
- Participant has SUS responses or an explicit missing-response reason in the KPI
report.
Evidence file: `01-sessions/session_completion.csv`
Required header:
```csv
participant_id,session_id,persona_id,started_at,ended_at,completion_state,turns_count,supervisor_reviewed_at
```
Allowed `completion_state` values:
- `completed`
- `abandoned`
- `operator_cancelled`
- `excluded_from_analysis`
Operational cadence:
1. Before each session, confirm the participant remains `active`.
2. After each session, export one row to `session_completion.csv`.
3. Supervisor reviews the persisted session and records `supervisor_reviewed_at`.
4. Any safety event or access override is linked in the KPI report, not copied into the
roster.
Session evidence query placeholder:
```sql
-- Run against the approved read-only reporting connection.
-- Replace table/column names with the deployed schema names if they differ.
SELECT
s.learner_id AS participant_id,
s.id AS session_id,
s.persona_id,
s.started_at,
s.ended_at,
CASE WHEN s.ended_at IS NOT NULL THEN 'completed' ELSE 'abandoned' END AS completion_state,
COUNT(t.id) FILTER (WHERE t.speaker = 'counselor') AS turns_count,
MAX(r.reviewed_at) AS supervisor_reviewed_at
FROM app.sessions s
LEFT JOIN app.turns t ON t.session_id = s.id
LEFT JOIN app.session_reviews r ON r.session_id = s.id
GROUP BY s.learner_id, s.id, s.persona_id, s.started_at, s.ended_at;
```
If the deployed schema does not yet have `app.session_reviews`, the operator must
attach a signed reviewer export and note the schema gap in `02-measures/kpi_report.json`.
## Measure Collection Gate
Evidence file: `02-measures/prepost_measures.csv`
Required header:
```csv
participant_id,measure_name,timepoint,score,collected_at
```
Allowed `timepoint` values:
- `pre`
- `post`
Evidence file: `02-measures/sus_responses.csv`
Required header:
```csv
participant_id,item,response,collected_at
```
Rules:
- `measure_name` must be stable across pre and post rows.
- SUS rows use item numbers and numeric responses only. Do not store free-text comments
in `sus_responses.csv`.
- Free-text participant feedback, if collected, must go through the privacy review path
before being added to any export.
## Exit Gate
Phase 3 pilot completion evidence is sufficient when:
- `pilot_roster.csv` contains at least 20 active, consented participants.
- At least 20 participants meet the minimum completion definition.
- `kpi_report.json` is generated and reviewed.
- `export_manifest.json` is generated, PII scan is pass or reviewed, and the export is
approved.
- `privacy_audit.md` is signed with no unresolved high severity item.
- `withdrawal_log.csv` exists, even if it only contains the header.
Archive command:
```powershell
python scripts\check-phase3-artifacts.py --check --evidence-root $root --output "$root\phase3_artifact_check.json"
```
The checker writes only the report path given by `--output`; otherwise it is read-only.

View file

@ -0,0 +1,152 @@
# Phase 3 Privacy, Consent, and Withdrawal Audit
## Purpose
Provide an operator checklist for consent, withdrawal, privacy review, and evidence
collection before and during the 20-person education pilot. This is an operational
artifact, not legal advice.
Primary evidence files:
- `04-privacy/privacy_audit.md`
- `04-privacy/withdrawal_log.csv`
## Pre-Pilot Privacy Gate
Copy this checklist into `04-privacy/privacy_audit.md` and complete it with reviewer
names, dates, and evidence links.
```markdown
# Phase 3 Privacy Audit
Pilot ID:
Evidence root:
Review date:
Operator:
Data steward:
Legal/privacy reviewer:
## Scope
- [ ] Intended use is education/training, not diagnosis, treatment, or emergency care.
- [ ] Participant population and inclusion/exclusion criteria are approved.
- [ ] Self-harm primary scenarios remain excluded unless separately approved.
- [ ] Minor/source-case data use, if any, has documented lawful consent or approval.
## Consent
- [ ] Consent form version is recorded.
- [ ] Consent covers session storage, review, KPI reporting, and any dataset export use.
- [ ] Consent explains third-party model or inference transfer, if applicable.
- [ ] Consent explains withdrawal process and practical limits after anonymized export.
- [ ] Consent receipt is captured before first session.
## Privacy Controls
- [ ] Evidence files use pseudonymous participant IDs only.
- [ ] Identity map location is restricted and not in the repository or export.
- [ ] PII masking is active before external model calls.
- [ ] Raw audio, raw source cases, and direct identifiers are excluded from Phase 3
artifacts unless separately approved.
- [ ] Retention and deletion schedule is documented.
- [ ] RBAC and audit logging are enabled for instructor/admin views.
- [ ] Export PII scan process is documented.
## Vendor and Transfer Review
- [ ] Model providers and regions are listed.
- [ ] Data sent to each provider is described at field level.
- [ ] Secrets are not stored in evidence files.
- [ ] Any overseas transfer, subprocessors, or institutional requirements have been
reviewed by counsel or the designated privacy reviewer.
## Decision
- [ ] Approved to start pilot.
- [ ] Approved to generate KPI report.
- [ ] Approved to generate anonymized export.
- [ ] Blocked pending legal/privacy review.
Open issues:
```
## Withdrawal Log
Evidence file: `04-privacy/withdrawal_log.csv`
Required header:
```csv
participant_id,requested_at,effective_at,scope,status,attestation_path
```
Allowed `scope` values:
- `future_sessions_only`
- `exclude_from_analysis`
- `exclude_from_export`
- `delete_where_policy_allows`
Allowed `status` values:
- `received`
- `in_progress`
- `completed`
- `rejected_by_policy`
- `needs_legal_review`
## Withdrawal Runbook
1. Record the request in `withdrawal_log.csv` using only `participant_id`.
2. Stop new sessions for that participant.
3. Mark the roster `withdrawal_state`.
4. Identify sessions, turns, measures, review records, and export rows in scope.
5. Exclude the participant from future exports unless legal review states otherwise.
6. Remove or anonymize records where the approved retention/deletion policy allows.
7. Create an attestation note at the path listed in `attestation_path`.
8. Re-run the artifact checker and KPI report.
9. If an export was already produced, add a manifest limitation and create a replacement
export if required by policy.
Attestation template:
```markdown
# Withdrawal Attestation
Participant ID:
Request received:
Effective date:
Scope:
Operator:
Reviewer:
Actions completed:
- [ ] Account/session access stopped.
- [ ] Roster updated.
- [ ] KPI denominator/exclusion notes updated.
- [ ] Export manifest updated or export blocked.
- [ ] Deletion/anonymization completed where policy allows.
Records not changed and reason:
Reviewer decision:
```
## Daily Operator Checklist During Pilot
- [ ] New sessions have consent receipts before start time.
- [ ] No direct identifiers were added to evidence CSVs.
- [ ] Instructor/admin access review entries are present for any manual review.
- [ ] Withdrawal requests are processed before KPI/export generation.
- [ ] Safety events are reviewed without copying sensitive free text into the roster.
- [ ] Evidence root checker is run after each pilot day:
```powershell
python scripts\check-phase3-artifacts.py --check --evidence-root $root
```
## Exit Gate
The privacy gate is complete only when:
- `privacy_audit.md` is signed and has no unresolved high severity issue.
- `withdrawal_log.csv` exists and all rows are terminal or explicitly marked
`needs_legal_review`.
- KPI and export artifacts use the same withdrawal/exclusion decisions.
- Legal/privacy reviewer has approved real participant evidence handling.