관리자 워크스페이스 접근 확장과 소유자 결정 7건 확정 반영
- 관리자(role=admin)가 학습자·교수자·관리자 워크스페이스를 모두 접근하도록 can_access_role/require_role와 프론트 auth 헬퍼·Sidebar 내비를 정리하고 admin 워크스페이스 내비 E2E를 추가. - 소유자 결정 7건 전건 확정(2026-06-30)을 SSOT 대시보드·백로그에 반영하고 결정 필요 7→0으로 동기화. SSOT drift 게이트 기대 카운트도 갱신. - 확정된 H1 평가설계(κ≥0.70·ICC≥0.75·환각률≤0.03·t-검정 α=0.05·무작위 배정)를 approved-export κ 게이트(checker·dataset_export·recursive export)와 KPI 측정계획·export manifest 문서에 반영. 검증: npm run typecheck, npm run check:api-types, 백엔드 pytest 290 passed, admin 내비 E2E 1 passed, 레이아웃 시각게이트 9/9, session-layout 4 passed, SSOT drift 게이트 PASS.
This commit is contained in:
parent
1274ba9ccc
commit
4e6b0045e3
17 changed files with 110 additions and 53 deletions
|
|
@ -17,7 +17,7 @@ The export may be marked `approved_for_recursive_learning_seed` only when:
|
|||
- Direct identifiers and raw identity mapping are absent from exported files.
|
||||
- Free text is masked or transformed according to the manifest.
|
||||
- PII scan is pass, or every finding has a reviewer disposition.
|
||||
- IAA gates pass for gold exports: kappa >=0.60 and ICC >=0.75.
|
||||
- IAA gates pass for gold exports: kappa >=0.70 and ICC >=0.75. (owner-confirmed 2026-06-30)
|
||||
- Legal/data-steward review is complete.
|
||||
|
||||
If any gate is missing, set `export_status` to `technical_dry_run` or `blocked`, not
|
||||
|
|
|
|||
|
|
@ -7,6 +7,14 @@ time. These metrics are pilot evidence, not claims of validated clinical effecti
|
|||
|
||||
Primary evidence file: `02-measures/kpi_report.json`
|
||||
|
||||
## Evaluation Design (owner-confirmed 2026-06-30)
|
||||
|
||||
- **표본·배정**: 20명을 실험군/통제군으로 **무작위 배정(randomized)**. 단회기 50분, 3척도(self-efficacy / skill proficiency / training satisfaction) pre-post.
|
||||
- **통계검정**: 군 간 비교는 **독립표본 t-검정**, 유의수준 **α=0.05**. 정규성 위반 시 Mann-Whitney U로 대체 가능하나 기본은 t-검정.
|
||||
- **신뢰도 목표**: 평가자 간 **κ≥0.70**(Cohen's kappa), 수치 평정 **ICC≥0.75**.
|
||||
- **환각률 목표**: 상담 응답 환각률 **≤0.03**.
|
||||
- 위 파라미터는 설계 확정값이며, 실제 효과성 판정·통계해석은 20명 파일럿 evidence 수집 후 수행한다(파일럿 게이트).
|
||||
|
||||
## Report Shape
|
||||
|
||||
```json
|
||||
|
|
@ -60,10 +68,10 @@ Each metric object must include:
|
|||
| Pilot completion | >=20 participants | Count active participants with consent, required sessions, pre/post, SUS, and supervisor review. | `pilot_roster.csv`, `session_completion.csv`, `prepost_measures.csv`, `sus_responses.csv` |
|
||||
| Session completion | >=20 participants | Count participants with at least two `completed` sessions and non-empty `supervisor_reviewed_at`. | `session_completion.csv` |
|
||||
| Embedding consistency | >=0.79 | Mean cosine similarity for persona invariant checks across masked session turns. Report model, embedding version, mean, p10, and count. | DB query export, `kpi_report.json` |
|
||||
| Hallucination rate | <=0.05, target <=0.01 | Unsupported or contradicted factual claims divided by reviewed eligible evaluator/client-response claims. | supervisor review export, `kpi_report.json` |
|
||||
| Hallucination rate | <=0.03 (owner 2026-06-30) | Unsupported or contradicted factual claims divided by reviewed eligible evaluator/client-response claims. | supervisor review export, `kpi_report.json` |
|
||||
| Top-1 retrieval | >=0.80 | Top-1 evidence match rate for golden feedback or RAG evaluation queries. | golden evaluation run, `kpi_report.json` |
|
||||
| SUS | >=80 | Standard 10-item SUS score transformed to a 0-100 score; report mean, median, and response count. | `sus_responses.csv` |
|
||||
| Cohen kappa | >=0.60 | Agreement between human reviewers or human-vs-AI labels for categorical tags used in dataset export. | annotation export, `kpi_report.json` |
|
||||
| Cohen kappa | >=0.70 (owner 2026-06-30) | Agreement between human reviewers or human-vs-AI labels for categorical tags used in dataset export. | annotation export, `kpi_report.json` |
|
||||
| ICC | >=0.75 | Intraclass correlation for numeric ratings such as feedback score dimensions. | annotation export, `kpi_report.json` |
|
||||
| Self-efficacy pre/post | Report and review | Paired pre/post change for the approved self-efficacy measure. Report effect direction, test used, p-value, and missingness. | `prepost_measures.csv` |
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue