vignette/docs/phase3/dataset-export-manifest.md

# Phase 3 Anonymized Dataset Export Manifest

## Purpose

Define the manifest and JSONL expectations for the first recursive-learning dataset
artifact. The export is allowed only after consent scope, withdrawal state, PII masking,
and reviewer agreement are documented.

Primary evidence file: `03-export/export_manifest.json`

## Export Gate

The export may be marked `approved_for_recursive_learning_seed` only when:

- Source participants are active or otherwise allowed by consent scope at export time.
- Withdrawn participants are excluded according to the approved withdrawal policy.
- Direct identifiers and raw identity mapping are absent from exported files.
- Free text is masked or transformed according to the manifest.
- PII scan is pass, or every finding has a reviewer disposition.
- IAA gates pass for gold exports: kappa >=0.60 and ICC >=0.75.
- Legal/data-steward review is complete.

If any gate is missing, set `export_status` to `technical_dry_run` or `blocked`, not
`approved_for_recursive_learning_seed`.

## Manifest Shape

```json
{
  "export_id": "phase3-rl-seed-YYYYMMDD-001",
  "dataset_name": "vignette_phase3_recursive_learning_seed",
  "export_status": "technical_dry_run",
  "created_at": "2026-06-26T00:00:00Z",
  "purpose": "recursive-learning seed dataset for education simulator improvement",
  "source_window": {
    "started_at": "2026-00-00T00:00:00Z",
    "ended_at": "2026-00-00T00:00:00Z"
  },
  "source_tables": [
    "app.sessions",
    "app.turns",
    "app.feedback_scores",
    "ds.annotation",
    "audit.persona_drift_log"
  ],
  "selection_criteria": {
    "cohort_id": "phase3",
    "min_completed_sessions": 2,
    "include_withdrawn": false,
    "excluded_safety_scope": ["self_harm_scenario_primary"]
  },
  "consent_scope": {
    "consent_version": "",
    "allowed_uses": ["education_quality_review", "recursive_learning_seed"],
    "withdrawal_cutoff_applied_at": "",
    "participants_included": 0,
    "participants_excluded": 0
  },
  "anonymization": {
    "participant_key": "pseudonymous export key; no identity map included",
    "text_transform": "masked_text_only",
    "direct_identifier_policy": "blocked",
    "salt_or_identity_map_location": "not in export"
  },
  "pii_scan": {
    "tool": "",
    "version": "",
    "ran_at": "",
    "status": "pending",
    "findings": []
  },
  "agreement": {
    "kappa": null,
    "icc": null,
    "gold_status": "not_gold"
  },
  "files": [
    {
      "path": "03-export/anonymized_dataset.jsonl",
      "rows": 0,
      "sha256": "",
      "schema": "phase3_dataset_item_v1"
    }
  ],
  "approvals": {
    "data_steward": "",
    "legal_or_privacy_reviewer": "",
    "technical_operator": "",
    "approved_at": ""
  },
  "known_limitations": []
}
```

## JSONL Record Shape

Each JSONL line must be one dataset item:

```json
{
  "schema": "phase3_dataset_item_v1",
  "item_id": "export-local-id",
  "participant_key": "PX-0001",
  "session_key": "SX-0001",
  "turn_key": "TX-0001",
  "persona_id": "P1",
  "stage": "rapport",
  "speaker": "counselor",
  "text_masked": "masked or transformed utterance text",
  "techniques": [],
  "client_states": [],
  "feedback_scores": [],
  "supervisor_comments": [],
  "source_refs": {
    "session_started_at": "2026-00-00T00:00:00Z",
    "export_manifest_id": "phase3-rl-seed-YYYYMMDD-001"
  },
  "privacy": {
    "direct_identifiers_removed": true,
    "pii_scan_status": "pass",
    "consent_scope": "recursive_learning_seed"
  }
}
```

Blocked fields:

- name
- email
- phone
- student_id
- national_id
- address
- date_of_birth
- raw_audio_path
- raw_voice
- raw_source_case
- identity_map
- API keys, tokens, cookies, or credentials

## Operator Commands

Generate the export through the read-only dry-run exporter unless the data steward has
approved writing `ds.*` rows. The default output is `technical_dry_run`; do not use
`approved_for_recursive_learning_seed` unless consent, withdrawal, PII, legal/privacy,
and IAA gates are all complete.

Read-only dry-run export:

```powershell
$dbUrl = (Get-Content apps\api\.env -Encoding UTF8 | Where-Object { $_ -match '^DATABASE_URL=' } | Select-Object -First 1) -replace '^DATABASE_URL=', ''
python scripts\export-recursive-dataset.py --database-url $dbUrl --output-root $root --limit 1000
```

Optional dataset table write for reviewer/gold workflow:

```powershell
python scripts\export-recursive-dataset.py --database-url $dbUrl --output-root $root --write-dataset
```

Approved export is blocked unless all gates are present:

```powershell
python scripts\export-recursive-dataset.py --database-url $dbUrl --output-root $root --export-status approved_for_recursive_learning_seed --allow-approved
```

Validate artifact presence and headers:

```powershell
python scripts\check-phase3-artifacts.py --check --evidence-root $root
```

Write a validation report:

```powershell
python scripts\check-phase3-artifacts.py --check --evidence-root $root --output "$root\phase3_artifact_check.json"
```

Compute a file hash for the manifest:

```powershell
Get-FileHash "$root\03-export\anonymized_dataset.jsonl" -Algorithm SHA256
```

## Export Review Checklist

- [ ] Manifest `export_status` matches the actual approval state.
- [ ] Files listed in `files[]` exist and have row counts and SHA256 hashes.
- [ ] `participants_included` plus `participants_excluded` matches the consent roster
  decision log.
- [ ] `include_withdrawn` is false unless legal review explicitly approved otherwise.
- [ ] Text fields use masked/transformed text only.
- [ ] PII scan report is attached or referenced.
- [ ] Agreement metrics match `02-measures/kpi_report.json`.
- [ ] Export limitations state whether the file is gold data or a dry-run seed.