G8 실제 rollback 증명 종료와 비-secure origin 회기 리뷰 크래시 수정
G8 마지막 게이트인 receipt-bound 실제 image rollback을 격리 NAS vignette-preview-20260807 에서 실행해 종료했다. Gate6 계약 정정: 감사 대상 current API 이미지가 com.docker.compose.project/service/version image label 을 갖고 있어 "helper 의 compose label 0개" 계약은 감사되지 않은 다른 이미지를 쓰지 않는 한 성립하지 않는다. 계약을 key 부재가 아니라 소속(membership) 으로 바꿔 launch-nas-preview-g8-helpers.py 에 구현했다. image 상속 label 을 baseline 으로 읽고 container 의 모든 compose label 이 baseline 과 같거나 선언된 격리 override 인지 검사하며, 최종 project 는 target 이 아니고 service 는 api/web/db/proxy 가 아니어야 한다. docker run argv 에 target label 을 주입하면 fake-runner 테스트가 먼저 깨진다 (37/37). 실행 결과: - rollback-old receipt nas-g8-723eeef22eab05e63e3fafb0 -> 79ec../c530.. - restore-current receipt nas-g8-2738846cf2cf4fbe8ce0fc26 -> 52e0../6fdb.. - release gate/approval 각 2회 멱등, audit.ci_lifecycle_event rollback/executed 2, audit.ci_human_approval_event authorize_rollback 2, silent auto-promotion 0 - HMAC journal 6-record 체인 검증, health 3/3, OpenAPI 126, auth 401, Web 200 - helper 0, listener 0, 비밀 env 파기. down/volume rm/prune 미실행, 공개 런타임 미접촉 - 계획했던 Windows SSH 터널은 NAS sshd 가 direct-tcpip 를 거부해 사용할 수 없어 sshd 설정 변경 대신 같은 격리 계약의 NAS-side probe 컨테이너로 실행했다 비-secure origin 크래시 수정: 배포된 NAS 프리뷰(평문 HTTP, 비-localhost)에 회기 스펙을 돌려 24건 실패를 확인했고 원인은 하나였다. crypto.randomUUID 는 secure context 전용인데 제품 코드 18곳이 fallback 없이 호출했고 RuptureRepairCard 는 렌더 시점 호출이라 회기 리뷰 라우트 전체가 error boundary 로 떨어졌다. 릴리스 게이트 108/108 은 localhost 후보 스택에서만 돌아 이 경로를 밟은 적이 없다. src/lib/uuid.ts 의 randomUuid() 로 통일하고 fallback 도 crypto.getRandomValues 를 우선 사용해 idempotency key 의 예측 불가능성을 유지했다. 회귀는 insecure-context-uuid.spec.ts 6/6 으로 고정했다(직접 호출 0건 검사 포함). 이 수정은 아직 NAS 에 배포하지 않았다. 검증: API 898, gateway 58, executor 28, probe 11, helper launcher 37, release agent 21, ruff clean, web api-types/typecheck/build, SSOT FAIL 0, SSOT unit 5/5, dashboard E2E 10/10, 학생 폐루프 실 DB 브라우저 4/4(일회용 클론), crypto 수정 후 기존 스펙 회귀 70/70, 복원된 NAS 실제 브라우저 SSE->DB 리뷰 PASS. 부수 발견(열린 항목): 공개 API 가 engine=false 로 degraded 인데 워치독이 이를 감지하지 못한다. engine 판정이 게이트웨이 /health 의 ok 만 보고 claude readiness probe 를 돌리지 않기 때문이다. 같은 .env 와 같은 CLI 로 새 게이트웨이를 다른 포트에 띄우면 즉시 ready 이므로 상주 프로세스의 세션만 죽은 형태다. TODO A절과 대시보드에 기록했다. 이 커밋은 파일 단위로 담겼다. 위 파일들에는 이전 세션의 미커밋 G0~G8 작업이 함께 들어 있으며, hunk 를 쪼개면 대시보드/체커/TODO 정합성이 깨져 SSOT 체커가 실패한다.
This commit is contained in:
Yun Chan
parent
76d0b9ae9b
commit
93dd8f82d7
22 changed files with 10057 additions and 473 deletions
|
|
@ -6,8 +6,10 @@ from __future__ import annotations
|
|||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import shutil
|
||||
import subprocess
|
||||
from dataclasses import dataclass
|
||||
from functools import lru_cache
|
||||
from html.parser import HTMLParser
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
|
@ -18,9 +20,11 @@ DEFAULT_SOURCE_GAPS = REPO_ROOT / "docs" / "guides" / "source-docs-and-gaps.md"
|
|||
DEFAULT_BACKLOG = REPO_ROOT / "docs" / "ops" / "backlog-2026-06-26.md"
|
||||
DEFAULT_TESTING = REPO_ROOT / "docs" / "guides" / "testing.md"
|
||||
DEFAULT_LOCAL_DEVELOPMENT = REPO_ROOT / "docs" / "guides" / "local-development.md"
|
||||
DEFAULT_TODO = REPO_ROOT / "docs" / "TODO.md"
|
||||
|
||||
EXPECTED_STATUS_COUNTS = {
|
||||
"done": 25,
|
||||
"done": 33,
|
||||
"doing": 2,
|
||||
"planned": 0,
|
||||
}
|
||||
EXPECTED_OWNER_COLUMN_COUNTS = {
|
||||
|
|
@ -31,6 +35,9 @@ EXPECTED_OWNER_COLUMN_COUNTS = {
|
|||
EXPECTED_CRITICAL_CARD_COUNT = 3
|
||||
EXPECTED_DECISION_DETAIL_ROWS = 0
|
||||
EXPECTED_EXTERNAL_GATE_ROWS = 6
|
||||
EXPECTED_OUTCOME_GATES = tuple(f"G{index}" for index in range(9))
|
||||
EXPECTED_AOS_IDS = tuple(f"AOS-{index:03d}" for index in range(1, 13))
|
||||
PLAYWRIGHT_ROOT = REPO_ROOT / "apps" / "web"
|
||||
|
||||
REQUIRED_SHARED_PHRASES = (
|
||||
"M2 digest worker + memory focused 30 passed",
|
||||
|
|
@ -43,7 +50,7 @@ REQUIRED_SHARED_PHRASES = (
|
|||
"owner 결정 7건은 2026-06-30 전건 확정",
|
||||
"결정 0개(전건 확정)",
|
||||
"MASTERPLAN_REVISIONS F-10 승인",
|
||||
"Public API <b>recovered OK</b>",
|
||||
"Public API <b>119 paths · G1~G8</b>",
|
||||
"error code 1033",
|
||||
"Docker Desktop/DB 재기동",
|
||||
"watchdog healthy: engine, api, web-preview, cloudflared, public-api",
|
||||
|
|
@ -84,10 +91,48 @@ class Check:
|
|||
detail: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PlaywrightInventory:
|
||||
tests: int
|
||||
files: int
|
||||
|
||||
|
||||
def _parse_playwright_inventory(output: str) -> PlaywrightInventory:
|
||||
match = re.search(r"^Total:\s+(\d+) tests in (\d+) files$", output, re.MULTILINE)
|
||||
if match is None:
|
||||
raise ValueError("playwright --list summary is missing")
|
||||
return PlaywrightInventory(tests=int(match.group(1)), files=int(match.group(2)))
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _collect_playwright_inventory() -> PlaywrightInventory:
|
||||
executable = shutil.which("npx.cmd") or shutil.which("npx")
|
||||
if executable is None:
|
||||
raise RuntimeError("npx is unavailable for Playwright inventory")
|
||||
completed = subprocess.run(
|
||||
[executable, "playwright", "test", "--list"],
|
||||
cwd=PLAYWRIGHT_ROOT,
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
encoding="utf-8",
|
||||
errors="replace",
|
||||
timeout=120,
|
||||
)
|
||||
if completed.returncode != 0:
|
||||
detail = (completed.stderr or completed.stdout).strip()
|
||||
raise RuntimeError(
|
||||
f"playwright --list failed ({completed.returncode}): {detail[-500:]}"
|
||||
)
|
||||
return _parse_playwright_inventory(completed.stdout)
|
||||
|
||||
|
||||
class DashboardStructureParser(HTMLParser):
|
||||
def __init__(self) -> None:
|
||||
super().__init__(convert_charrefs=True)
|
||||
self.status_counts: dict[str, int] = {}
|
||||
self.status_counts: dict[str, int] = {
|
||||
status: 0 for status in EXPECTED_STATUS_COUNTS
|
||||
}
|
||||
self.owner_column_counts: dict[str, int] = {}
|
||||
self.decision_panel_rows = 0
|
||||
self.external_gate_rows = 0
|
||||
|
|
@ -164,15 +209,75 @@ def _has_check_status_markup(dashboard: str) -> bool:
|
|||
)
|
||||
|
||||
|
||||
def run_checks(paths: argparse.Namespace) -> dict[str, Any]:
|
||||
def run_checks(
|
||||
paths: argparse.Namespace,
|
||||
*,
|
||||
playwright_inventory: PlaywrightInventory | None = None,
|
||||
) -> dict[str, Any]:
|
||||
dashboard = _read(paths.dashboard)
|
||||
source_gaps = _read(paths.source_gaps)
|
||||
backlog = _read(paths.backlog)
|
||||
testing = _read(paths.testing)
|
||||
local_development = _read(paths.local_development)
|
||||
todo = _read(paths.todo)
|
||||
joined_docs = "\n".join([dashboard, source_gaps, backlog, testing, local_development])
|
||||
|
||||
checks: list[Check] = []
|
||||
inventory_error: str | None = None
|
||||
if playwright_inventory is None:
|
||||
try:
|
||||
playwright_inventory = _collect_playwright_inventory()
|
||||
except (OSError, RuntimeError, ValueError, subprocess.SubprocessError) as exc:
|
||||
inventory_error = str(exc)
|
||||
if playwright_inventory is None:
|
||||
checks.append(
|
||||
Check(
|
||||
name="playwright_inventory_collected",
|
||||
passed=False,
|
||||
detail=inventory_error or "unknown inventory failure",
|
||||
)
|
||||
)
|
||||
else:
|
||||
inventory_phrase = (
|
||||
f"{playwright_inventory.tests} tests / {playwright_inventory.files} files"
|
||||
)
|
||||
checks.append(
|
||||
Check(
|
||||
name="playwright_inventory_collected",
|
||||
passed=True,
|
||||
detail=inventory_phrase,
|
||||
)
|
||||
)
|
||||
checks.append(
|
||||
Check(
|
||||
name="playwright_inventory_dashboard_current",
|
||||
passed=inventory_phrase in dashboard,
|
||||
detail=(
|
||||
f"expected current inventory in dashboard: {inventory_phrase}"
|
||||
),
|
||||
)
|
||||
)
|
||||
checks.append(
|
||||
Check(
|
||||
name="playwright_inventory_testing_guide_current",
|
||||
passed=inventory_phrase in testing,
|
||||
detail=(
|
||||
f"expected current inventory in testing guide: {inventory_phrase}"
|
||||
),
|
||||
)
|
||||
)
|
||||
stale_inventory_claims = (
|
||||
"현재 수집 기준 215 tests / 20 files",
|
||||
"**총 215 tests / 20 files**",
|
||||
)
|
||||
for phrase in stale_inventory_claims:
|
||||
checks.append(
|
||||
Check(
|
||||
name=f"stale_playwright_inventory::{phrase}",
|
||||
passed=phrase not in joined_docs,
|
||||
detail="absent" if phrase not in joined_docs else "present",
|
||||
)
|
||||
)
|
||||
structure = _dashboard_structure(dashboard)
|
||||
counts = structure.status_counts
|
||||
for status, expected in EXPECTED_STATUS_COUNTS.items():
|
||||
|
|
@ -184,12 +289,11 @@ def run_checks(paths: argparse.Namespace) -> dict[str, Any]:
|
|||
detail=f"expected={expected} actual={actual}",
|
||||
)
|
||||
)
|
||||
doing_count = counts.get("doing", 0)
|
||||
checks.append(
|
||||
Check(
|
||||
name="no_doing_cards",
|
||||
passed=doing_count == 0 and not _has_check_status_markup(dashboard),
|
||||
detail=f"doing={doing_count}",
|
||||
name="no_legacy_check_status_markup",
|
||||
passed=not _has_check_status_markup(dashboard),
|
||||
detail="task-status s-check/CHECK markup absent",
|
||||
)
|
||||
)
|
||||
|
||||
|
|
@ -202,6 +306,36 @@ def run_checks(paths: argparse.Namespace) -> dict[str, Any]:
|
|||
)
|
||||
)
|
||||
|
||||
outcome_gates = tuple(
|
||||
re.findall(
|
||||
r'<span class="chip c-(?:plan|doing|done)">(G[0-8]) (?:GATE|BUILD|DONE)(?: · [^<]+)?</span>',
|
||||
dashboard,
|
||||
)
|
||||
)
|
||||
checks.append(
|
||||
Check(
|
||||
name="outcome_alliance_gate_cards_complete",
|
||||
passed=outcome_gates == EXPECTED_OUTCOME_GATES,
|
||||
detail=f"expected={EXPECTED_OUTCOME_GATES} actual={outcome_gates}",
|
||||
)
|
||||
)
|
||||
todo_gate_headings = tuple(re.findall(r"^### I-[0-8]\. (G[0-8])\b", todo, re.MULTILINE))
|
||||
checks.append(
|
||||
Check(
|
||||
name="outcome_alliance_todo_gates_complete",
|
||||
passed=todo_gate_headings == EXPECTED_OUTCOME_GATES,
|
||||
detail=f"expected={EXPECTED_OUTCOME_GATES} actual={todo_gate_headings}",
|
||||
)
|
||||
)
|
||||
todo_aos_ids = tuple(re.findall(r"\*\*(AOS-\d{3})\*\*", todo))
|
||||
checks.append(
|
||||
Check(
|
||||
name="outcome_alliance_first_dag_complete",
|
||||
passed=todo_aos_ids == EXPECTED_AOS_IDS,
|
||||
detail=f"expected={EXPECTED_AOS_IDS} actual={todo_aos_ids}",
|
||||
)
|
||||
)
|
||||
|
||||
for owner_column, expected in EXPECTED_OWNER_COLUMN_COUNTS.items():
|
||||
actual = structure.owner_column_counts.get(owner_column, 0)
|
||||
checks.append(
|
||||
|
|
@ -279,6 +413,7 @@ def build_parser() -> argparse.ArgumentParser:
|
|||
parser.add_argument("--backlog", type=Path, default=DEFAULT_BACKLOG)
|
||||
parser.add_argument("--testing", type=Path, default=DEFAULT_TESTING)
|
||||
parser.add_argument("--local-development", type=Path, default=DEFAULT_LOCAL_DEVELOPMENT)
|
||||
parser.add_argument("--todo", type=Path, default=DEFAULT_TODO)
|
||||
parser.add_argument("--json", action="store_true", help="Emit machine-readable JSON.")
|
||||
return parser
|
||||
|
||||
|
|
|
|||
718
scripts/launch-nas-preview-g8-helpers.py
Normal file
718
scripts/launch-nas-preview-g8-helpers.py
Normal file
|
|
@ -0,0 +1,718 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Launch and verify the isolated G8 rollback helper containers on the NAS.
|
||||
|
||||
The exact current preview API image carries ``com.docker.compose.*`` *image*
|
||||
labels (``project=vignette-preview-20260807``, ``service=api``,
|
||||
``version=2.20.1``). Every container started from that image inherits them, so
|
||||
the earlier "helper must expose zero ``com.docker.compose.*`` keys" rule can
|
||||
never hold while the audited runtime image is reused. Requiring zero keys would
|
||||
only push an operator toward a different, unaudited image.
|
||||
|
||||
The enforced contract is therefore *membership*, not key absence:
|
||||
|
||||
1. The helper must not belong to the target Compose project.
|
||||
2. Image-inherited labels are the baseline. Every ``com.docker.compose.*`` key
|
||||
on the helper container must either equal that baseline or be one of the
|
||||
explicitly declared isolated overrides.
|
||||
3. The effective ``com.docker.compose.project`` must never be the target project.
|
||||
4. The effective ``com.docker.compose.service`` must never be ``api``, ``web``,
|
||||
``db`` or ``proxy``.
|
||||
5. The ``docker run`` argv must never carry a target project/service label.
|
||||
6. The exact container name, container id and ``docker inspect`` labels are
|
||||
recorded as evidence.
|
||||
|
||||
Docker is always invoked with argument arrays through an injectable runner, so
|
||||
the argv contract is pinned by fake-runner tests instead of by live execution.
|
||||
No command in this module stops, removes, recreates or prunes anything that
|
||||
belongs to the target Compose project.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import PurePosixPath
|
||||
from typing import Any, Iterable, Protocol
|
||||
|
||||
|
||||
TARGET_PROJECT = "vignette-preview-20260807"
|
||||
TARGET_ROOT = PurePosixPath("/volume1/docker/vignette-preview-20260807")
|
||||
TARGET_SERVICES = ("api", "web", "db", "proxy")
|
||||
TARGET_DOCKER_SOCKET = "/var/run/docker.sock"
|
||||
|
||||
STATE_DIR = TARGET_ROOT / ".rollback-executor"
|
||||
MANIFEST_PATH = STATE_DIR / "rollback-manifest.json"
|
||||
EXECUTOR_SCRIPT = STATE_DIR / "bin" / "serve-nas-preview-rollback-executor.py"
|
||||
|
||||
COMPOSE_LABEL_PREFIX = "com.docker.compose."
|
||||
PROJECT_LABEL = f"{COMPOSE_LABEL_PREFIX}project"
|
||||
SERVICE_LABEL = f"{COMPOSE_LABEL_PREFIX}service"
|
||||
HELPER_LABEL = "net.chanpaca.vignette.g8-helper"
|
||||
HELPER_RUN_LABEL = "net.chanpaca.vignette.g8-helper-run"
|
||||
|
||||
HELPER_PROJECT_PREFIX = "vignette-g8-helper-"
|
||||
HELPER_ROLES = {
|
||||
"executor": "g8-rollback-executor",
|
||||
"control-plane": "g8-rollback-control-plane",
|
||||
"probe": "g8-rollback-probe",
|
||||
}
|
||||
|
||||
DOCKER_BINARY_SOURCE = "/volume1/@appstore/ContainerManager/usr/bin/docker"
|
||||
DOCKER_BINARY_TARGET = "/usr/local/bin/docker"
|
||||
COMPOSE_PLUGIN_SOURCE = "/volume1/@appstore/ContainerManager/usr/bin/docker-compose"
|
||||
COMPOSE_PLUGIN_TARGET = "/usr/local/lib/docker/cli-plugins/docker-compose"
|
||||
|
||||
HELPER_UID = 1028
|
||||
HELPER_GID = 100
|
||||
HELPER_SUPPLEMENTAL_GID = 101
|
||||
|
||||
EXECUTOR_HOST = "127.0.0.1"
|
||||
EXECUTOR_PORT = 18149
|
||||
CONTROL_PLANE_HOST = "127.0.0.1"
|
||||
CONTROL_PLANE_PORT = 8018
|
||||
CONTROL_PLANE_BASE_URL = f"http://{CONTROL_PLANE_HOST}:{CONTROL_PLANE_PORT}"
|
||||
PREVIEW_BASE_URL = "http://127.0.0.1:8088"
|
||||
PROBE_SCRIPT = STATE_DIR / "bin" / "probe-nas-preview-g8-rollback.py"
|
||||
EVIDENCE_DIR = STATE_DIR / "evidence"
|
||||
PLANS_DIR = STATE_DIR / "plans"
|
||||
PLAN_NAME_RE = re.compile(r"^[a-z][a-z0-9-]{2,63}$")
|
||||
|
||||
RUN_ID_RE = re.compile(r"^[a-z0-9][a-z0-9-]{5,31}$")
|
||||
IMAGE_RE = re.compile(r"^sha256:[a-f0-9]{64}$")
|
||||
SHA256_RE = re.compile(r"^[a-f0-9]{64}$")
|
||||
CONTAINER_ID_RE = re.compile(r"^[a-f0-9]{64}$")
|
||||
EVIDENCE_SCHEMA = "vignette.nas-preview-g8-helper-isolation.v1"
|
||||
|
||||
|
||||
class HelperError(RuntimeError):
|
||||
"""Fail-closed error carrying a stable, non-sensitive code."""
|
||||
|
||||
def __init__(self, code: str):
|
||||
self.code = code
|
||||
super().__init__(code)
|
||||
|
||||
|
||||
class CommandRunner(Protocol):
|
||||
def run(
|
||||
self, operation: str, argv: list[str], *, timeout: float
|
||||
) -> str: ...
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Mount:
|
||||
source: str
|
||||
target: str
|
||||
read_only: bool = True
|
||||
|
||||
def as_argument(self) -> str:
|
||||
parts = [
|
||||
"type=bind",
|
||||
f"source={self.source}",
|
||||
f"target={self.target}",
|
||||
]
|
||||
if self.read_only:
|
||||
parts.append("readonly")
|
||||
return ",".join(parts)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class HelperSpec:
|
||||
"""One isolated helper container derived from the audited runtime image."""
|
||||
|
||||
role: str
|
||||
run_id: str
|
||||
image: str
|
||||
env_file: str
|
||||
entrypoint: str
|
||||
command: tuple[str, ...]
|
||||
mounts: tuple[Mount, ...] = ()
|
||||
tmpfs: tuple[str, ...] = ()
|
||||
supplemental_groups: tuple[int, ...] = ()
|
||||
read_only_rootfs: bool = True
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if self.role not in HELPER_ROLES:
|
||||
raise HelperError("helper_role_unknown")
|
||||
if RUN_ID_RE.fullmatch(self.run_id) is None:
|
||||
raise HelperError("helper_run_id_invalid")
|
||||
if IMAGE_RE.fullmatch(self.image) is None:
|
||||
raise HelperError("helper_image_invalid")
|
||||
if not self.env_file.startswith(f"{STATE_DIR}/"):
|
||||
raise HelperError("helper_env_file_outside_state_dir")
|
||||
if not self.command:
|
||||
raise HelperError("helper_command_required")
|
||||
|
||||
@property
|
||||
def isolated_project(self) -> str:
|
||||
return f"{HELPER_PROJECT_PREFIX}{self.run_id}"
|
||||
|
||||
@property
|
||||
def isolated_service(self) -> str:
|
||||
return HELPER_ROLES[self.role]
|
||||
|
||||
@property
|
||||
def container_name(self) -> str:
|
||||
return f"{self.isolated_service}-{self.run_id}"
|
||||
|
||||
@property
|
||||
def isolated_labels(self) -> dict[str, str]:
|
||||
"""Compose labels this launcher deliberately overrides on the helper."""
|
||||
|
||||
return {
|
||||
PROJECT_LABEL: self.isolated_project,
|
||||
SERVICE_LABEL: self.isolated_service,
|
||||
}
|
||||
|
||||
@property
|
||||
def provenance_labels(self) -> dict[str, str]:
|
||||
return {
|
||||
HELPER_LABEL: self.role,
|
||||
HELPER_RUN_LABEL: self.run_id,
|
||||
}
|
||||
|
||||
|
||||
def executor_spec(*, run_id: str, image: str, manifest_sha256: str) -> HelperSpec:
|
||||
if SHA256_RE.fullmatch(manifest_sha256) is None:
|
||||
raise HelperError("manifest_sha256_invalid")
|
||||
return HelperSpec(
|
||||
role="executor",
|
||||
run_id=run_id,
|
||||
image=image,
|
||||
env_file=str(STATE_DIR / f"executor-{run_id}.env"),
|
||||
entrypoint="python3.11",
|
||||
command=(
|
||||
"-B",
|
||||
str(EXECUTOR_SCRIPT),
|
||||
"--manifest",
|
||||
str(MANIFEST_PATH),
|
||||
"--manifest-sha256",
|
||||
manifest_sha256,
|
||||
"--host",
|
||||
EXECUTOR_HOST,
|
||||
"--port",
|
||||
str(EXECUTOR_PORT),
|
||||
"--enable",
|
||||
),
|
||||
mounts=(
|
||||
Mount(str(TARGET_ROOT), str(TARGET_ROOT), read_only=True),
|
||||
Mount(str(STATE_DIR), str(STATE_DIR), read_only=False),
|
||||
Mount(DOCKER_BINARY_SOURCE, DOCKER_BINARY_TARGET, read_only=True),
|
||||
Mount(COMPOSE_PLUGIN_SOURCE, COMPOSE_PLUGIN_TARGET, read_only=True),
|
||||
Mount(TARGET_DOCKER_SOCKET, TARGET_DOCKER_SOCKET, read_only=False),
|
||||
),
|
||||
tmpfs=("/tmp:rw,nosuid,nodev,size=16m",),
|
||||
supplemental_groups=(HELPER_SUPPLEMENTAL_GID,),
|
||||
)
|
||||
|
||||
|
||||
def control_plane_spec(*, run_id: str, image: str) -> HelperSpec:
|
||||
return HelperSpec(
|
||||
role="control-plane",
|
||||
run_id=run_id,
|
||||
image=image,
|
||||
env_file=str(STATE_DIR / f"control-plane-{run_id}.env"),
|
||||
entrypoint="uvicorn",
|
||||
command=(
|
||||
"app.main:app",
|
||||
"--host",
|
||||
CONTROL_PLANE_HOST,
|
||||
"--port",
|
||||
str(CONTROL_PLANE_PORT),
|
||||
"--workers",
|
||||
"1",
|
||||
),
|
||||
tmpfs=(
|
||||
"/tmp:rw,nosuid,nodev,size=16m",
|
||||
"/app/uploads:rw,nosuid,nodev,size=16m",
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def probe_spec(*, run_id: str, image: str, plan_name: str) -> HelperSpec:
|
||||
"""One-shot probe client. The NAS sshd forbids TCP forwarding, so the
|
||||
|
||||
control-plane probe runs from an isolated NAS-side container on loopback
|
||||
instead of through a Windows SSH tunnel. It never mounts the Docker socket.
|
||||
"""
|
||||
|
||||
if PLAN_NAME_RE.fullmatch(plan_name) is None:
|
||||
raise HelperError("probe_plan_name_invalid")
|
||||
return HelperSpec(
|
||||
role="probe",
|
||||
run_id=run_id,
|
||||
image=image,
|
||||
env_file=str(STATE_DIR / f"probe-{run_id}.env"),
|
||||
entrypoint="python3.11",
|
||||
command=(
|
||||
"-B",
|
||||
str(PROBE_SCRIPT),
|
||||
"execute",
|
||||
"--plan",
|
||||
str(PLANS_DIR / f"{plan_name}.plan.json"),
|
||||
"--control-plane-base-url",
|
||||
CONTROL_PLANE_BASE_URL,
|
||||
"--preview-base-url",
|
||||
PREVIEW_BASE_URL,
|
||||
"--timeout",
|
||||
"180",
|
||||
"--output",
|
||||
str(EVIDENCE_DIR / f"{plan_name}-{run_id}.result.json"),
|
||||
),
|
||||
mounts=(
|
||||
Mount(str(TARGET_ROOT), str(TARGET_ROOT), read_only=True),
|
||||
Mount(str(EVIDENCE_DIR), str(EVIDENCE_DIR), read_only=False),
|
||||
),
|
||||
tmpfs=("/tmp:rw,nosuid,nodev,size=16m",),
|
||||
)
|
||||
|
||||
|
||||
def build_run_argv(spec: HelperSpec, *, docker_bin: str = "docker") -> list[str]:
|
||||
"""Exact ``docker run`` argv for one isolated helper container."""
|
||||
|
||||
argv = [
|
||||
docker_bin,
|
||||
"run",
|
||||
"--detach",
|
||||
"--name",
|
||||
spec.container_name,
|
||||
"--restart",
|
||||
"no",
|
||||
"--network",
|
||||
"host",
|
||||
"--user",
|
||||
f"{HELPER_UID}:{HELPER_GID}",
|
||||
"--cap-drop",
|
||||
"ALL",
|
||||
"--security-opt",
|
||||
"no-new-privileges",
|
||||
"--pids-limit",
|
||||
"256",
|
||||
"--env-file",
|
||||
spec.env_file,
|
||||
]
|
||||
for group in spec.supplemental_groups:
|
||||
argv += ["--group-add", str(group)]
|
||||
if spec.read_only_rootfs:
|
||||
argv.append("--read-only")
|
||||
for entry in spec.tmpfs:
|
||||
argv += ["--tmpfs", entry]
|
||||
for mount in spec.mounts:
|
||||
argv += ["--mount", mount.as_argument()]
|
||||
for key, value in sorted(
|
||||
{**spec.isolated_labels, **spec.provenance_labels}.items()
|
||||
):
|
||||
argv += ["--label", f"{key}={value}"]
|
||||
argv += ["--entrypoint", spec.entrypoint, spec.image, *spec.command]
|
||||
assert_argv_isolation(argv, spec)
|
||||
return argv
|
||||
|
||||
|
||||
def assert_argv_isolation(argv: Iterable[str], spec: HelperSpec) -> None:
|
||||
"""Reject any argv that would enlist the helper into the target project."""
|
||||
|
||||
items = list(argv)
|
||||
for index, item in enumerate(items):
|
||||
if item != "--label":
|
||||
continue
|
||||
if index + 1 >= len(items):
|
||||
raise HelperError("helper_label_argument_missing")
|
||||
key, separator, value = items[index + 1].partition("=")
|
||||
if not separator:
|
||||
raise HelperError("helper_label_argument_missing")
|
||||
if key == PROJECT_LABEL and value != spec.isolated_project:
|
||||
raise HelperError("helper_argv_targets_project")
|
||||
if key == SERVICE_LABEL and value != spec.isolated_service:
|
||||
raise HelperError("helper_argv_targets_service")
|
||||
if not key.startswith(COMPOSE_LABEL_PREFIX):
|
||||
continue
|
||||
if key not in spec.isolated_labels:
|
||||
raise HelperError("helper_argv_undeclared_compose_label")
|
||||
for forbidden in ("--label-file", "--network-alias"):
|
||||
if forbidden in items:
|
||||
raise HelperError("helper_argv_forbidden_flag")
|
||||
if any(item.startswith("--network=") or item == "--network" for item in items):
|
||||
network_values = [
|
||||
items[index + 1]
|
||||
for index, item in enumerate(items)
|
||||
if item == "--network" and index + 1 < len(items)
|
||||
]
|
||||
network_values += [
|
||||
item.split("=", 1)[1] for item in items if item.startswith("--network=")
|
||||
]
|
||||
if any(value != "host" for value in network_values):
|
||||
raise HelperError("helper_argv_network_not_isolated")
|
||||
|
||||
|
||||
def verify_container_isolation(
|
||||
*,
|
||||
spec: HelperSpec,
|
||||
baseline_labels: dict[str, str],
|
||||
container_labels: dict[str, str],
|
||||
target_project_members: Iterable[str],
|
||||
) -> dict[str, Any]:
|
||||
"""Check the helper against image baseline and target-project membership."""
|
||||
|
||||
if not isinstance(baseline_labels, dict) or not isinstance(
|
||||
container_labels, dict
|
||||
):
|
||||
raise HelperError("helper_labels_unreadable")
|
||||
inherited: dict[str, str] = {}
|
||||
overridden: dict[str, str] = {}
|
||||
for key, value in sorted(container_labels.items()):
|
||||
if not key.startswith(COMPOSE_LABEL_PREFIX):
|
||||
continue
|
||||
if key in spec.isolated_labels:
|
||||
if value != spec.isolated_labels[key]:
|
||||
raise HelperError("helper_isolated_label_mismatch")
|
||||
overridden[key] = value
|
||||
continue
|
||||
if baseline_labels.get(key) != value:
|
||||
raise HelperError("helper_compose_label_drift")
|
||||
inherited[key] = value
|
||||
if set(overridden) != set(spec.isolated_labels):
|
||||
raise HelperError("helper_isolated_label_missing")
|
||||
if container_labels.get(PROJECT_LABEL) == TARGET_PROJECT:
|
||||
raise HelperError("helper_joined_target_project")
|
||||
if container_labels.get(SERVICE_LABEL) in TARGET_SERVICES:
|
||||
raise HelperError("helper_claimed_target_service")
|
||||
if container_labels.get(HELPER_LABEL) != spec.role:
|
||||
raise HelperError("helper_provenance_label_missing")
|
||||
members = list(target_project_members)
|
||||
if any(not isinstance(name, str) for name in members):
|
||||
raise HelperError("helper_membership_unreadable")
|
||||
if spec.container_name in members:
|
||||
raise HelperError("helper_listed_in_target_project")
|
||||
return {
|
||||
"baseline_compose_labels": {
|
||||
key: value
|
||||
for key, value in sorted(baseline_labels.items())
|
||||
if key.startswith(COMPOSE_LABEL_PREFIX)
|
||||
},
|
||||
"inherited_compose_labels": inherited,
|
||||
"isolated_override_labels": overridden,
|
||||
"target_project_members": sorted(members),
|
||||
}
|
||||
|
||||
|
||||
class SubprocessRunner:
|
||||
"""Run one fixed argv without echoing arguments or command output."""
|
||||
|
||||
def run(self, operation: str, argv: list[str], *, timeout: float) -> str:
|
||||
if (
|
||||
not isinstance(argv, list)
|
||||
or not argv
|
||||
or any(not isinstance(item, str) for item in argv)
|
||||
):
|
||||
raise HelperError("helper_argv_invalid")
|
||||
try:
|
||||
completed = subprocess.run(
|
||||
argv,
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
encoding="utf-8",
|
||||
errors="replace",
|
||||
timeout=timeout,
|
||||
shell=False,
|
||||
)
|
||||
except (OSError, subprocess.TimeoutExpired) as exc:
|
||||
raise HelperError(f"command_failed_{operation}") from exc
|
||||
if completed.returncode != 0:
|
||||
raise HelperError(f"command_failed_{operation}")
|
||||
return completed.stdout
|
||||
|
||||
|
||||
@dataclass
|
||||
class SshCommandRunner:
|
||||
"""Send one argv to the NAS over SSH with explicit POSIX quoting."""
|
||||
|
||||
destination: str
|
||||
identity_file: str
|
||||
inner: CommandRunner = field(default_factory=SubprocessRunner)
|
||||
|
||||
def run(self, operation: str, argv: list[str], *, timeout: float) -> str:
|
||||
remote = " ".join(shlex.quote(item) for item in argv)
|
||||
ssh_argv = [
|
||||
"ssh",
|
||||
"-i",
|
||||
self.identity_file,
|
||||
"-o",
|
||||
"IdentitiesOnly=yes",
|
||||
"-o",
|
||||
"BatchMode=yes",
|
||||
"-o",
|
||||
"StrictHostKeyChecking=accept-new",
|
||||
self.destination,
|
||||
remote,
|
||||
]
|
||||
return self.inner.run(operation, ssh_argv, timeout=timeout)
|
||||
|
||||
|
||||
class HelperLauncher:
|
||||
"""Start, verify and stop isolated helper containers through one runner."""
|
||||
|
||||
def __init__(self, runner: CommandRunner, *, docker_bin: str = "docker"):
|
||||
self.runner = runner
|
||||
self.docker_bin = docker_bin
|
||||
|
||||
def image_labels(self, image: str) -> dict[str, str]:
|
||||
if IMAGE_RE.fullmatch(image) is None:
|
||||
raise HelperError("helper_image_invalid")
|
||||
raw = self.runner.run(
|
||||
"inspect_image_labels",
|
||||
[
|
||||
self.docker_bin,
|
||||
"image",
|
||||
"inspect",
|
||||
"--format={{json .Config.Labels}}",
|
||||
image,
|
||||
],
|
||||
timeout=30,
|
||||
)
|
||||
return _parse_labels(raw)
|
||||
|
||||
def container_labels(self, container: str) -> dict[str, str]:
|
||||
raw = self.runner.run(
|
||||
"inspect_container_labels",
|
||||
[
|
||||
self.docker_bin,
|
||||
"inspect",
|
||||
"--format={{json .Config.Labels}}",
|
||||
container,
|
||||
],
|
||||
timeout=30,
|
||||
)
|
||||
return _parse_labels(raw)
|
||||
|
||||
def container_id(self, container: str) -> str:
|
||||
raw = self.runner.run(
|
||||
"inspect_container_id",
|
||||
[self.docker_bin, "inspect", "--format={{.Id}}", container],
|
||||
timeout=30,
|
||||
).strip()
|
||||
if CONTAINER_ID_RE.fullmatch(raw) is None:
|
||||
raise HelperError("helper_container_id_invalid")
|
||||
return raw
|
||||
|
||||
def container_exit_code(self, container: str) -> int:
|
||||
raw = self.runner.run(
|
||||
"inspect_container_exit_code",
|
||||
[self.docker_bin, "inspect", "--format={{.State.ExitCode}}", container],
|
||||
timeout=30,
|
||||
).strip()
|
||||
try:
|
||||
return int(raw)
|
||||
except ValueError as exc:
|
||||
raise HelperError("helper_exit_code_invalid") from exc
|
||||
|
||||
def container_running(self, container: str) -> bool:
|
||||
raw = self.runner.run(
|
||||
"inspect_container_state",
|
||||
[self.docker_bin, "inspect", "--format={{.State.Running}}", container],
|
||||
timeout=30,
|
||||
).strip()
|
||||
return raw == "true"
|
||||
|
||||
def target_project_members(self) -> list[str]:
|
||||
raw = self.runner.run(
|
||||
"list_target_project_members",
|
||||
[
|
||||
self.docker_bin,
|
||||
"ps",
|
||||
"--all",
|
||||
"--filter",
|
||||
f"label={PROJECT_LABEL}={TARGET_PROJECT}",
|
||||
"--format={{.Names}}",
|
||||
],
|
||||
timeout=30,
|
||||
)
|
||||
return [line.strip() for line in raw.splitlines() if line.strip()]
|
||||
|
||||
def start(self, spec: HelperSpec) -> str:
|
||||
argv = build_run_argv(spec, docker_bin=self.docker_bin)
|
||||
self.runner.run("run_helper", argv, timeout=120)
|
||||
return self.container_id(spec.container_name)
|
||||
|
||||
def verify(self, spec: HelperSpec) -> dict[str, Any]:
|
||||
container_id = self.container_id(spec.container_name)
|
||||
evidence = verify_container_isolation(
|
||||
spec=spec,
|
||||
baseline_labels=self.image_labels(spec.image),
|
||||
container_labels=self.container_labels(spec.container_name),
|
||||
target_project_members=self.target_project_members(),
|
||||
)
|
||||
return {
|
||||
"schema_version": EVIDENCE_SCHEMA,
|
||||
"role": spec.role,
|
||||
"run_id": spec.run_id,
|
||||
"image": spec.image,
|
||||
"container_name": spec.container_name,
|
||||
"container_id": container_id,
|
||||
"container_running": self.container_running(spec.container_name),
|
||||
"isolated_project": spec.isolated_project,
|
||||
"isolated_service": spec.isolated_service,
|
||||
"target_project": TARGET_PROJECT,
|
||||
"run_argv": build_run_argv(spec, docker_bin=self.docker_bin),
|
||||
**evidence,
|
||||
}
|
||||
|
||||
def wait(
|
||||
self, spec: HelperSpec, *, timeout_seconds: float = 300.0
|
||||
) -> dict[str, Any]:
|
||||
"""Block until a one-shot helper exits and report its exit code."""
|
||||
|
||||
if not 1 <= timeout_seconds <= 1800:
|
||||
raise HelperError("helper_wait_timeout_invalid")
|
||||
raw = self.runner.run(
|
||||
"wait_helper",
|
||||
[self.docker_bin, "wait", spec.container_name],
|
||||
timeout=timeout_seconds,
|
||||
).strip()
|
||||
try:
|
||||
exit_code = int(raw.splitlines()[-1])
|
||||
except (ValueError, IndexError) as exc:
|
||||
raise HelperError("helper_exit_code_invalid") from exc
|
||||
return {
|
||||
"schema_version": EVIDENCE_SCHEMA,
|
||||
"role": spec.role,
|
||||
"run_id": spec.run_id,
|
||||
"container_name": spec.container_name,
|
||||
"container_id": self.container_id(spec.container_name),
|
||||
"exit_code": exit_code,
|
||||
"container_running": self.container_running(spec.container_name),
|
||||
}
|
||||
|
||||
def stop(self, spec: HelperSpec) -> None:
|
||||
"""Stop and remove one helper container by its exact isolated name."""
|
||||
|
||||
if spec.container_name in self.target_project_members():
|
||||
raise HelperError("helper_listed_in_target_project")
|
||||
labels = self.container_labels(spec.container_name)
|
||||
if labels.get(PROJECT_LABEL) != spec.isolated_project:
|
||||
raise HelperError("helper_stop_project_mismatch")
|
||||
if labels.get(HELPER_LABEL) != spec.role:
|
||||
raise HelperError("helper_stop_provenance_mismatch")
|
||||
self.runner.run(
|
||||
"stop_helper",
|
||||
[self.docker_bin, "stop", "--time", "20", spec.container_name],
|
||||
timeout=90,
|
||||
)
|
||||
self.runner.run(
|
||||
"remove_helper",
|
||||
[self.docker_bin, "rm", spec.container_name],
|
||||
timeout=60,
|
||||
)
|
||||
|
||||
|
||||
def _parse_labels(raw: str) -> dict[str, str]:
|
||||
try:
|
||||
value = json.loads(raw.strip() or "null")
|
||||
except json.JSONDecodeError as exc:
|
||||
raise HelperError("helper_labels_unreadable") from exc
|
||||
if value is None:
|
||||
return {}
|
||||
if not isinstance(value, dict) or any(
|
||||
not isinstance(key, str) or not isinstance(item, str)
|
||||
for key, item in value.items()
|
||||
):
|
||||
raise HelperError("helper_labels_unreadable")
|
||||
return value
|
||||
|
||||
|
||||
def build_spec(args: argparse.Namespace) -> HelperSpec:
|
||||
if args.role == "executor":
|
||||
return executor_spec(
|
||||
run_id=args.run_id,
|
||||
image=args.image,
|
||||
manifest_sha256=args.manifest_sha256,
|
||||
)
|
||||
if args.role == "probe":
|
||||
return probe_spec(
|
||||
run_id=args.run_id, image=args.image, plan_name=args.plan_name
|
||||
)
|
||||
return control_plane_spec(run_id=args.run_id, image=args.image)
|
||||
|
||||
|
||||
def _build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument(
|
||||
"command", choices=("plan", "start", "verify", "wait", "stop")
|
||||
)
|
||||
parser.add_argument("--role", required=True, choices=sorted(HELPER_ROLES))
|
||||
parser.add_argument("--run-id", required=True)
|
||||
parser.add_argument("--image", required=True)
|
||||
parser.add_argument("--manifest-sha256", default="")
|
||||
parser.add_argument("--plan-name", default="")
|
||||
parser.add_argument("--wait-seconds", type=float, default=300.0)
|
||||
parser.add_argument("--ssh-destination")
|
||||
parser.add_argument("--ssh-identity-file")
|
||||
parser.add_argument("--docker-bin", default="docker")
|
||||
parser.add_argument("--output")
|
||||
return parser
|
||||
|
||||
|
||||
def _runner(args: argparse.Namespace) -> CommandRunner:
|
||||
if not args.ssh_destination or not args.ssh_identity_file:
|
||||
raise HelperError("ssh_target_required")
|
||||
return SshCommandRunner(
|
||||
destination=args.ssh_destination,
|
||||
identity_file=args.ssh_identity_file,
|
||||
)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = _build_parser().parse_args(argv)
|
||||
try:
|
||||
spec = build_spec(args)
|
||||
if args.command == "plan":
|
||||
payload: dict[str, Any] = {
|
||||
"schema_version": EVIDENCE_SCHEMA,
|
||||
"role": spec.role,
|
||||
"run_id": spec.run_id,
|
||||
"image": spec.image,
|
||||
"container_name": spec.container_name,
|
||||
"isolated_project": spec.isolated_project,
|
||||
"isolated_service": spec.isolated_service,
|
||||
"target_project": TARGET_PROJECT,
|
||||
"env_file": spec.env_file,
|
||||
"run_argv": build_run_argv(spec, docker_bin=args.docker_bin),
|
||||
}
|
||||
else:
|
||||
launcher = HelperLauncher(_runner(args), docker_bin=args.docker_bin)
|
||||
if args.command == "start":
|
||||
launcher.start(spec)
|
||||
payload = launcher.verify(spec)
|
||||
elif args.command == "verify":
|
||||
payload = launcher.verify(spec)
|
||||
elif args.command == "wait":
|
||||
payload = launcher.wait(spec, timeout_seconds=args.wait_seconds)
|
||||
else:
|
||||
launcher.stop(spec)
|
||||
payload = {
|
||||
"schema_version": EVIDENCE_SCHEMA,
|
||||
"role": spec.role,
|
||||
"run_id": spec.run_id,
|
||||
"container_name": spec.container_name,
|
||||
"stopped": True,
|
||||
}
|
||||
except HelperError as exc:
|
||||
print(
|
||||
json.dumps({"ok": False, "error": exc.code}, separators=(",", ":")),
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 2
|
||||
text = json.dumps(payload, ensure_ascii=False, indent=2, sort_keys=True)
|
||||
if args.output:
|
||||
with open(args.output, "w", encoding="utf-8", newline="\n") as handle:
|
||||
handle.write(text + "\n")
|
||||
print(text)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
534
scripts/test_launch_nas_preview_g8_helpers.py
Normal file
534
scripts/test_launch_nas_preview_g8_helpers.py
Normal file
|
|
@ -0,0 +1,534 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import json
|
||||
import sys
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
SCRIPT_PATH = Path(__file__).with_name("launch-nas-preview-g8-helpers.py")
|
||||
SPEC = importlib.util.spec_from_file_location(
|
||||
"launch_nas_preview_g8_helpers", SCRIPT_PATH
|
||||
)
|
||||
assert SPEC is not None and SPEC.loader is not None
|
||||
MODULE = importlib.util.module_from_spec(SPEC)
|
||||
sys.modules[SPEC.name] = MODULE
|
||||
SPEC.loader.exec_module(MODULE)
|
||||
|
||||
|
||||
RUN_ID = "20260807a1"
|
||||
IMAGE = "sha256:" + "5e" * 32
|
||||
MANIFEST_SHA = "e" * 64
|
||||
BASELINE_LABELS = {
|
||||
"com.docker.compose.project": MODULE.TARGET_PROJECT,
|
||||
"com.docker.compose.service": "api",
|
||||
"com.docker.compose.version": "2.20.1",
|
||||
}
|
||||
|
||||
|
||||
def executor_spec() -> object:
|
||||
return MODULE.executor_spec(
|
||||
run_id=RUN_ID, image=IMAGE, manifest_sha256=MANIFEST_SHA
|
||||
)
|
||||
|
||||
|
||||
def control_plane_spec() -> object:
|
||||
return MODULE.control_plane_spec(run_id=RUN_ID, image=IMAGE)
|
||||
|
||||
|
||||
def probe_spec(plan_name: str = "rollback-old") -> object:
|
||||
return MODULE.probe_spec(run_id=RUN_ID, image=IMAGE, plan_name=plan_name)
|
||||
|
||||
|
||||
def container_labels(spec: object, **overrides: str) -> dict[str, str]:
|
||||
labels = dict(BASELINE_LABELS)
|
||||
labels.update(spec.isolated_labels)
|
||||
labels.update(spec.provenance_labels)
|
||||
labels.update(overrides)
|
||||
return labels
|
||||
|
||||
|
||||
class FakeRunner:
|
||||
"""Record every argv and answer from a canned, operation-keyed map."""
|
||||
|
||||
def __init__(self, responses: dict[str, str] | None = None):
|
||||
self.calls: list[tuple[str, list[str]]] = []
|
||||
self.responses = responses or {}
|
||||
|
||||
def run(self, operation: str, argv: list[str], *, timeout: float) -> str:
|
||||
assert isinstance(argv, list) and argv
|
||||
assert all(isinstance(item, str) for item in argv)
|
||||
assert timeout > 0
|
||||
self.calls.append((operation, list(argv)))
|
||||
return self.responses.get(operation, "")
|
||||
|
||||
def argv_for(self, operation: str) -> list[str]:
|
||||
matches = [argv for name, argv in self.calls if name == operation]
|
||||
assert len(matches) == 1, operation
|
||||
return matches[0]
|
||||
|
||||
|
||||
def _label_values(argv: list[str]) -> dict[str, str]:
|
||||
values: dict[str, str] = {}
|
||||
for index, item in enumerate(argv):
|
||||
if item == "--label":
|
||||
key, _, value = argv[index + 1].partition("=")
|
||||
values[key] = value
|
||||
return values
|
||||
|
||||
|
||||
class SpecContractTest(unittest.TestCase):
|
||||
def test_isolated_project_is_never_the_target_project(self) -> None:
|
||||
for spec in (executor_spec(), control_plane_spec(), probe_spec()):
|
||||
self.assertNotEqual(spec.isolated_project, MODULE.TARGET_PROJECT)
|
||||
self.assertTrue(
|
||||
spec.isolated_project.startswith(MODULE.HELPER_PROJECT_PREFIX)
|
||||
)
|
||||
|
||||
def test_isolated_service_is_never_a_target_service(self) -> None:
|
||||
for spec in (executor_spec(), control_plane_spec(), probe_spec()):
|
||||
self.assertNotIn(spec.isolated_service, MODULE.TARGET_SERVICES)
|
||||
|
||||
def test_container_name_is_not_a_target_compose_container(self) -> None:
|
||||
for spec in (executor_spec(), control_plane_spec(), probe_spec()):
|
||||
for service in MODULE.TARGET_SERVICES:
|
||||
self.assertNotEqual(
|
||||
spec.container_name, f"{MODULE.TARGET_PROJECT}-{service}-1"
|
||||
)
|
||||
|
||||
def test_invalid_run_id_and_image_fail_closed(self) -> None:
|
||||
with self.assertRaises(MODULE.HelperError):
|
||||
MODULE.control_plane_spec(run_id="BAD ID", image=IMAGE)
|
||||
with self.assertRaises(MODULE.HelperError):
|
||||
MODULE.control_plane_spec(run_id=RUN_ID, image="latest")
|
||||
with self.assertRaises(MODULE.HelperError):
|
||||
MODULE.executor_spec(run_id=RUN_ID, image=IMAGE, manifest_sha256="nope")
|
||||
|
||||
|
||||
class RunArgvContractTest(unittest.TestCase):
|
||||
def test_argv_never_labels_the_target_project_or_service(self) -> None:
|
||||
for spec in (executor_spec(), control_plane_spec(), probe_spec()):
|
||||
argv = MODULE.build_run_argv(spec)
|
||||
labels = _label_values(argv)
|
||||
self.assertEqual(
|
||||
labels[MODULE.PROJECT_LABEL], spec.isolated_project
|
||||
)
|
||||
self.assertEqual(
|
||||
labels[MODULE.SERVICE_LABEL], spec.isolated_service
|
||||
)
|
||||
self.assertNotIn(MODULE.TARGET_PROJECT, labels.values())
|
||||
self.assertNotIn(labels[MODULE.SERVICE_LABEL], MODULE.TARGET_SERVICES)
|
||||
|
||||
def test_argv_declares_only_the_two_isolated_compose_labels(self) -> None:
|
||||
argv = MODULE.build_run_argv(executor_spec())
|
||||
compose_labels = {
|
||||
key
|
||||
for key in _label_values(argv)
|
||||
if key.startswith(MODULE.COMPOSE_LABEL_PREFIX)
|
||||
}
|
||||
self.assertEqual(
|
||||
compose_labels, {MODULE.PROJECT_LABEL, MODULE.SERVICE_LABEL}
|
||||
)
|
||||
|
||||
def test_injected_target_project_label_is_rejected(self) -> None:
|
||||
spec = executor_spec()
|
||||
argv = MODULE.build_run_argv(spec)
|
||||
tampered = argv + [
|
||||
"--label",
|
||||
f"{MODULE.PROJECT_LABEL}={MODULE.TARGET_PROJECT}",
|
||||
]
|
||||
with self.assertRaises(MODULE.HelperError) as ctx:
|
||||
MODULE.assert_argv_isolation(tampered, spec)
|
||||
self.assertEqual(ctx.exception.code, "helper_argv_targets_project")
|
||||
|
||||
def test_injected_target_service_label_is_rejected(self) -> None:
|
||||
spec = control_plane_spec()
|
||||
tampered = MODULE.build_run_argv(spec) + [
|
||||
"--label",
|
||||
f"{MODULE.SERVICE_LABEL}=api",
|
||||
]
|
||||
with self.assertRaises(MODULE.HelperError) as ctx:
|
||||
MODULE.assert_argv_isolation(tampered, spec)
|
||||
self.assertEqual(ctx.exception.code, "helper_argv_targets_service")
|
||||
|
||||
def test_undeclared_compose_label_is_rejected(self) -> None:
|
||||
spec = executor_spec()
|
||||
tampered = MODULE.build_run_argv(spec) + [
|
||||
"--label",
|
||||
f"{MODULE.COMPOSE_LABEL_PREFIX}config-hash=deadbeef",
|
||||
]
|
||||
with self.assertRaises(MODULE.HelperError) as ctx:
|
||||
MODULE.assert_argv_isolation(tampered, spec)
|
||||
self.assertEqual(ctx.exception.code, "helper_argv_undeclared_compose_label")
|
||||
|
||||
def test_label_file_and_non_host_network_are_rejected(self) -> None:
|
||||
spec = executor_spec()
|
||||
with self.assertRaises(MODULE.HelperError):
|
||||
MODULE.assert_argv_isolation(
|
||||
MODULE.build_run_argv(spec) + ["--label-file", "/tmp/labels"], spec
|
||||
)
|
||||
with self.assertRaises(MODULE.HelperError) as ctx:
|
||||
MODULE.assert_argv_isolation(
|
||||
[
|
||||
"docker",
|
||||
"run",
|
||||
"--network",
|
||||
f"{MODULE.TARGET_PROJECT}_vignette",
|
||||
spec.image,
|
||||
],
|
||||
spec,
|
||||
)
|
||||
self.assertEqual(ctx.exception.code, "helper_argv_network_not_isolated")
|
||||
|
||||
def test_executor_argv_carries_the_hardening_flags(self) -> None:
|
||||
argv = MODULE.build_run_argv(executor_spec())
|
||||
self.assertIn("--read-only", argv)
|
||||
self.assertIn("--cap-drop", argv)
|
||||
self.assertEqual(argv[argv.index("--cap-drop") + 1], "ALL")
|
||||
self.assertIn("--security-opt", argv)
|
||||
self.assertEqual(
|
||||
argv[argv.index("--security-opt") + 1], "no-new-privileges"
|
||||
)
|
||||
self.assertEqual(
|
||||
argv[argv.index("--user") + 1],
|
||||
f"{MODULE.HELPER_UID}:{MODULE.HELPER_GID}",
|
||||
)
|
||||
self.assertEqual(
|
||||
argv[argv.index("--group-add") + 1],
|
||||
str(MODULE.HELPER_SUPPLEMENTAL_GID),
|
||||
)
|
||||
self.assertEqual(argv[argv.index("--network") + 1], "host")
|
||||
|
||||
def test_executor_argv_mounts_target_root_read_only_and_state_rw(self) -> None:
|
||||
argv = MODULE.build_run_argv(executor_spec())
|
||||
mounts = [
|
||||
argv[index + 1]
|
||||
for index, item in enumerate(argv)
|
||||
if item == "--mount"
|
||||
]
|
||||
self.assertIn(
|
||||
f"type=bind,source={MODULE.TARGET_ROOT},"
|
||||
f"target={MODULE.TARGET_ROOT},readonly",
|
||||
mounts,
|
||||
)
|
||||
self.assertIn(
|
||||
f"type=bind,source={MODULE.STATE_DIR},target={MODULE.STATE_DIR}",
|
||||
mounts,
|
||||
)
|
||||
self.assertIn(
|
||||
f"type=bind,source={MODULE.DOCKER_BINARY_SOURCE},"
|
||||
f"target={MODULE.DOCKER_BINARY_TARGET},readonly",
|
||||
mounts,
|
||||
)
|
||||
self.assertIn(
|
||||
f"type=bind,source={MODULE.COMPOSE_PLUGIN_SOURCE},"
|
||||
f"target={MODULE.COMPOSE_PLUGIN_TARGET},readonly",
|
||||
mounts,
|
||||
)
|
||||
self.assertIn(
|
||||
f"type=bind,source={MODULE.TARGET_DOCKER_SOCKET},"
|
||||
f"target={MODULE.TARGET_DOCKER_SOCKET}",
|
||||
mounts,
|
||||
)
|
||||
|
||||
def test_executor_argv_binds_loopback_and_the_audited_manifest(self) -> None:
|
||||
argv = MODULE.build_run_argv(executor_spec())
|
||||
self.assertEqual(argv[argv.index("--entrypoint") + 1], "python3.11")
|
||||
self.assertEqual(argv[argv.index("--host") + 1], MODULE.EXECUTOR_HOST)
|
||||
self.assertEqual(argv[argv.index("--port") + 1], str(MODULE.EXECUTOR_PORT))
|
||||
self.assertEqual(argv[argv.index("--manifest") + 1], str(MODULE.MANIFEST_PATH))
|
||||
self.assertEqual(argv[argv.index("--manifest-sha256") + 1], MANIFEST_SHA)
|
||||
self.assertIn("--enable", argv)
|
||||
|
||||
def test_control_plane_argv_binds_its_own_loopback_port(self) -> None:
|
||||
argv = MODULE.build_run_argv(control_plane_spec())
|
||||
self.assertEqual(argv[argv.index("--entrypoint") + 1], "uvicorn")
|
||||
self.assertEqual(argv[argv.index("--host") + 1], MODULE.CONTROL_PLANE_HOST)
|
||||
self.assertEqual(
|
||||
argv[argv.index("--port") + 1], str(MODULE.CONTROL_PLANE_PORT)
|
||||
)
|
||||
self.assertNotIn("--publish", argv)
|
||||
|
||||
def test_argv_never_mutates_the_target_compose_project(self) -> None:
|
||||
for spec in (executor_spec(), control_plane_spec(), probe_spec()):
|
||||
argv = MODULE.build_run_argv(spec)
|
||||
self.assertEqual(argv[1], "run")
|
||||
for forbidden in ("compose", "down", "prune", "rm", "stop", "kill"):
|
||||
self.assertNotIn(forbidden, argv[1:2])
|
||||
self.assertNotIn(MODULE.TARGET_PROJECT, argv[3:])
|
||||
|
||||
|
||||
class ProbeRoleTest(unittest.TestCase):
|
||||
def test_probe_never_mounts_the_docker_socket(self) -> None:
|
||||
argv = MODULE.build_run_argv(probe_spec())
|
||||
mounts = [
|
||||
argv[index + 1] for index, item in enumerate(argv) if item == "--mount"
|
||||
]
|
||||
self.assertFalse(
|
||||
any(MODULE.TARGET_DOCKER_SOCKET in mount for mount in mounts)
|
||||
)
|
||||
self.assertNotIn("--group-add", argv)
|
||||
|
||||
def test_probe_targets_the_separate_control_plane_origin(self) -> None:
|
||||
argv = MODULE.build_run_argv(probe_spec())
|
||||
control = argv[argv.index("--control-plane-base-url") + 1]
|
||||
preview = argv[argv.index("--preview-base-url") + 1]
|
||||
self.assertEqual(control, MODULE.CONTROL_PLANE_BASE_URL)
|
||||
self.assertEqual(preview, MODULE.PREVIEW_BASE_URL)
|
||||
self.assertNotEqual(control, preview)
|
||||
|
||||
def test_probe_plan_and_output_stay_inside_the_state_dir(self) -> None:
|
||||
for plan_name in ("rollback-old", "restore-current"):
|
||||
argv = MODULE.build_run_argv(probe_spec(plan_name))
|
||||
plan = argv[argv.index("--plan") + 1]
|
||||
output = argv[argv.index("--output") + 1]
|
||||
self.assertEqual(
|
||||
plan, f"{MODULE.PLANS_DIR}/{plan_name}.plan.json"
|
||||
)
|
||||
self.assertTrue(output.startswith(f"{MODULE.EVIDENCE_DIR}/"))
|
||||
self.assertIn("execute", argv)
|
||||
|
||||
def test_probe_plan_name_is_validated(self) -> None:
|
||||
for bad in ("../escape", "Rollback", "a", "x" * 80):
|
||||
with self.assertRaises(MODULE.HelperError):
|
||||
MODULE.probe_spec(run_id=RUN_ID, image=IMAGE, plan_name=bad)
|
||||
|
||||
def test_wait_reports_the_exit_code(self) -> None:
|
||||
spec = probe_spec()
|
||||
runner = FakeRunner(
|
||||
{
|
||||
"wait_helper": "0\n",
|
||||
"inspect_container_id": "b" * 64 + "\n",
|
||||
"inspect_container_state": "false\n",
|
||||
}
|
||||
)
|
||||
result = MODULE.HelperLauncher(runner).wait(spec, timeout_seconds=60)
|
||||
self.assertEqual(result["exit_code"], 0)
|
||||
self.assertFalse(result["container_running"])
|
||||
self.assertEqual(
|
||||
runner.argv_for("wait_helper"), ["docker", "wait", spec.container_name]
|
||||
)
|
||||
|
||||
def test_wait_rejects_an_unbounded_timeout(self) -> None:
|
||||
with self.assertRaises(MODULE.HelperError):
|
||||
MODULE.HelperLauncher(FakeRunner()).wait(probe_spec(), timeout_seconds=0)
|
||||
|
||||
|
||||
class IsolationVerificationTest(unittest.TestCase):
|
||||
def test_inherited_baseline_labels_are_accepted(self) -> None:
|
||||
spec = executor_spec()
|
||||
evidence = MODULE.verify_container_isolation(
|
||||
spec=spec,
|
||||
baseline_labels=BASELINE_LABELS,
|
||||
container_labels=container_labels(spec),
|
||||
target_project_members=[f"{MODULE.TARGET_PROJECT}-api-1"],
|
||||
)
|
||||
self.assertEqual(
|
||||
evidence["inherited_compose_labels"],
|
||||
{"com.docker.compose.version": "2.20.1"},
|
||||
)
|
||||
self.assertEqual(
|
||||
evidence["isolated_override_labels"],
|
||||
{
|
||||
MODULE.PROJECT_LABEL: spec.isolated_project,
|
||||
MODULE.SERVICE_LABEL: spec.isolated_service,
|
||||
},
|
||||
)
|
||||
self.assertEqual(
|
||||
evidence["baseline_compose_labels"]["com.docker.compose.project"],
|
||||
MODULE.TARGET_PROJECT,
|
||||
)
|
||||
|
||||
def test_zero_compose_labels_is_not_required(self) -> None:
|
||||
spec = executor_spec()
|
||||
labels = container_labels(spec)
|
||||
self.assertTrue(
|
||||
any(key.startswith(MODULE.COMPOSE_LABEL_PREFIX) for key in labels)
|
||||
)
|
||||
MODULE.verify_container_isolation(
|
||||
spec=spec,
|
||||
baseline_labels=BASELINE_LABELS,
|
||||
container_labels=labels,
|
||||
target_project_members=[],
|
||||
)
|
||||
|
||||
def test_target_project_membership_is_rejected(self) -> None:
|
||||
spec = executor_spec()
|
||||
with self.assertRaises(MODULE.HelperError) as ctx:
|
||||
MODULE.verify_container_isolation(
|
||||
spec=spec,
|
||||
baseline_labels=BASELINE_LABELS,
|
||||
container_labels=container_labels(
|
||||
spec, **{MODULE.PROJECT_LABEL: MODULE.TARGET_PROJECT}
|
||||
),
|
||||
target_project_members=[],
|
||||
)
|
||||
self.assertEqual(ctx.exception.code, "helper_isolated_label_mismatch")
|
||||
|
||||
def test_target_service_claim_is_rejected(self) -> None:
|
||||
spec = MODULE.HelperSpec(
|
||||
role="executor",
|
||||
run_id=RUN_ID,
|
||||
image=IMAGE,
|
||||
env_file=str(MODULE.STATE_DIR / "x.env"),
|
||||
entrypoint="python3.11",
|
||||
command=("-c", "pass"),
|
||||
)
|
||||
labels = container_labels(spec)
|
||||
labels[MODULE.SERVICE_LABEL] = "db"
|
||||
with self.assertRaises(MODULE.HelperError):
|
||||
MODULE.verify_container_isolation(
|
||||
spec=spec,
|
||||
baseline_labels=BASELINE_LABELS,
|
||||
container_labels=labels,
|
||||
target_project_members=[],
|
||||
)
|
||||
|
||||
def test_unexpected_compose_label_drift_is_rejected(self) -> None:
|
||||
spec = executor_spec()
|
||||
labels = container_labels(spec)
|
||||
labels["com.docker.compose.version"] = "9.9.9"
|
||||
with self.assertRaises(MODULE.HelperError) as ctx:
|
||||
MODULE.verify_container_isolation(
|
||||
spec=spec,
|
||||
baseline_labels=BASELINE_LABELS,
|
||||
container_labels=labels,
|
||||
target_project_members=[],
|
||||
)
|
||||
self.assertEqual(ctx.exception.code, "helper_compose_label_drift")
|
||||
|
||||
def test_missing_isolated_override_is_rejected(self) -> None:
|
||||
spec = executor_spec()
|
||||
labels = container_labels(spec)
|
||||
del labels[MODULE.SERVICE_LABEL]
|
||||
with self.assertRaises(MODULE.HelperError) as ctx:
|
||||
MODULE.verify_container_isolation(
|
||||
spec=spec,
|
||||
baseline_labels=BASELINE_LABELS,
|
||||
container_labels=labels,
|
||||
target_project_members=[],
|
||||
)
|
||||
self.assertEqual(ctx.exception.code, "helper_isolated_label_missing")
|
||||
|
||||
def test_helper_listed_in_target_project_is_rejected(self) -> None:
|
||||
spec = executor_spec()
|
||||
with self.assertRaises(MODULE.HelperError) as ctx:
|
||||
MODULE.verify_container_isolation(
|
||||
spec=spec,
|
||||
baseline_labels=BASELINE_LABELS,
|
||||
container_labels=container_labels(spec),
|
||||
target_project_members=[spec.container_name],
|
||||
)
|
||||
self.assertEqual(ctx.exception.code, "helper_listed_in_target_project")
|
||||
|
||||
def test_missing_provenance_label_is_rejected(self) -> None:
|
||||
spec = executor_spec()
|
||||
labels = container_labels(spec)
|
||||
del labels[MODULE.HELPER_LABEL]
|
||||
with self.assertRaises(MODULE.HelperError) as ctx:
|
||||
MODULE.verify_container_isolation(
|
||||
spec=spec,
|
||||
baseline_labels=BASELINE_LABELS,
|
||||
container_labels=labels,
|
||||
target_project_members=[],
|
||||
)
|
||||
self.assertEqual(ctx.exception.code, "helper_provenance_label_missing")
|
||||
|
||||
|
||||
class LauncherRunnerTest(unittest.TestCase):
|
||||
def _responses(self, spec: object) -> dict[str, str]:
|
||||
return {
|
||||
"inspect_image_labels": json.dumps(BASELINE_LABELS),
|
||||
"inspect_container_labels": json.dumps(container_labels(spec)),
|
||||
"inspect_container_id": "a" * 64 + "\n",
|
||||
"inspect_container_state": "true\n",
|
||||
"list_target_project_members": (
|
||||
f"{MODULE.TARGET_PROJECT}-api-1\n{MODULE.TARGET_PROJECT}-web-1\n"
|
||||
),
|
||||
}
|
||||
|
||||
def test_start_uses_the_exact_argv_and_returns_the_container_id(self) -> None:
|
||||
spec = executor_spec()
|
||||
runner = FakeRunner(self._responses(spec))
|
||||
launcher = MODULE.HelperLauncher(runner)
|
||||
container_id = launcher.start(spec)
|
||||
self.assertEqual(container_id, "a" * 64)
|
||||
self.assertEqual(
|
||||
runner.argv_for("run_helper"), MODULE.build_run_argv(spec)
|
||||
)
|
||||
|
||||
def test_verify_emits_exact_container_identity_evidence(self) -> None:
|
||||
spec = control_plane_spec()
|
||||
runner = FakeRunner(self._responses(spec))
|
||||
evidence = MODULE.HelperLauncher(runner).verify(spec)
|
||||
self.assertEqual(evidence["schema_version"], MODULE.EVIDENCE_SCHEMA)
|
||||
self.assertEqual(evidence["container_name"], spec.container_name)
|
||||
self.assertEqual(evidence["container_id"], "a" * 64)
|
||||
self.assertTrue(evidence["container_running"])
|
||||
self.assertEqual(evidence["target_project"], MODULE.TARGET_PROJECT)
|
||||
self.assertNotIn(spec.container_name, evidence["target_project_members"])
|
||||
|
||||
def test_membership_query_filters_on_the_target_project_label(self) -> None:
|
||||
spec = executor_spec()
|
||||
runner = FakeRunner(self._responses(spec))
|
||||
MODULE.HelperLauncher(runner).verify(spec)
|
||||
argv = runner.argv_for("list_target_project_members")
|
||||
self.assertIn("ps", argv)
|
||||
self.assertIn(
|
||||
f"label={MODULE.PROJECT_LABEL}={MODULE.TARGET_PROJECT}", argv
|
||||
)
|
||||
|
||||
def test_stop_refuses_a_container_outside_the_isolated_project(self) -> None:
|
||||
spec = executor_spec()
|
||||
responses = self._responses(spec)
|
||||
foreign = container_labels(spec)
|
||||
foreign[MODULE.PROJECT_LABEL] = "someone-else"
|
||||
responses["inspect_container_labels"] = json.dumps(foreign)
|
||||
launcher = MODULE.HelperLauncher(FakeRunner(responses))
|
||||
with self.assertRaises(MODULE.HelperError) as ctx:
|
||||
launcher.stop(spec)
|
||||
self.assertEqual(ctx.exception.code, "helper_stop_project_mismatch")
|
||||
|
||||
def test_stop_only_touches_the_exact_helper_name(self) -> None:
|
||||
spec = executor_spec()
|
||||
runner = FakeRunner(self._responses(spec))
|
||||
MODULE.HelperLauncher(runner).stop(spec)
|
||||
stop_argv = runner.argv_for("stop_helper")
|
||||
remove_argv = runner.argv_for("remove_helper")
|
||||
self.assertEqual(stop_argv[-1], spec.container_name)
|
||||
self.assertEqual(remove_argv, ["docker", "rm", spec.container_name])
|
||||
self.assertNotIn("--force", remove_argv)
|
||||
self.assertNotIn("--volumes", remove_argv)
|
||||
|
||||
def test_invalid_label_json_fails_closed(self) -> None:
|
||||
spec = executor_spec()
|
||||
responses = self._responses(spec)
|
||||
responses["inspect_image_labels"] = "not-json"
|
||||
launcher = MODULE.HelperLauncher(FakeRunner(responses))
|
||||
with self.assertRaises(MODULE.HelperError) as ctx:
|
||||
launcher.image_labels(spec.image)
|
||||
self.assertEqual(ctx.exception.code, "helper_labels_unreadable")
|
||||
|
||||
def test_null_image_labels_are_read_as_empty(self) -> None:
|
||||
runner = FakeRunner({"inspect_image_labels": "null\n"})
|
||||
launcher = MODULE.HelperLauncher(runner)
|
||||
self.assertEqual(launcher.image_labels(IMAGE), {})
|
||||
|
||||
def test_ssh_runner_quotes_the_argv_into_one_remote_command(self) -> None:
|
||||
inner = FakeRunner()
|
||||
runner = MODULE.SshCommandRunner(
|
||||
destination="user@host",
|
||||
identity_file="C:/key",
|
||||
inner=inner,
|
||||
)
|
||||
runner.run("probe", ["docker", "ps", "--filter", "label=a=b c"], timeout=5)
|
||||
argv = inner.argv_for("probe")
|
||||
self.assertEqual(argv[0], "ssh")
|
||||
self.assertIn("BatchMode=yes", argv)
|
||||
self.assertEqual(argv[-2], "user@host")
|
||||
self.assertEqual(argv[-1], "docker ps --filter 'label=a=b c'")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Loading…
Add table
Add a link
Reference in a new issue