134 lines
4.5 KiB
Markdown
134 lines
4.5 KiB
Markdown
# Public Runtime Watchdog
|
|
|
|
This runbook keeps the public Vignette runtime recoverable after Windows
|
|
reboot, update, or process crash. It covers:
|
|
|
|
- engine gateway: `http://127.0.0.1:9099`
|
|
- prod API: `http://127.0.0.1:8001`
|
|
- Cloudflare tunnel for `https://api-vignette.chanpaca.net`
|
|
|
|
## Secret Handling
|
|
|
|
Do not put secrets in scheduled task arguments.
|
|
|
|
The scripts use the existing runtime locations:
|
|
|
|
- API secrets stay in `apps/api/.env`.
|
|
- Cloudflared credentials stay under the current user's `.cloudflared` config.
|
|
- Claude CLI OAuth/config stays in the current Windows user profile.
|
|
|
|
The installer creates a per-user interactive scheduled task. It starts at user
|
|
logon and repeats as a watchdog. Fully unattended boot before any user logs in
|
|
requires an operator-managed service account or Task Scheduler credential; do
|
|
that in Windows, not by adding secrets to these scripts.
|
|
|
|
## Install Or Update
|
|
|
|
From the repo root:
|
|
|
|
```powershell
|
|
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\install-public-runtime-task.ps1 -RunNow
|
|
```
|
|
|
|
The installer is idempotent. Re-running it updates the same task:
|
|
|
|
```powershell
|
|
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\install-public-runtime-task.ps1 -IntervalMinutes 5
|
|
```
|
|
|
|
Do not add future domains to the default watchdog until DNS and Cloudflare
|
|
routing are live. For example, `api-vnet.18ka.net` is intentionally excluded
|
|
from the default checks while that domain has no DNS. After a public domain is
|
|
actually reachable, add it explicitly:
|
|
|
|
```powershell
|
|
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\install-public-runtime-task.ps1 `
|
|
-AdditionalPublicHealthUrls https://api-vnet.18ka.net/health
|
|
```
|
|
|
|
Task name:
|
|
|
|
```powershell
|
|
VignettePublicRuntimeWatchdog
|
|
```
|
|
|
|
## Manual Start
|
|
|
|
Use this when you want to force a runtime restore immediately:
|
|
|
|
```powershell
|
|
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\start-public-runtime.ps1
|
|
```
|
|
|
|
`start-public-runtime.ps1`는 관리자·인증 제어면과 엔진을 분리한다. 이미
|
|
`environment=prod`, `db=true`인 API와 정상 웹·터널은 유지하고, 엔진만 실패한 경우
|
|
엔진만 복구한다. 엔진이 늦게 준비돼도 관리자·인증 API 기동을 막지 않는다.
|
|
|
|
API 코드 변경을 운영 프로세스에 반영할 때는 다른 표면을 유지한 채 API만 명시적으로 교체한다.
|
|
|
|
```powershell
|
|
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\start-public-runtime.ps1 `
|
|
-ForceApiRestart `
|
|
-SkipEngineRestart `
|
|
-SkipWebRestart `
|
|
-SkipCloudflaredRestart
|
|
```
|
|
|
|
## Health Checks
|
|
|
|
```powershell
|
|
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\watch-public-runtime.ps1 -CheckOnly
|
|
Invoke-RestMethod http://127.0.0.1:9099/health
|
|
Invoke-RestMethod http://127.0.0.1:8001/health
|
|
Invoke-RestMethod https://api-vignette.chanpaca.net/health
|
|
```
|
|
|
|
Optional extra public host check, only after the host resolves:
|
|
|
|
```powershell
|
|
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\watch-public-runtime.ps1 `
|
|
-CheckOnly `
|
|
-AdditionalPublicHealthUrls https://api-vnet.18ka.net/health
|
|
```
|
|
|
|
Optional real Claude CLI readiness smoke:
|
|
|
|
```powershell
|
|
Invoke-RestMethod http://127.0.0.1:9099/ready
|
|
```
|
|
|
|
`/ready` can consume a small Claude budget because it performs a real generation.
|
|
|
|
## Logs And Task State
|
|
|
|
```powershell
|
|
Get-ScheduledTask -TaskName VignettePublicRuntimeWatchdog
|
|
Get-ScheduledTaskInfo -TaskName VignettePublicRuntimeWatchdog
|
|
Get-Content .\public-runtime-watchdog.log -Tail 50
|
|
Get-Content .\apps\api\engine.public.err.log -Tail 50
|
|
Get-Content .\apps\api\api.public.err.log -Tail 50
|
|
Get-Content .\cloudflared.public.err.log -Tail 50
|
|
```
|
|
|
|
## Remove
|
|
|
|
```powershell
|
|
Unregister-ScheduledTask -TaskName VignettePublicRuntimeWatchdog -Confirm:$false
|
|
```
|
|
|
|
## Recovery Notes
|
|
|
|
If local health is good but public health fails, inspect the cloudflared process
|
|
and `C:\Users\<user>\.cloudflared\vignette-config.yml`.
|
|
|
|
If engine health fails, verify that `claude` runs for the same Windows user that
|
|
owns the scheduled task and that the user has completed Claude CLI login.
|
|
|
|
엔진 장애 중에도 `http://127.0.0.1:8001/health`의 `environment=prod`, `db=true`가
|
|
유지되면 관리자·인증 제어면은 정상이다. 이때 watchdog은 API·웹·터널을 재시작하지
|
|
않는다. 예약 작업 확인 기준은 `VignettePublicRuntime`의 `LastTaskResult=0`과
|
|
`public-runtime-watchdog.failcount=0`이다.
|
|
|
|
If API health fails with `environment`, `db`, or auth configuration errors,
|
|
inspect `apps/api/.env`; do not copy secret values into scripts or task
|
|
arguments.
|