fix(desktop): stop pinning local models in memory and cap audio buffers
Some checks failed
ci / 정본·보안·린트·타입·테스트 (push) Successful in 56s
ci / .NET API 서버 테스트 (push) Successful in 10s
deploy-site / deploy (push) Successful in 38s
ci / 모바일 린트·타입·Jest (push) Successful in 44s
ci / 워크스페이스 빌드 검증 (push) Successful in 54s
ci / Supabase Edge Functions + Cloudflare Worker (push) Successful in 22s
portable-unsigned / portable-windows (push) Failing after 35s
release / release-windows (push) Failing after 13s

The PC-wide slowdown was not a leak but model residency: boot warmup loaded
gemma4:e4b (~11.6 GB RAM incl. 5.9 GB pinned host buffer, no mmap on Windows
CUDA) and kept it for 30m, next to the large-v3 sidecar. Measured on the HEAD
build: free RAM 7.9 GB -> 32 MB within 20 s; after this change 10.8 GB with no
Ollama model loaded until typing starts.

- single LOCAL_LLM_KEEP_ALIVE (10m) for every local request; no boot warmup
- unload this app's resident Ollama models (keep_alive 0) on quit
- sidecar POST /unload for the aux slot; captions release it on stop
- partial dictation copies only the tail window, not the whole recording
- cap the voice-conversation listen buffer at 5 minutes
- mic test: generation guard so re-entry cannot orphan lease/timers
- startRealtime: single-flight and removeChannel to stop duplicate bindings
- meeting WAV save writes header and PCM without a concat copy

Docs: AI-01 policy note, GAP-PERF-01 with remaining work.
This commit is contained in:
Yun Chan 2026-10-11 11:35:15 +09:00
parent a7c0d6fb21
commit d63854db67
19 changed files with 268 additions and 45 deletions

View file

@ -24,6 +24,7 @@ from __future__ import annotations
import argparse
import asyncio
import fnmatch
import gc
import logging
import os
import signal
@ -595,6 +596,33 @@ async def load_model(body: dict) -> JSONResponse: # noqa: ANN001
)
@app.post("/unload")
async def unload_model(request: Request) -> JSONResponse:
"""보조 자리 모델을 내려 VRAM/RAM 을 돌려준다.
자막·회의가 끝난 뒤에도 보조 모델(예: large-v3-turbo)이 받아쓰기 모델과 함께 상주하면
GPU 여유가 줄어 Ollama 모델 로드가 느려지거나 OOM 이 난다. 기본(primary) 모델은 받아쓰기
지연을 막기 위해 내리지 않는다.
Request body: { "slot": "aux" }
"""
body = await request.json()
slot: str = body.get("slot", "aux")
if slot != "aux":
return JSONResponse(
status_code=400,
content={"status": "error", "message": "only the aux slot can be unloaded"},
)
async with _inference_lock:
released = list(_aux_models)
_aux_models.clear()
if released:
gc.collect()
logger.info("보조 모델 언로드: %s", ", ".join(released))
return JSONResponse(content={"status": "unloaded", "models": released})
async def _watch_disconnect(request: Request, cancel: threading.Event) -> None:
"""클라이언트가 연결을 끊으면(타임아웃 abort 등) cancel 을 세운다."""
while not cancel.is_set():