fix(desktop): stop pinning local models in memory and cap audio buffers
Some checks failed
ci / 정본·보안·린트·타입·테스트 (push) Successful in 56s
ci / .NET API 서버 테스트 (push) Successful in 10s
deploy-site / deploy (push) Successful in 38s
ci / 모바일 린트·타입·Jest (push) Successful in 44s
ci / 워크스페이스 빌드 검증 (push) Successful in 54s
ci / Supabase Edge Functions + Cloudflare Worker (push) Successful in 22s
portable-unsigned / portable-windows (push) Failing after 35s
release / release-windows (push) Failing after 13s
Some checks failed
ci / 정본·보안·린트·타입·테스트 (push) Successful in 56s
ci / .NET API 서버 테스트 (push) Successful in 10s
deploy-site / deploy (push) Successful in 38s
ci / 모바일 린트·타입·Jest (push) Successful in 44s
ci / 워크스페이스 빌드 검증 (push) Successful in 54s
ci / Supabase Edge Functions + Cloudflare Worker (push) Successful in 22s
portable-unsigned / portable-windows (push) Failing after 35s
release / release-windows (push) Failing after 13s
The PC-wide slowdown was not a leak but model residency: boot warmup loaded gemma4:e4b (~11.6 GB RAM incl. 5.9 GB pinned host buffer, no mmap on Windows CUDA) and kept it for 30m, next to the large-v3 sidecar. Measured on the HEAD build: free RAM 7.9 GB -> 32 MB within 20 s; after this change 10.8 GB with no Ollama model loaded until typing starts. - single LOCAL_LLM_KEEP_ALIVE (10m) for every local request; no boot warmup - unload this app's resident Ollama models (keep_alive 0) on quit - sidecar POST /unload for the aux slot; captions release it on stop - partial dictation copies only the tail window, not the whole recording - cap the voice-conversation listen buffer at 5 minutes - mic test: generation guard so re-entry cannot orphan lease/timers - startRealtime: single-flight and removeChannel to stop duplicate bindings - meeting WAV save writes header and PCM without a concat copy Docs: AI-01 policy note, GAP-PERF-01 with remaining work.
This commit is contained in:
parent
a7c0d6fb21
commit
d63854db67
19 changed files with 268 additions and 45 deletions
|
|
@ -24,6 +24,7 @@ from __future__ import annotations
|
|||
import argparse
|
||||
import asyncio
|
||||
import fnmatch
|
||||
import gc
|
||||
import logging
|
||||
import os
|
||||
import signal
|
||||
|
|
@ -595,6 +596,33 @@ async def load_model(body: dict) -> JSONResponse: # noqa: ANN001
|
|||
)
|
||||
|
||||
|
||||
@app.post("/unload")
|
||||
async def unload_model(request: Request) -> JSONResponse:
|
||||
"""보조 자리 모델을 내려 VRAM/RAM 을 돌려준다.
|
||||
|
||||
자막·회의가 끝난 뒤에도 보조 모델(예: large-v3-turbo)이 받아쓰기 모델과 함께 상주하면
|
||||
GPU 여유가 줄어 Ollama 모델 로드가 느려지거나 OOM 이 난다. 기본(primary) 모델은 받아쓰기
|
||||
지연을 막기 위해 내리지 않는다.
|
||||
|
||||
Request body: { "slot": "aux" }
|
||||
"""
|
||||
body = await request.json()
|
||||
slot: str = body.get("slot", "aux")
|
||||
if slot != "aux":
|
||||
return JSONResponse(
|
||||
status_code=400,
|
||||
content={"status": "error", "message": "only the aux slot can be unloaded"},
|
||||
)
|
||||
|
||||
async with _inference_lock:
|
||||
released = list(_aux_models)
|
||||
_aux_models.clear()
|
||||
if released:
|
||||
gc.collect()
|
||||
logger.info("보조 모델 언로드: %s", ", ".join(released))
|
||||
return JSONResponse(content={"status": "unloaded", "models": released})
|
||||
|
||||
|
||||
async def _watch_disconnect(request: Request, cancel: threading.Event) -> None:
|
||||
"""클라이언트가 연결을 끊으면(타임아웃 abort 등) cancel 을 세운다."""
|
||||
while not cancel.is_set():
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue