d3ro-voice/docs/map/11-gap-backlog.md
Yun Chan 2d585bfc29 feat(desktop): make local speech transcription work end to end
Local dictation had never produced a transcript on an installed build. The
engine itself was healthy; every connection to it was broken.

Installed builds shipped no speech engine at all: the packaging config had no
entry for the faster-whisper sidecar and no pipeline step built one, so the app
always fell back to a system Python without the runtime. Development was broken
too, because the sidecar and SoX paths were resolved against the Vite output
directory instead of the app root, which also meant recording failed with a SoX
ENOENT. On hosts where localhost resolves only to IPv6, every local request was
refused outright, which silently disabled both local transcription and the local
LLM.

The sidecar is now built and bundled (including the Silero VAD data it needs),
gated by a packaging check that fails when the engine or its data is missing.
Paths are discovered from the app root and fail loudly when the engine is
absent. Local engine URLs are normalized to the IPv4 loopback, decoding is tuned
so repeated hallucinations cannot compound (the same transcript now takes about
a fifth of the time), the engine is warmed up at startup, and holding the hotkey
now shows the text forming live in the recording tip.
2026-09-18 00:48:47 +09:00

144 lines
19 KiB
Markdown

# 11 — Gap & Backlog Register
> The maintained list of what is **under-developed, deferred, or externally blocked**.
> Status: living document. Every feature change updates this file (see [`12-update-protocol.md`](./12-update-protocol.md)).
> External items are marked `EXT`; they block "done" but must not block code, tests, or local fixtures.
Legend: `[ ]` open · `[~]` in progress · `[!]` blocked externally · `[x]` resolved (keep for one cycle, then prune).
---
## 0. How to read this
- An item here is **not** a failure. It is a known state with an owner and a next step.
- When you close an item, flip it to `[x]`, add the date + evidence path, and also update `10-feature-catalog.md`.
- Grandfathered detail lives in `docs/v3/MOBILE_APP_COMPLETION_SSOT.md`; this file is the cross-surface roll-up. When the two disagree, the SSOT wins for mobile and must be reconciled here.
---
## 1. High impact — real capability gaps (no external blocker)
| ID | Area | Gap | Evidence | Suggested next step |
|---|---|---|---|---|
| GAP-QA-01 | Quality | Extreme Red Team: headful end-to-end bug hunting across real desktop Electron, Web Next.js, and CI pipelines. | `red_team_log.md`, `tests/e2e/red_team_cycle*.spec.ts`, `apps/web/e2e/red_team_cycle4_web.spec.ts` | `[x]` 2026-09-15: 18 scenarios executed, 14 defects caught and 100% resolved (infinite chunking loop DEF-008, IPC signature mismatch DEF-004, markdown editor typing rollback DEF-006, Web RSC Link serialization DEF-012, secret scanner lookahead DEF-013, etc.). All 18 scenarios GREEN with zero regressions. |
| GAP-REL-01 | Release | Official release publication to Forgejo and active public download center deployment. | `scripts/ci/publish-forgejo-release.mjs`, `apps/web/src/app/download/page.tsx`, `site/src/sections/Download.tsx`, `apps/web/e2e/red_team_cycle4_web.spec.ts` | `[~]` 2026-09-15: v1.1.0 release assets (`D3RO-Voice-Setup-1.1.0-x64.exe`, `.blockmap`, `latest.yml`, `update-policy.json`) published to canonical Forgejo registry and release hub. 2026-09-16: the published 1.1.0 installer carries no Authenticode signature, so it does not satisfy the release policy; product version moved to `1.2.0` and publication must come from CI with the signing gate GREEN. Download centers in `apps/web` (`/download`) and `site` (`#download`) link the canonical Forgejo feed. |
| GAP-REL-02 | Release | Windows stable publication needs an external public-trust Authenticode PFX, its password, the exact signer subject, and a Forgejo token, none of which live in the repository. | `.forgejo/workflows/release.yml`, `.gitlab-ci.yml`, `scripts/ci/verify-windows-release-artifact.ps1` | `[!]` 2026-09-16: every publisher fails closed without `WIN_CSC_*` and `FORGEJO_TOKEN`; provide them as protected CI secrets, then re-run the tag pipeline. Still open as of `1.3.0` (2026-09-18): the `v1.3.0` tag must be built by CI with the signing gate GREEN. Local packaging cannot produce a signed installer (`forceCodeSigning: true`). |
| GAP-ADS-01 | Ads | 9 of 10 desktop ad adapters still extend `UnavailableAdAdapter` (`provider_not_integrated`). | `apps/desktop/src/main/services/ads/*` | `[~]` 2026-09-13: `DirectHouseSponsorAdapter` is now a real configurable REST adapter (bid/impression/click/reward via `endpointUrl`; fail-closed when unconfigured; 22 unit tests GREEN). Remaining 9 need official SDKs/authenticated endpoints. |
| GAP-ADS-02 | Ads | Desktop mediation reward accounting is not wired to license quota (`claimReward` still returns no tokens). | `AdMediationEngine.ts`, `AppLayout.tsx` | Wire verified `reportRewardCompletion` to `LicenseService` quota after the direct sponsor endpoint exists. |
| GAP-ID-01 | Identity | Supabase, .NET JWT/SQLite, and the desktop offline license each had their own tier/role shape. | `@d3ro/core/entitlement`, `LicenseService`, `entitlement-context` | `[~]` 2026-09-13: canonical `EntitlementSnapshot` + `resolveEntitlement` added with tests; desktop tier normalization + `isPro` fixed. Full adoption tracked as GAP-ID-02. |
| GAP-ID-02 | Identity | Web and mobile still hand-roll tier/role normalization instead of consuming the canonical resolver; .NET identity is still a separate store. | `apps/web/src/lib`, `apps/mobile-rn/src/lib/entitlement-context.tsx` | Adopt `resolveEntitlement` in web/mobile; decide whether to retire the .NET user store or keep explicit mapping. |
| GAP-TEAM-01 | Teams | Team comments / activity feed was not implemented. | `server/supabase/migrations/20260913000033_team_activities.sql`, web `activity-feed.tsx`, mobile `team-service.ts` | `[x]` 2026-09-13: `team_activities` table + `create_team_activity` RPC + RLS + realtime publication; web + mobile UI. |
| GAP-MEM-01 | Knowledge | Web file upload was deferred and semantic search was mislabeled future. | `apps/web/src/components/knowledge/*` | `[x]` 2026-09-13: `.txt`/`.md` picker, newline-aware chunking, `embed-chunks` on submit, `search-knowledge` confirmed live. |
| GAP-MEM-02 | Dictionary | No dictionary import/export on any surface. | catalog MEM-07 | `[x]` 2026-09-13: desktop `dictionary:import/export` (JSON/CSV), web serialize/parse + download/upload, mobile via `data-portability`; 8 new unit tests. |
| GAP-MTG-01 | Meetings | Audio seek ↔ transcript timestamp sync missing. | SSOT H-011 | Store segment timestamps; wire player seek. |
| GAP-MTG-02 | Meetings | Speaker diarization only partially done on desktop; absent web/mobile. | `docs/phases/phase-15.5-speaker-diarization.md`, SSOT H-014 | Finish desktop pyannote path; expose speaker labels cross-surface. |
| GAP-INFRA-01 | Build | `apps/mobile-rn` is outside npm workspaces, so root `typecheck`/`test`/`lint` skip it. `typecheck:mobile`/`lint:mobile`/`test:mobile`/`verify:all` root scripts added 2026-09-13 (`package.json`), but membership/CI integration is still open. | `package.json` | Decide: add mobile to workspaces, or wire `verify:all` into CI. |
| GAP-MOB-01 | Legacy | `apps/mobile` Expo skeleton duplicated auth/data code and got version-sync edits. | SSOT G-002 | `[x]` 2026-09-13: deleted `apps/mobile`, removed from `sync-version.mjs` and `package-lock.json`; `version:check` GREEN. |
| GAP-INFRA-02 | CI | GitLab admin NAS deploy job is disabled; admin deploy is manual/GHCR. | `.gitlab-ci.yml` (admin job comments) | Re-enable with a protected environment, or document the manual runbook as canonical. |
| GAP-INFRA-03 | Release | Desktop auto-update was single-channel and pointed at GitLab while the public hub was Forgejo; no channel, mandatory-update, major-vs-delta, staging, or kill-switch policy. | `apps/desktop/src/main/update-feed.ts`, `electron-builder.yml`, `.forgejo/workflows` | `[x]` 2026-09-13: canonical Forgejo feed + legacy GitLab mirror, `release/update-policy.json` + runtime policy, `publish-forgejo-release.mjs` + Forgejo release workflow, 21 policy tests, verifier self-test 13 cases. See `docs/deployment/update-system-assessment.md`. |
| GAP-INFRA-04 | Quality | Desktop `typecheck` is a no-op: `tsconfig.json` is `files: []` + references, so `tsc --noEmit` checks nothing. Real `tsc -p tsconfig.node.json --noEmit` surfaces many pre-existing errors. | `apps/desktop/package.json`, `apps/desktop/tsconfig.json` | Switch to `tsc -b` (or per-project `-p`) and clear the existing errors in a dedicated workstream; do not treat "typecheck GREEN" as evidence until then. |
| GAP-SHELL-01 | Support | Admin `/support` is a stub panel; desktop-only `SupportModal`. No shared ticket contract. | `apps/admin/src/app/(admin)/support/page.tsx` | Define a ticket/diagnostics contract or keep stub and mark N/A in catalog. |
| GAP-PUSH-01 | Push | `send-push` accepted `webpush`/`apns` but marked them `push_provider_not_supported`. | `_shared/webpush.ts`, `_shared/apns.ts`, `send-push/index.ts` | `[x]` 2026-09-13: VAPID Web Push (RFC 8291) + APNs `.p8` transports implemented and routed; 10 new tests incl. an encryption round-trip. Client registration for those providers still pending (see GAP-PUSH-03). |
| GAP-PUSH-02 | Push | Nothing triggered `send-push?mode=drain`; enqueued notifications never left the outbox. | `server/cloudflare-worker/src/push-drain.ts`, `wrangler.toml` | `[x]` 2026-09-13: Cloudflare Cron Trigger (`* * * * *`) drains the outbox; tests in CI. Requires `SUPABASE_URL` + `SUPABASE_SERVICE_ROLE_KEY` secret on the worker. |
| GAP-PUSH-03 | Push | Mobile/web clients register only `fcm`; no service worker subscription or APNs device token. | `apps/mobile-rn/src/features/notifications/*`, `apps/web` | Add web service worker + `pushManager.subscribe` (store JSON subscription) and iOS APNs token registration. |
| GAP-PUSH-04 | Push | Android still depends on FCM (`google-services.json`). | `apps/mobile-rn/android`, `send-push` | Decide: minimal Firebase project, or UnifiedPush/ntfy. See `docs/deployment/push-transport-without-firebase.md`. |
| GAP-STT-01 | Local STT | Packaged desktop builds shipped **no** faster-whisper sidecar: `electron-builder.yml` had no `extraResources` entry for it and no pipeline job built it. Local transcription was impossible on any installed build; the app fell back to a system Python without the runtime. | `apps/desktop/electron-builder.yml`, `.gitlab-ci.yml`, `apps/desktop/scripts/build-sidecar.mjs` | `[x]` 2026-09-18: sidecar bundled via `extraResources` (`sidecar-dist/sidecar``resources/sidecar`), `sidecar:setup`/`sidecar:build` scripts, and `scripts/ci/verify-sidecar-bundle.mjs` fails packaging when the engine or its Silero VAD data is missing. Verified on the real 242 MB bundle. |
| GAP-STT-02 | Local STT | Dev resolved the sidecar/SoX paths against the Vite output dir (`out/main`), so the sidecar ran a Python without faster-whisper and SoX recording died with `spawn sox ENOENT`. | `apps/desktop/src/main/utils/paths.ts`, `apps/desktop/tests/main/utils/paths.test.ts` | `[x]` 2026-09-18: app root is discovered by walking up for app markers; packaged mode fails loudly instead of silently falling back. Unit tests pin both branches. |
| GAP-STT-03 | Local engines | On hosts where `localhost` resolves only to IPv6, every local engine call (STT sidecar and Ollama) was refused. Audio capture and local LLM appeared dead. | `apps/desktop/src/main/utils/loopback.ts`, `LocalSTTService`, `LocalLLMService`, `RAGService`, `OnlineLLMService`, `STTManager` | `[x]` 2026-09-18: loopback normalization to `127.0.0.1` for all local engine URLs; defaults updated; 9 unit tests. Verified against the live sidecar and Ollama on a host with an IPv6-only `localhost`. |
| GAP-STT-04 | Local STT | Live partial transcript (`CAP-03`, `voice:partialTranscript`) was marked done but had **no producer**: the channel, popup UI, and preload existed, nothing ever emitted. | `apps/desktop/src/main/services/VoiceModeService.ts`, `LocalSTTService.transcribePartial`, `STTManager.transcribePartial` | `[x]` 2026-09-18: 1.5 s cadence over a 7.5 s trailing window, greedy decode, drained before the final transcription; never inserted. |
| GAP-STT-05 | Local STT | The bundled sidecar lacked faster-whisper's Silero VAD data, so `vad_filter=true` transcription would have failed at runtime even with the engine bundled. | `apps/desktop/scripts/build-sidecar.mjs`, `scripts/ci/verify-sidecar-bundle.mjs` | `[x]` 2026-09-18: `--collect-all faster_whisper` plus a packaging-time presence check for `assets/silero_vad_v6.onnx`. |
| GAP-STT-06 | Local STT | Decode settings were untuned: previous-text conditioning let repeated hallucinations compound, and no VAD parameters meant slow, uneven segments. | `apps/desktop/sidecar/main.py` | `[x]` 2026-09-18: `condition_on_previous_text=false`, bounded low-temperature fallback, `no_speech`/`compression_ratio`/`log_prob` thresholds, 300 ms silence trimming. Same transcript, ~5x faster on the reference machine (7.7 s audio: 1609 ms → 303 ms). |
---
## 2. Mobile checklist roll-up (from `MOBILE_APP_COMPLETION_SSOT.md` §4)
These are the mobile SSOT rows still `[ ]` / `[~]`. Do not duplicate the full text here; open the SSOT for detail.
| Group | Open rows | Theme |
|---|---|---|
| G (governance) | G-002, G-006, G-007, G-008, G-009, G-010, G-011 | Type/error/identity consolidation, legacy bundle separation, credential rotation |
| A (auth/account) | A-001..A-013, A-018; A-014..A-017 `[~]` | Session restore, signup, deep links, OAuth, profile, provider E2E |
| O (onboarding/a11y) | O-001..O-012 | Onboarding branches, permissions, tutorial, a11y, full ko/en |
| D (data/sync/offline) | D-001..D-014 | Schema/RLS, devices, cross-surface read/write, offline queue, conflict policy |
| R (record/transcribe) | R-001..R-009, R-011..R-015, R-017..R-020; R-016 `[x]` | Permissions, recorder, FGS, upload queue, job state, sharing |
| H (history/meetings) | H-001..H-005, H-007..H-015; H-006 `[x]` | Pagination, filters, detail, meeting timeline, docs, diarization, cross-app |
| F (feature parity) | F-001..F-010, F-013, F-014; F-011/F-012 `[x]` | Dashboard, dictionary, commands, actions, chat, conversation, knowledge |
| T (teams/admin/notify) | T-001..T-005, T-009..T-012; T-006..T-008, T-013 `[x]` | Team CRUD/roles, push delivery/deep links, notification settings |
| M (monetization) | M-025, M-027; M-013 `[~]` | Play Billing license test, Payple webhook signature |
| Q (quality/release) | Q-012..Q-014, Q-019..Q-022, Q-026; Q-010/Q-011/Q-023/Q-027/Q-028 `[~]` | Emulator scripts, OAuth E2E, billing E2E, visual/a11y gates, env unification, production AAB |
---
## 3. External blockers (`EXT`) — require action outside the repo
| ID | Blocker | What is needed | Where tracked |
|---|---|---|---|
| EXT-FIREBASE-01 | No production Firebase project | Android FCM only; web/iOS can avoid Firebase (see push transport doc). Create project + Android app + Play fingerprint + FCM, or adopt UnifiedPush for Android. | SSOT EXT-011, `docs/deployment/push-transport-without-firebase.md` |
| EXT-ADMOB-01 | AdMob `검토 필요` / `광고 게재 제한` / store not linked / payment profile incomplete | Complete console review + store link + payment profile. Playwright tooling ready: `scripts/admob-login.mjs` (one interactive login on a visible desktop) then `scripts/admob-automate.mjs --apply` creates/verifies units and reports store link. Neither persisted profile is signed in yet. | SSOT §0, EXT-006 |
| EXT-PLAY-01 | Play Billing license tester + test payment method | Configure license testers | SSOT EXT-004 |
| EXT-PLAY-02 | Play product/offer/base-plan + tracks | Create Pro/Pro+ products and tracks | SSOT EXT-003 |
| EXT-PLAY-03 | Production AAB + CI secret injection + recovery backups | Inject Firebase/AdMob/signing/evidence CI secrets | SSOT EXT-008 |
| EXT-PLAY-04 | Closed test 0/12 members, 14 days; production access disabled | Run closed test, request production | SSOT EXT-010 |
| EXT-OAUTH-01 | GitHub/Apple provider secrets; real mobile Google consent→callback | Configure providers, verify consent | SSOT EXT-001 |
| EXT-APPSIGN-01 | Live App Links still old certificate | Deploy updated `assetlinks.json`, re-verify live | SSOT App Links row |
| EXT-PHYS-01 | Physical Fold6 install/OAuth/purchase evidence | User runs the artifact on device | SSOT EXT-009 |
| EXT-PAY-01 | Payple live history + webhook signature verification | Provider contract + signature scheme | SSOT M-027 |
| EXT-STRIPE-01 | Production Stripe/Payple cross-verification | Live payment E2E | SSOT M-013 |
| EXT-STT-01 | Production provider keys (Groq/OpenAI/Deepgram/Gemini) | Inject provider secrets | `apps/api-server/Program.cs` env docs |
---
## 4. Documentation & drift watch
| ID | Item | Note |
|---|---|---|
| DOC-01 | `CLAUDE.md` describes only Phase 1-15.5 desktop; does not mention web/mobile/admin/api-server. | This map supersedes it for IA; consider trimming CLAUDE.md to rules + map pointer. |
| DOC-02 | `docs/design/*` reflects an older desktop-only architecture and contains known naming drift (`06-gap-analysis.md`). | Treat this map + code as current; design docs are historical. |
| DOC-03 | `memory/project_status.md` is a chronological log, not current state. | Use `docs/map/*` for current state. |
| DOC-04 | `docs/v2/00-v2-master-plan.md` targets Expo for mobile. | Actual mobile is RN CLI (`apps/mobile-rn`); the Expo app was removed 2026-09-13. v2 plan is historical. |
| DOC-05 | This map itself must be regenerated after large refactors. | See update protocol. |
---
## 5. Quick "is X done?" lookup
- **Desktop local dictation / LLM / history / meetings / RAG / conversation:** local
dictation works in dev **and** in packaged builds as of `1.3.0` (engine bundled, paths
fixed, IPv4 loopback). Ads: one real adapter, rest stubs.
- **Web console:** yes, feature-complete for server-shared data; knowledge upload/search and team feed implemented.
- **Mobile:** code complete for most flows and tested locally; blocked mainly by external store/console gates, plus a11y and some E2E depth.
- **Backend:** fail-closed AI proxies, RLS, billing, ads SSV implemented; push transports (FCM + webpush + APNs) and cron drain implemented.
- **Admin:** complete with deliberate "unavailable" states; support page is a stub.
---
## 6. Immediate TODO — push transports, drain, AdMob (2026-09-13)
Actionable checklist for the work started this session. Fields to fill are blank in
`.env` (git-ignored) and mirrored in `.env.example`.
**Push — server (code done, config pending)**
- [ ] Set Supabase Edge secrets for the transports you deploy: `WEBPUSH_VAPID_PUBLIC_KEY`, `WEBPUSH_VAPID_PRIVATE_KEY`, `WEBPUSH_SUBJECT` (web); `APNS_KEY_ID`, `APNS_TEAM_ID`, `APNS_PRIVATE_KEY`, `APNS_TOPIC`, `APNS_ENVIRONMENT` (iOS); `FCM_SERVICE_ACCOUNT_JSON`, `FCM_PROJECT_ID` (Android).
- [ ] Deploy the updated `send-push` (`supabase functions deploy send-push`).
- [ ] Generate a VAPID keypair (P-256) and store the private/public pair; subject must be `mailto:` or `https:`.
- [ ] Create the APNs `.p8` key in the Apple Developer portal and record Key ID + Team ID.
**Push — Cloudflare cron drain**
- [ ] Set worker vars: `SUPABASE_URL` (`wrangler secret`/`[vars]`) and batch limit.
- [ ] Set worker secret: `wrangler secret put SUPABASE_SERVICE_ROLE_KEY`.
- [ ] `wrangler deploy` from `server/cloudflare-worker` and confirm the `scheduled` runs (Cloudflare dashboard → Cron Triggers).
- [ ] Verify end-to-end: enqueue a test event (e.g. team invite) → drain → delivery row `delivered`.
**Push — clients**
- [ ] Web: add a service worker + `pushManager.subscribe`, store the JSON subscription as the registration id, register `provider: 'webpush'`. (GAP-PUSH-03)
- [ ] iOS: register the APNs device token with `provider: 'apns'`. (GAP-PUSH-03)
- [ ] Android: decide FCM vs UnifiedPush/ntfy. (GAP-PUSH-04)
**AdMob (Playwright tooling ready)**
- [ ] Run `run-admob-login.bat` (or `node scripts/admob-login.mjs`) once on a visible desktop to persist the Google session.
- [ ] Run `node scripts/admob-automate.mjs --apply` to create/verify the banner + rewarded units and report the Play-store link.
- [ ] Complete AdMob console gates externally: review, serving limits, store link, payment profile. (EXT-ADMOB-01)
**Docs**
- [ ] Keep `docs/deployment/push-transport-without-firebase.md` and this file in sync when transports or clients change.