yunchan/d3ro-voice
1
0
Fork 0
Code Issues Pull requests Projects Releases 2 Packages Wiki Activity Actions
d3ro-voice/docs/map/11-gap-backlog.md
Yun Chan 2d585bfc29 feat(desktop): make local speech transcription work end to end 
Local dictation had never produced a transcript on an installed build. The
engine itself was healthy; every connection to it was broken.

Installed builds shipped no speech engine at all: the packaging config had no
entry for the faster-whisper sidecar and no pipeline step built one, so the app
always fell back to a system Python without the runtime. Development was broken
too, because the sidecar and SoX paths were resolved against the Vite output
directory instead of the app root, which also meant recording failed with a SoX
ENOENT. On hosts where localhost resolves only to IPv6, every local request was
refused outright, which silently disabled both local transcription and the local
LLM.

The sidecar is now built and bundled (including the Silero VAD data it needs),
gated by a packaging check that fails when the engine or its data is missing.
Paths are discovered from the app root and fail loudly when the engine is
absent. Local engine URLs are normalized to the IPv4 loopback, decoding is tuned
so repeated hallucinations cannot compound (the same transcript now takes about
a fifth of the time), the engine is warmed up at startup, and holding the hotkey
now shows the text forming live in the recording tip.
2026-09-18 00:48:47 +09:00

19 KiB
Raw Blame History

11 — Gap & Backlog Register

The maintained list of what is under-developed, deferred, or externally blocked. Status: living document. Every feature change updates this file (see 12-update-protocol.md). External items are marked EXT; they block "done" but must not block code, tests, or local fixtures.

Legend: [ ] open · [~] in progress · [!] blocked externally · [x] resolved (keep for one cycle, then prune).

0. How to read this

  • An item here is not a failure. It is a known state with an owner and a next step.
  • When you close an item, flip it to [x], add the date + evidence path, and also update 10-feature-catalog.md.
  • Grandfathered detail lives in docs/v3/MOBILE_APP_COMPLETION_SSOT.md; this file is the cross-surface roll-up. When the two disagree, the SSOT wins for mobile and must be reconciled here.

1. High impact — real capability gaps (no external blocker)

ID Area Gap Evidence Suggested next step
GAP-QA-01 Quality Extreme Red Team: headful end-to-end bug hunting across real desktop Electron, Web Next.js, and CI pipelines. red_team_log.md, tests/e2e/red_team_cycle*.spec.ts, apps/web/e2e/red_team_cycle4_web.spec.ts [x] 2026-09-15: 18 scenarios executed, 14 defects caught and 100% resolved (infinite chunking loop DEF-008, IPC signature mismatch DEF-004, markdown editor typing rollback DEF-006, Web RSC Link serialization DEF-012, secret scanner lookahead DEF-013, etc.). All 18 scenarios GREEN with zero regressions.
GAP-REL-01 Release Official release publication to Forgejo and active public download center deployment. scripts/ci/publish-forgejo-release.mjs, apps/web/src/app/download/page.tsx, site/src/sections/Download.tsx, apps/web/e2e/red_team_cycle4_web.spec.ts [~] 2026-09-15: v1.1.0 release assets (D3RO-Voice-Setup-1.1.0-x64.exe, .blockmap, latest.yml, update-policy.json) published to canonical Forgejo registry and release hub. 2026-09-16: the published 1.1.0 installer carries no Authenticode signature, so it does not satisfy the release policy; product version moved to 1.2.0 and publication must come from CI with the signing gate GREEN. Download centers in apps/web (/download) and site (#download) link the canonical Forgejo feed.
GAP-REL-02 Release Windows stable publication needs an external public-trust Authenticode PFX, its password, the exact signer subject, and a Forgejo token, none of which live in the repository. .forgejo/workflows/release.yml, .gitlab-ci.yml, scripts/ci/verify-windows-release-artifact.ps1 [!] 2026-09-16: every publisher fails closed without WIN_CSC_* and FORGEJO_TOKEN; provide them as protected CI secrets, then re-run the tag pipeline. Still open as of 1.3.0 (2026-09-18): the v1.3.0 tag must be built by CI with the signing gate GREEN. Local packaging cannot produce a signed installer (forceCodeSigning: true).
GAP-ADS-01 Ads 9 of 10 desktop ad adapters still extend UnavailableAdAdapter (provider_not_integrated). apps/desktop/src/main/services/ads/* [~] 2026-09-13: DirectHouseSponsorAdapter is now a real configurable REST adapter (bid/impression/click/reward via endpointUrl; fail-closed when unconfigured; 22 unit tests GREEN). Remaining 9 need official SDKs/authenticated endpoints.
GAP-ADS-02 Ads Desktop mediation reward accounting is not wired to license quota (claimReward still returns no tokens). AdMediationEngine.ts, AppLayout.tsx Wire verified reportRewardCompletion to LicenseService quota after the direct sponsor endpoint exists.
GAP-ID-01 Identity Supabase, .NET JWT/SQLite, and the desktop offline license each had their own tier/role shape. @d3ro/core/entitlement, LicenseService, entitlement-context [~] 2026-09-13: canonical EntitlementSnapshot + resolveEntitlement added with tests; desktop tier normalization + isPro fixed. Full adoption tracked as GAP-ID-02.
GAP-ID-02 Identity Web and mobile still hand-roll tier/role normalization instead of consuming the canonical resolver; .NET identity is still a separate store. apps/web/src/lib, apps/mobile-rn/src/lib/entitlement-context.tsx Adopt resolveEntitlement in web/mobile; decide whether to retire the .NET user store or keep explicit mapping.
GAP-TEAM-01 Teams Team comments / activity feed was not implemented. server/supabase/migrations/20260913000033_team_activities.sql, web activity-feed.tsx, mobile team-service.ts [x] 2026-09-13: team_activities table + create_team_activity RPC + RLS + realtime publication; web + mobile UI.
GAP-MEM-01 Knowledge Web file upload was deferred and semantic search was mislabeled future. apps/web/src/components/knowledge/* [x] 2026-09-13: .txt/.md picker, newline-aware chunking, embed-chunks on submit, search-knowledge confirmed live.
GAP-MEM-02 Dictionary No dictionary import/export on any surface. catalog MEM-07 [x] 2026-09-13: desktop dictionary:import/export (JSON/CSV), web serialize/parse + download/upload, mobile via data-portability; 8 new unit tests.
GAP-MTG-01 Meetings Audio seek ↔ transcript timestamp sync missing. SSOT H-011 Store segment timestamps; wire player seek.
GAP-MTG-02 Meetings Speaker diarization only partially done on desktop; absent web/mobile. docs/phases/phase-15.5-speaker-diarization.md, SSOT H-014 Finish desktop pyannote path; expose speaker labels cross-surface.
GAP-INFRA-01 Build apps/mobile-rn is outside npm workspaces, so root typecheck/test/lint skip it. typecheck:mobile/lint:mobile/test:mobile/verify:all root scripts added 2026-09-13 (package.json), but membership/CI integration is still open. package.json Decide: add mobile to workspaces, or wire verify:all into CI.
GAP-MOB-01 Legacy apps/mobile Expo skeleton duplicated auth/data code and got version-sync edits. SSOT G-002 [x] 2026-09-13: deleted apps/mobile, removed from sync-version.mjs and package-lock.json; version:check GREEN.
GAP-INFRA-02 CI GitLab admin NAS deploy job is disabled; admin deploy is manual/GHCR. .gitlab-ci.yml (admin job comments) Re-enable with a protected environment, or document the manual runbook as canonical.
GAP-INFRA-03 Release Desktop auto-update was single-channel and pointed at GitLab while the public hub was Forgejo; no channel, mandatory-update, major-vs-delta, staging, or kill-switch policy. apps/desktop/src/main/update-feed.ts, electron-builder.yml, .forgejo/workflows [x] 2026-09-13: canonical Forgejo feed + legacy GitLab mirror, release/update-policy.json + runtime policy, publish-forgejo-release.mjs + Forgejo release workflow, 21 policy tests, verifier self-test 13 cases. See docs/deployment/update-system-assessment.md.
GAP-INFRA-04 Quality Desktop typecheck is a no-op: tsconfig.json is files: [] + references, so tsc --noEmit checks nothing. Real tsc -p tsconfig.node.json --noEmit surfaces many pre-existing errors. apps/desktop/package.json, apps/desktop/tsconfig.json Switch to tsc -b (or per-project -p) and clear the existing errors in a dedicated workstream; do not treat "typecheck GREEN" as evidence until then.
GAP-SHELL-01 Support Admin /support is a stub panel; desktop-only SupportModal. No shared ticket contract. apps/admin/src/app/(admin)/support/page.tsx Define a ticket/diagnostics contract or keep stub and mark N/A in catalog.
GAP-PUSH-01 Push send-push accepted webpush/apns but marked them push_provider_not_supported. _shared/webpush.ts, _shared/apns.ts, send-push/index.ts [x] 2026-09-13: VAPID Web Push (RFC 8291) + APNs .p8 transports implemented and routed; 10 new tests incl. an encryption round-trip. Client registration for those providers still pending (see GAP-PUSH-03).
GAP-PUSH-02 Push Nothing triggered send-push?mode=drain; enqueued notifications never left the outbox. server/cloudflare-worker/src/push-drain.ts, wrangler.toml [x] 2026-09-13: Cloudflare Cron Trigger (* * * * *) drains the outbox; tests in CI. Requires SUPABASE_URL + SUPABASE_SERVICE_ROLE_KEY secret on the worker.
GAP-PUSH-03 Push Mobile/web clients register only fcm; no service worker subscription or APNs device token. apps/mobile-rn/src/features/notifications/*, apps/web Add web service worker + pushManager.subscribe (store JSON subscription) and iOS APNs token registration.
GAP-PUSH-04 Push Android still depends on FCM (google-services.json). apps/mobile-rn/android, send-push Decide: minimal Firebase project, or UnifiedPush/ntfy. See docs/deployment/push-transport-without-firebase.md.
GAP-STT-01 Local STT Packaged desktop builds shipped no faster-whisper sidecar: electron-builder.yml had no extraResources entry for it and no pipeline job built it. Local transcription was impossible on any installed build; the app fell back to a system Python without the runtime. apps/desktop/electron-builder.yml, .gitlab-ci.yml, apps/desktop/scripts/build-sidecar.mjs [x] 2026-09-18: sidecar bundled via extraResources (sidecar-dist/sidecar → resources/sidecar), sidecar:setup/sidecar:build scripts, and scripts/ci/verify-sidecar-bundle.mjs fails packaging when the engine or its Silero VAD data is missing. Verified on the real 242 MB bundle.
GAP-STT-02 Local STT Dev resolved the sidecar/SoX paths against the Vite output dir (out/main), so the sidecar ran a Python without faster-whisper and SoX recording died with spawn sox ENOENT. apps/desktop/src/main/utils/paths.ts, apps/desktop/tests/main/utils/paths.test.ts [x] 2026-09-18: app root is discovered by walking up for app markers; packaged mode fails loudly instead of silently falling back. Unit tests pin both branches.
GAP-STT-03 Local engines On hosts where localhost resolves only to IPv6, every local engine call (STT sidecar and Ollama) was refused. Audio capture and local LLM appeared dead. apps/desktop/src/main/utils/loopback.ts, LocalSTTService, LocalLLMService, RAGService, OnlineLLMService, STTManager [x] 2026-09-18: loopback normalization to 127.0.0.1 for all local engine URLs; defaults updated; 9 unit tests. Verified against the live sidecar and Ollama on a host with an IPv6-only localhost.
GAP-STT-04 Local STT Live partial transcript (CAP-03, voice:partialTranscript) was marked done but had no producer: the channel, popup UI, and preload existed, nothing ever emitted. apps/desktop/src/main/services/VoiceModeService.ts, LocalSTTService.transcribePartial, STTManager.transcribePartial [x] 2026-09-18: 1.5 s cadence over a 7.5 s trailing window, greedy decode, drained before the final transcription; never inserted.
GAP-STT-05 Local STT The bundled sidecar lacked faster-whisper's Silero VAD data, so vad_filter=true transcription would have failed at runtime even with the engine bundled. apps/desktop/scripts/build-sidecar.mjs, scripts/ci/verify-sidecar-bundle.mjs [x] 2026-09-18: --collect-all faster_whisper plus a packaging-time presence check for assets/silero_vad_v6.onnx.
GAP-STT-06 Local STT Decode settings were untuned: previous-text conditioning let repeated hallucinations compound, and no VAD parameters meant slow, uneven segments. apps/desktop/sidecar/main.py [x] 2026-09-18: condition_on_previous_text=false, bounded low-temperature fallback, no_speech/compression_ratio/log_prob thresholds, 300 ms silence trimming. Same transcript, ~5x faster on the reference machine (7.7 s audio: 1609 ms → 303 ms).

2. Mobile checklist roll-up (from MOBILE_APP_COMPLETION_SSOT.md §4)

These are the mobile SSOT rows still [ ] / [~]. Do not duplicate the full text here; open the SSOT for detail.

Group Open rows Theme
G (governance) G-002, G-006, G-007, G-008, G-009, G-010, G-011 Type/error/identity consolidation, legacy bundle separation, credential rotation
A (auth/account) A-001..A-013, A-018; A-014..A-017 [~] Session restore, signup, deep links, OAuth, profile, provider E2E
O (onboarding/a11y) O-001..O-012 Onboarding branches, permissions, tutorial, a11y, full ko/en
D (data/sync/offline) D-001..D-014 Schema/RLS, devices, cross-surface read/write, offline queue, conflict policy
R (record/transcribe) R-001..R-009, R-011..R-015, R-017..R-020; R-016 [x] Permissions, recorder, FGS, upload queue, job state, sharing
H (history/meetings) H-001..H-005, H-007..H-015; H-006 [x] Pagination, filters, detail, meeting timeline, docs, diarization, cross-app
F (feature parity) F-001..F-010, F-013, F-014; F-011/F-012 [x] Dashboard, dictionary, commands, actions, chat, conversation, knowledge
T (teams/admin/notify) T-001..T-005, T-009..T-012; T-006..T-008, T-013 [x] Team CRUD/roles, push delivery/deep links, notification settings
M (monetization) M-025, M-027; M-013 [~] Play Billing license test, Payple webhook signature
Q (quality/release) Q-012..Q-014, Q-019..Q-022, Q-026; Q-010/Q-011/Q-023/Q-027/Q-028 [~] Emulator scripts, OAuth E2E, billing E2E, visual/a11y gates, env unification, production AAB

3. External blockers (EXT) — require action outside the repo

ID Blocker What is needed Where tracked
EXT-FIREBASE-01 No production Firebase project Android FCM only; web/iOS can avoid Firebase (see push transport doc). Create project + Android app + Play fingerprint + FCM, or adopt UnifiedPush for Android. SSOT EXT-011, docs/deployment/push-transport-without-firebase.md
EXT-ADMOB-01 AdMob 검토 필요 / 광고 게재 제한 / store not linked / payment profile incomplete Complete console review + store link + payment profile. Playwright tooling ready: scripts/admob-login.mjs (one interactive login on a visible desktop) then scripts/admob-automate.mjs --apply creates/verifies units and reports store link. Neither persisted profile is signed in yet. SSOT §0, EXT-006
EXT-PLAY-01 Play Billing license tester + test payment method Configure license testers SSOT EXT-004
EXT-PLAY-02 Play product/offer/base-plan + tracks Create Pro/Pro+ products and tracks SSOT EXT-003
EXT-PLAY-03 Production AAB + CI secret injection + recovery backups Inject Firebase/AdMob/signing/evidence CI secrets SSOT EXT-008
EXT-PLAY-04 Closed test 0/12 members, 14 days; production access disabled Run closed test, request production SSOT EXT-010
EXT-OAUTH-01 GitHub/Apple provider secrets; real mobile Google consent→callback Configure providers, verify consent SSOT EXT-001
EXT-APPSIGN-01 Live App Links still old certificate Deploy updated assetlinks.json, re-verify live SSOT App Links row
EXT-PHYS-01 Physical Fold6 install/OAuth/purchase evidence User runs the artifact on device SSOT EXT-009
EXT-PAY-01 Payple live history + webhook signature verification Provider contract + signature scheme SSOT M-027
EXT-STRIPE-01 Production Stripe/Payple cross-verification Live payment E2E SSOT M-013
EXT-STT-01 Production provider keys (Groq/OpenAI/Deepgram/Gemini) Inject provider secrets apps/api-server/Program.cs env docs

4. Documentation & drift watch

ID Item Note
DOC-01 CLAUDE.md describes only Phase 1-15.5 desktop; does not mention web/mobile/admin/api-server. This map supersedes it for IA; consider trimming CLAUDE.md to rules + map pointer.
DOC-02 docs/design/* reflects an older desktop-only architecture and contains known naming drift (06-gap-analysis.md). Treat this map + code as current; design docs are historical.
DOC-03 memory/project_status.md is a chronological log, not current state. Use docs/map/* for current state.
DOC-04 docs/v2/00-v2-master-plan.md targets Expo for mobile. Actual mobile is RN CLI (apps/mobile-rn); the Expo app was removed 2026-09-13. v2 plan is historical.
DOC-05 This map itself must be regenerated after large refactors. See update protocol.

5. Quick "is X done?" lookup

  • Desktop local dictation / LLM / history / meetings / RAG / conversation: local dictation works in dev and in packaged builds as of 1.3.0 (engine bundled, paths fixed, IPv4 loopback). Ads: one real adapter, rest stubs.
  • Web console: yes, feature-complete for server-shared data; knowledge upload/search and team feed implemented.
  • Mobile: code complete for most flows and tested locally; blocked mainly by external store/console gates, plus a11y and some E2E depth.
  • Backend: fail-closed AI proxies, RLS, billing, ads SSV implemented; push transports (FCM + webpush + APNs) and cron drain implemented.
  • Admin: complete with deliberate "unavailable" states; support page is a stub.

6. Immediate TODO — push transports, drain, AdMob (2026-09-13)

Actionable checklist for the work started this session. Fields to fill are blank in .env (git-ignored) and mirrored in .env.example.

Push — server (code done, config pending)

  • Set Supabase Edge secrets for the transports you deploy: WEBPUSH_VAPID_PUBLIC_KEY, WEBPUSH_VAPID_PRIVATE_KEY, WEBPUSH_SUBJECT (web); APNS_KEY_ID, APNS_TEAM_ID, APNS_PRIVATE_KEY, APNS_TOPIC, APNS_ENVIRONMENT (iOS); FCM_SERVICE_ACCOUNT_JSON, FCM_PROJECT_ID (Android).
  • Deploy the updated send-push (supabase functions deploy send-push).
  • Generate a VAPID keypair (P-256) and store the private/public pair; subject must be mailto: or https:.
  • Create the APNs .p8 key in the Apple Developer portal and record Key ID + Team ID.

Push — Cloudflare cron drain

  • Set worker vars: SUPABASE_URL (wrangler secret/[vars]) and batch limit.
  • Set worker secret: wrangler secret put SUPABASE_SERVICE_ROLE_KEY.
  • wrangler deploy from server/cloudflare-worker and confirm the scheduled runs (Cloudflare dashboard → Cron Triggers).
  • Verify end-to-end: enqueue a test event (e.g. team invite) → drain → delivery row delivered.

Push — clients

  • Web: add a service worker + pushManager.subscribe, store the JSON subscription as the registration id, register provider: 'webpush'. (GAP-PUSH-03)
  • iOS: register the APNs device token with provider: 'apns'. (GAP-PUSH-03)
  • Android: decide FCM vs UnifiedPush/ntfy. (GAP-PUSH-04)

AdMob (Playwright tooling ready)

  • Run run-admob-login.bat (or node scripts/admob-login.mjs) once on a visible desktop to persist the Google session.
  • Run node scripts/admob-automate.mjs --apply to create/verify the banner + rewarded units and report the Play-store link.
  • Complete AdMob console gates externally: review, serving limits, store link, payment profile. (EXT-ADMOB-01)

Docs

  • Keep docs/deployment/push-transport-without-firebase.md and this file in sync when transports or clients change.