feat(commands): multi-select ordered command pipeline with step reordering and single-pass composition (AI-05, AI-07)
This commit is contained in:
parent
f49acf0105
commit
10af20a0ad
157 changed files with 29455 additions and 913 deletions
18
scripts/verification/README.md
Normal file
18
scripts/verification/README.md
Normal file
|
|
@ -0,0 +1,18 @@
|
|||
# Typing-loss verification
|
||||
|
||||
These tools separate a proven service race from unverified claims about physical keys and real CPU/model load.
|
||||
|
||||
1. Run `node scripts/verification/typing-race-proof.mjs` from the checkout root. It overlays three HEAD services in memory while keeping current dependencies and tests. A successful comparison requires the old services to corrupt the next Hangul word in three independent timing cases, the current services to preserve it, and unchanged-input correction to work in both. The other 13 assertions cover UIA snapshot contracts. This is not a complete installed-release A/B.
|
||||
2. Build `typing-fixture/build.ps1` and `typing-runtime/build.ps1`. Quit the installed app and manually open the isolated runtime launcher as required by repository AGENTS.md. Do not run `run-desktop.bat` for this experiment; it kills unrelated Electron processes. Keep the test profile separate from the personal profile. The runtime receipt records the product bundle hash and product package version; the wrapper itself is not a released installer.
|
||||
3. Run the fixture with correction off, then correction on, saving each JSON under a distinct name. Use default Hangul IME mode first. Require exact calibration text and native IME composition messages. Compare all three requested paces under idle and bounded CPU load. A missing/aborted case, mismatched text, partial SendInput return or absent IME evidence is not a pass. Report achieved pacing, accepted key events and CPU samples during the input window, not just the requested load.
|
||||
4. Prove the correction path is active with Latin layout correction mode and a correction-enabled profile. A zero-loss Hangul run alone could simply mean the correction path never ran. Disable learned-text persistence throughout.
|
||||
5. For model loading, record local Ollama `/api/ps` before and after a request, request start/end, model load duration when available, and fixture input timestamps. Use only a model already installed locally, and do not unload a model being used by the user. A request made while a model is already resident is a warm-model test. A load ending before typing starts is not load-overlap proof. Record product-triggered warmup separately from direct Ollama stress.
|
||||
6. After a clean WinForms result, expand to Chromium/Electron and terminal input surfaces, then physical keyboard typing. Preserve failures and restore the normal installed app after closing the isolated runtime. Do not mark the full typing-loss issue verified on the basis of synthetic scan codes alone.
|
||||
|
||||
The fixture edits only its own text field, aborts when focus leaves, and bounds every CPU worker to 20 seconds. No clipboard or Unicode injection is used for the test corpus. The narrow check-to-SendInput focus race and synthetic-versus-hardware input remain limitations.
|
||||
|
||||
Windows contracts checked for this method:
|
||||
|
||||
- [SendInput](https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-sendinput) returns the count inserted, serializes each array, and does not reset existing key state. Do not count a short return as success or retry a partial replacement.
|
||||
- [GetLastInputInfo](https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-getlastinputinfo) is session-local and its tick value need not increase. Compare change, not wall-clock ordering.
|
||||
- [LowLevelKeyboardProc](https://learn.microsoft.com/en-us/windows/win32/winmsg/lowlevelkeyboardproc) has a timeout and can be silently removed. Zero missing text alone does not establish continued hook health; use a positive correction control.
|
||||
15
scripts/verification/typing-fixture/README.md
Normal file
15
scripts/verification/typing-fixture/README.md
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
# Windows typing fixture
|
||||
|
||||
Build without downloading dependencies:
|
||||
|
||||
```powershell
|
||||
pwsh -File scripts/verification/typing-fixture/build.ps1
|
||||
```
|
||||
|
||||
Open `test-results/typing-load/fixture/D3ROTypingFixture.exe` from the interactive desktop, then click **Start verification**. The default report is `test-results/typing-load/current-fixture.json`; an optional first executable argument chooses another report file.
|
||||
|
||||
The fixture only edits its own multiline textbox. It selects an already installed Korean input language on its own UI thread and restores that selection afterward. It never installs an IME, changes the default input method, reads other app documents, launches D3RO, or modifies D3RO configuration. Foreground/focus changes cancel injection; pressing Stop or closing the fixture cancels CPU workers. Each CPU phase stops after at most 20 seconds even if the UI is delayed.
|
||||
|
||||
Calibration must produce `안녕하세요 ` from the scan-code sequence `dkssudgktpdy ` and receive native `WM_IME_COMPOSITION` messages before any result is accepted. This prevents automatic Latin-to-Korean correction alone from masquerading as IME proof. The matrix uses 60/30/15 ms requested per-key spacing, both idle and optional CPU load. Read the recorded timestamps for achieved pacing. CPU load uses logical processor count minus one dedicated workers and records actual system/fixture CPU utilization. The sequence contains `안녕하세요 감사합니다 반갑습니다 ` repeated three times. The **Latin layout correction** option keeps the same keys and expected Korean text but switches this fixture's IME off after calibration; use this only when the D3RO configuration under test should automatically correct layout mistakes.
|
||||
|
||||
Record the application build/configuration under test in the label. Preserve each report under a distinct name before rerunning. Exact text match, native accepted-event counts, observed key counts, per-case text changes, focus validity, pacing and CPU overlap are separate evidence. Synthetic scan codes exercise OS input/IME and the global hook; they are not physical-hardware keyboard proof. This WinForms surface also does not prove Chromium/Electron or terminal behavior.
|
||||
365
scripts/verification/typing-fixture/TypingFixture.cs
Normal file
365
scripts/verification/typing-fixture/TypingFixture.cs
Normal file
|
|
@ -0,0 +1,365 @@
|
|||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Diagnostics;
|
||||
using System.Drawing;
|
||||
using System.IO;
|
||||
using System.Linq;
|
||||
using System.Runtime.InteropServices;
|
||||
using System.Text;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using System.Web.Script.Serialization;
|
||||
using System.Windows.Forms;
|
||||
|
||||
// Test fixture only: sends scan codes solely while its own textbox has focus.
|
||||
// It never changes the system/default IME, launches D3RO, or inspects user documents.
|
||||
internal static class Program
|
||||
{
|
||||
[STAThread]
|
||||
private static void Main(string[] args)
|
||||
{
|
||||
Application.EnableVisualStyles();
|
||||
Application.SetCompatibleTextRenderingDefault(false);
|
||||
string output = args.Length > 0 ? Path.GetFullPath(args[0]) : Path.GetFullPath(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "..", "current-fixture.json"));
|
||||
Application.Run(new Fixture(output));
|
||||
}
|
||||
}
|
||||
|
||||
internal sealed class Fixture : Form
|
||||
{
|
||||
private readonly ObservedTextBox input = new ObservedTextBox();
|
||||
private readonly Label status = new Label();
|
||||
private readonly Button start = new Button();
|
||||
private readonly Button stop = new Button();
|
||||
private readonly CheckBox stress = new CheckBox();
|
||||
private readonly ComboBox mode = new ComboBox();
|
||||
private readonly TextBox runLabel = new TextBox();
|
||||
private readonly string output;
|
||||
private InputLanguage originalLanguage;
|
||||
private CancellationTokenSource cancellation;
|
||||
private RunReport report;
|
||||
private CaseReport current;
|
||||
private Stopwatch clock = new Stopwatch();
|
||||
private readonly System.Windows.Forms.Timer sampler = new System.Windows.Forms.Timer();
|
||||
private ulong lastIdle, lastKernel, lastUser;
|
||||
private long lastCpuTicks, lastWallTicks;
|
||||
private int ownPid;
|
||||
|
||||
public Fixture(string reportPath)
|
||||
{
|
||||
output = reportPath;
|
||||
ownPid = Process.GetCurrentProcess().Id;
|
||||
Text = "D3RO Typing Verification — isolated input fixture";
|
||||
Width = 940;
|
||||
Height = 540;
|
||||
StartPosition = FormStartPosition.CenterScreen;
|
||||
var panel = new FlowLayoutPanel { Dock = DockStyle.Top, Height = 90, Padding = new Padding(8) };
|
||||
start.Text = "Start verification";
|
||||
start.Width = 145;
|
||||
stop.Text = "Stop";
|
||||
stop.Enabled = false;
|
||||
stop.Width = 70;
|
||||
stress.Text = "Include CPU load";
|
||||
stress.Checked = true;
|
||||
stress.Width = 145;
|
||||
mode.DropDownStyle = ComboBoxStyle.DropDownList;
|
||||
mode.Items.AddRange(new object[] { "Hangul IME", "Latin layout correction" });
|
||||
mode.SelectedIndex = 0;
|
||||
mode.Width = 165;
|
||||
runLabel.Text = "label-current-app-state";
|
||||
runLabel.Width = 210;
|
||||
panel.Controls.Add(start);
|
||||
panel.Controls.Add(stop);
|
||||
panel.Controls.Add(stress);
|
||||
panel.Controls.Add(mode);
|
||||
panel.Controls.Add(runLabel);
|
||||
panel.Controls.Add(new Label { Text = "The test types only into the box below. Changing focus aborts. No clipboard or Unicode text injection.", AutoSize = true });
|
||||
input.Multiline = true;
|
||||
input.AcceptsReturn = true;
|
||||
input.ScrollBars = ScrollBars.Vertical;
|
||||
input.Dock = DockStyle.Fill;
|
||||
input.Font = new Font("Malgun Gothic", 20);
|
||||
input.AccessibleName = "Verification input only";
|
||||
input.ImeMode = ImeMode.Hangul;
|
||||
status.Dock = DockStyle.Bottom;
|
||||
status.Height = 75;
|
||||
status.Text = "Ready. Select the app-state label, then Start verification.\r\nReport: " + output;
|
||||
Controls.Add(input);
|
||||
Controls.Add(panel);
|
||||
Controls.Add(status);
|
||||
start.Click += async delegate { await Run(); };
|
||||
stop.Click += delegate { if (cancellation != null) cancellation.Cancel(); };
|
||||
input.KeyDown += delegate { if (current != null) current.ObservedKeyDowns++; };
|
||||
input.KeyUp += delegate { if (current != null) current.ObservedKeyUps++; };
|
||||
input.TextChanged += delegate {
|
||||
if (current != null) current.Changes.Add(new TextObservation { ElapsedMs = clock.ElapsedMilliseconds, Text = input.Text });
|
||||
};
|
||||
input.ImeComposition += delegate(bool committed) {
|
||||
if (current != null) {
|
||||
current.ImeCompositionMessages++;
|
||||
if (committed) current.ImeResultMessages++;
|
||||
}
|
||||
};
|
||||
sampler.Interval = 500;
|
||||
sampler.Tick += delegate { SampleCpu(); };
|
||||
FormClosing += delegate { if (cancellation != null) cancellation.Cancel(); RestoreLanguage(); };
|
||||
}
|
||||
|
||||
private async Task Run()
|
||||
{
|
||||
if (cancellation != null) return;
|
||||
cancellation = new CancellationTokenSource();
|
||||
var token = cancellation.Token;
|
||||
start.Enabled = false;
|
||||
stop.Enabled = true;
|
||||
mode.Enabled = false;
|
||||
stress.Enabled = false;
|
||||
runLabel.Enabled = false;
|
||||
bool layout = mode.SelectedIndex == 1;
|
||||
report = new RunReport {
|
||||
StartedAtUtc = DateTime.UtcNow.ToString("o"), Label = runLabel.Text,
|
||||
ProcessId = ownPid, WindowHandle = Handle.ToInt64(), LogicalProcessors = Environment.ProcessorCount,
|
||||
Mode = layout ? "latin-layout-correction" : "hangul-ime",
|
||||
ReportPath = output, Status = "running", Injection = "SendInput KEYEVENTF_SCANCODE key-down/key-up pairs",
|
||||
Limitations = "Synthetic scan codes exercise Windows IME but do not prove hardware keyboard delivery. CPU workers are bounded; compare observed CPU samples. D3RO version/config is not inferred by this fixture."
|
||||
};
|
||||
clock.Restart();
|
||||
InitializeCpu();
|
||||
sampler.Start();
|
||||
Save();
|
||||
try {
|
||||
originalLanguage = InputLanguage.CurrentInputLanguage;
|
||||
InputLanguage korean = InputLanguage.InstalledInputLanguages.Cast<InputLanguage>().FirstOrDefault(l => l.Culture.TwoLetterISOLanguageName == "ko");
|
||||
if (korean == null) throw new InvalidOperationException("No installed Korean input language; the fixture will not install or change system settings.");
|
||||
InputLanguage.CurrentInputLanguage = korean;
|
||||
input.ImeMode = ImeMode.Hangul;
|
||||
input.Focus();
|
||||
await Task.Delay(500, token);
|
||||
report.KeyboardLayout = Native.GetKeyboardLayout(0).ToInt64().ToString("X");
|
||||
// Calibration always proves IME composition independently of the layout-correction mode.
|
||||
CaseReport calibration = await TypeCase("calibration", "dkssudgktpdy ", "안녕하세요 ", 60, false, token);
|
||||
report.CalibrationPassed = calibration.ExactMatch && calibration.ImeCompositionMessages > 0;
|
||||
if (!report.CalibrationPassed) throw new InvalidOperationException("Hangul calibration needs exact text and native IME composition messages. Result is not accepted as IME typing proof.");
|
||||
if (layout) input.ImeMode = ImeMode.Off;
|
||||
foreach (bool cpu in stress.Checked ? new bool[] { false, true } : new bool[] { false }) {
|
||||
foreach (int pace in new int[] { 60, 30, 15 }) {
|
||||
token.ThrowIfCancellationRequested();
|
||||
string keys = Repeat("dkssudgktpdy rkatkgkqslek qksrkqtmqslek ", 3);
|
||||
string expected = Repeat("안녕하세요 감사합니다 반갑습니다 ", 3);
|
||||
await TypeCase((cpu ? "cpu" : "idle") + "-" + pace, keys, expected, pace, cpu, token);
|
||||
}
|
||||
}
|
||||
report.Status = report.Cases.All(c => c.ExactMatch) ? "passed" : "failed";
|
||||
} catch (OperationCanceledException) {
|
||||
report.Status = "aborted";
|
||||
report.Error = "Cancelled or foreground/focus changed; input was stopped.";
|
||||
} catch (Exception error) {
|
||||
report.Status = "failed";
|
||||
report.Error = error.GetType().Name + ": " + error.Message;
|
||||
} finally {
|
||||
sampler.Stop();
|
||||
current = null;
|
||||
report.FinishedAtUtc = DateTime.UtcNow.ToString("o");
|
||||
report.ElapsedMs = clock.ElapsedMilliseconds;
|
||||
Save();
|
||||
RestoreLanguage();
|
||||
cancellation.Dispose();
|
||||
cancellation = null;
|
||||
start.Enabled = true;
|
||||
stop.Enabled = false;
|
||||
mode.Enabled = true;
|
||||
stress.Enabled = true;
|
||||
runLabel.Enabled = true;
|
||||
status.Text = report.Status.ToUpperInvariant() + ": " + report.Cases.Count(c => c.ExactMatch) + "/" + report.Cases.Count + " exact text matches.\r\n" + (report.Error ?? "") + "\r\n" + output;
|
||||
}
|
||||
}
|
||||
|
||||
private async Task<CaseReport> TypeCase(string name, string keys, string expected, int pace, bool load, CancellationToken token)
|
||||
{
|
||||
AssertOwnFocus(token);
|
||||
current = null;
|
||||
input.Clear(); // Only the fixture's own test field; never another app.
|
||||
await Task.Delay(200, token);
|
||||
current = new CaseReport { Name = name, PaceMs = pace, Expected = expected, Keys = keys, StartedAtUtc = DateTime.UtcNow.ToString("o"), StartedElapsedMs = clock.ElapsedMilliseconds };
|
||||
report.Cases.Add(current);
|
||||
var item = current;
|
||||
using (var workers = new CpuWorkers(load ? Math.Max(1, Environment.ProcessorCount - 1) : 0, 20000)) {
|
||||
item.CpuWorkers = workers.Count;
|
||||
if (load) await Task.Delay(900, token);
|
||||
status.Text = "Running " + name + ". Keep focus inside the test input.\r\n" + output;
|
||||
try {
|
||||
foreach (char key in keys) {
|
||||
AssertOwnFocus(token);
|
||||
long requested = clock.ElapsedMilliseconds;
|
||||
uint sent = Native.SendScanPair(key);
|
||||
item.SentEvents += (int)sent;
|
||||
item.RequestedEvents += 2;
|
||||
item.KeyTimesMs.Add(requested);
|
||||
if (sent != 2) throw new InvalidOperationException("SendInput inserted " + sent + "/2 events; no retry.");
|
||||
await Task.Delay(pace, token);
|
||||
}
|
||||
await Task.Delay(1500, token);
|
||||
AssertOwnFocus(token);
|
||||
item.Actual = input.Text;
|
||||
item.ExactMatch = string.Equals(item.Actual, item.Expected, StringComparison.Ordinal);
|
||||
item.FirstMismatchIndex = FirstMismatch(item.Expected, item.Actual);
|
||||
} finally {
|
||||
item.Actual = input.Text;
|
||||
item.ElapsedMs = clock.ElapsedMilliseconds - item.StartedElapsedMs;
|
||||
item.FinishedAtUtc = DateTime.UtcNow.ToString("o");
|
||||
Save();
|
||||
current = null;
|
||||
}
|
||||
}
|
||||
return item;
|
||||
}
|
||||
|
||||
private void AssertOwnFocus(CancellationToken token)
|
||||
{
|
||||
token.ThrowIfCancellationRequested();
|
||||
if (Native.GetForegroundWindow() != Handle || Native.GetFocus() != input.Handle) {
|
||||
cancellation.Cancel();
|
||||
throw new OperationCanceledException("Fixture textbox is not the active focused receiver.");
|
||||
}
|
||||
}
|
||||
|
||||
private void InitializeCpu()
|
||||
{
|
||||
Native.ReadSystemTimes(out lastIdle, out lastKernel, out lastUser);
|
||||
lastCpuTicks = Process.GetCurrentProcess().TotalProcessorTime.Ticks;
|
||||
lastWallTicks = Stopwatch.GetTimestamp();
|
||||
}
|
||||
|
||||
private void SampleCpu()
|
||||
{
|
||||
ulong idle, kernel, user;
|
||||
Native.ReadSystemTimes(out idle, out kernel, out user);
|
||||
ulong total = kernel - lastKernel + user - lastUser;
|
||||
double system = total > 0 ? 100.0 * (total - Math.Min(total, idle - lastIdle)) / total : 0;
|
||||
long cpu = Process.GetCurrentProcess().TotalProcessorTime.Ticks;
|
||||
long now = Stopwatch.GetTimestamp();
|
||||
double seconds = (double)(now - lastWallTicks) / Stopwatch.Frequency;
|
||||
double process = seconds > 0 ? 100.0 * (cpu - lastCpuTicks) / TimeSpan.TicksPerSecond / seconds / Environment.ProcessorCount : 0;
|
||||
report.CpuSamples.Add(new CpuSample { ElapsedMs = clock.ElapsedMilliseconds, Case = current == null ? "between" : current.Name, SystemPercent = system, FixturePercent = process });
|
||||
lastIdle = idle; lastKernel = kernel; lastUser = user; lastCpuTicks = cpu; lastWallTicks = now;
|
||||
}
|
||||
|
||||
private void Save()
|
||||
{
|
||||
Directory.CreateDirectory(Path.GetDirectoryName(output));
|
||||
var json = new JavaScriptSerializer { MaxJsonLength = 16 * 1024 * 1024 }.Serialize(report);
|
||||
File.WriteAllText(output, json, new UTF8Encoding(false));
|
||||
}
|
||||
|
||||
private void RestoreLanguage()
|
||||
{
|
||||
if (originalLanguage == null) return;
|
||||
try { InputLanguage.CurrentInputLanguage = originalLanguage; } catch { }
|
||||
originalLanguage = null;
|
||||
}
|
||||
|
||||
private static string Repeat(string value, int count) { return string.Concat(Enumerable.Repeat(value, count)); }
|
||||
private static int FirstMismatch(string expected, string actual)
|
||||
{
|
||||
for (int i = 0; i < Math.Min(expected.Length, actual.Length); i++) if (expected[i] != actual[i]) return i;
|
||||
return expected.Length == actual.Length ? -1 : Math.Min(expected.Length, actual.Length);
|
||||
}
|
||||
}
|
||||
|
||||
internal sealed class CpuWorkers : IDisposable
|
||||
{
|
||||
private readonly CancellationTokenSource stop = new CancellationTokenSource();
|
||||
private readonly List<Thread> threads = new List<Thread>();
|
||||
public int Count { get { return threads.Count; } }
|
||||
public CpuWorkers(int count, int maximumMs)
|
||||
{
|
||||
for (int i = 0; i < count; i++) {
|
||||
var thread = new Thread(delegate() {
|
||||
var timer = Stopwatch.StartNew();
|
||||
while (!stop.IsCancellationRequested && timer.ElapsedMilliseconds < maximumMs) Thread.SpinWait(10000);
|
||||
});
|
||||
thread.IsBackground = true;
|
||||
thread.Name = "D3RO fixture bounded CPU worker";
|
||||
threads.Add(thread);
|
||||
thread.Start();
|
||||
}
|
||||
}
|
||||
public void Dispose()
|
||||
{
|
||||
stop.Cancel();
|
||||
foreach (Thread thread in threads) thread.Join(1000);
|
||||
stop.Dispose();
|
||||
}
|
||||
}
|
||||
|
||||
internal sealed class ObservedTextBox : TextBox
|
||||
{
|
||||
internal event Action<bool> ImeComposition;
|
||||
protected override void WndProc(ref Message message)
|
||||
{
|
||||
if (message.Msg == 0x10f && ImeComposition != null) {
|
||||
ImeComposition((message.LParam.ToInt64() & 0x800) != 0);
|
||||
}
|
||||
base.WndProc(ref message);
|
||||
}
|
||||
}
|
||||
|
||||
internal static class Native
|
||||
{
|
||||
[StructLayout(LayoutKind.Sequential)] private struct KeyboardInput { public ushort Vk, Scan; public uint Flags, Time; public IntPtr Extra; }
|
||||
[StructLayout(LayoutKind.Explicit, Size = 40)] private struct Input { [FieldOffset(0)] public uint Type; [FieldOffset(8)] public KeyboardInput Keyboard; }
|
||||
[StructLayout(LayoutKind.Sequential)] private struct FileTime { public uint Low, High; public ulong Value { get { return ((ulong)High << 32) | Low; } } }
|
||||
[DllImport("user32.dll")] internal static extern IntPtr GetForegroundWindow();
|
||||
[DllImport("user32.dll")] internal static extern IntPtr GetFocus();
|
||||
[DllImport("user32.dll")] internal static extern IntPtr GetKeyboardLayout(uint thread);
|
||||
[DllImport("user32.dll", SetLastError = true)] private static extern uint SendInput(uint count, Input[] inputs, int size);
|
||||
[DllImport("kernel32.dll")] private static extern bool GetSystemTimes(out FileTime idle, out FileTime kernel, out FileTime user);
|
||||
internal static void ReadSystemTimes(out ulong idle, out ulong kernel, out ulong user)
|
||||
{
|
||||
FileTime i, k, u;
|
||||
if (!GetSystemTimes(out i, out k, out u)) throw new InvalidOperationException("GetSystemTimes failed");
|
||||
idle = i.Value; kernel = k.Value; user = u.Value;
|
||||
}
|
||||
internal static uint SendScanPair(char key)
|
||||
{
|
||||
ushort scan;
|
||||
if (key == ' ') scan = 0x39;
|
||||
else {
|
||||
string[] rows = { "qwertyuiop", "asdfghjkl", "zxcvbnm" };
|
||||
int[] starts = { 0x10, 0x1e, 0x2c };
|
||||
scan = 0;
|
||||
for (int row = 0; row < rows.Length; row++) {
|
||||
int index = rows[row].IndexOf(key);
|
||||
if (index >= 0) { scan = (ushort)(starts[row] + index); break; }
|
||||
}
|
||||
if (scan == 0) throw new InvalidOperationException("Unsupported synthetic corpus key");
|
||||
}
|
||||
var pair = new Input[2];
|
||||
pair[0].Type = pair[1].Type = 1;
|
||||
pair[0].Keyboard.Scan = pair[1].Keyboard.Scan = scan;
|
||||
pair[0].Keyboard.Flags = 8;
|
||||
pair[1].Keyboard.Flags = 8 | 2;
|
||||
return SendInput(2, pair, Marshal.SizeOf(typeof(Input)));
|
||||
}
|
||||
}
|
||||
|
||||
internal sealed class RunReport
|
||||
{
|
||||
public string StartedAtUtc, FinishedAtUtc, Label, Mode, ReportPath, Status, Error, Injection, Limitations, KeyboardLayout;
|
||||
public int ProcessId, LogicalProcessors;
|
||||
public long WindowHandle, ElapsedMs;
|
||||
public bool CalibrationPassed;
|
||||
public List<CaseReport> Cases = new List<CaseReport>();
|
||||
public List<CpuSample> CpuSamples = new List<CpuSample>();
|
||||
}
|
||||
internal sealed class CaseReport
|
||||
{
|
||||
public string Name, Expected, Actual, Keys, StartedAtUtc, FinishedAtUtc;
|
||||
public int PaceMs, CpuWorkers, SentEvents, RequestedEvents, ObservedKeyDowns, ObservedKeyUps, FirstMismatchIndex, ImeCompositionMessages, ImeResultMessages;
|
||||
public long StartedElapsedMs, ElapsedMs;
|
||||
public bool ExactMatch;
|
||||
public List<long> KeyTimesMs = new List<long>();
|
||||
public List<TextObservation> Changes = new List<TextObservation>();
|
||||
}
|
||||
internal sealed class TextObservation { public long ElapsedMs; public string Text; }
|
||||
internal sealed class CpuSample { public long ElapsedMs; public string Case; public double SystemPercent, FixturePercent; }
|
||||
12
scripts/verification/typing-fixture/build.ps1
Normal file
12
scripts/verification/typing-fixture/build.ps1
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
param([string]$OutputDirectory = '')
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$repoRoot = [IO.Path]::GetFullPath((Join-Path $PSScriptRoot '../../..'))
|
||||
if (-not $OutputDirectory) { $OutputDirectory = Join-Path $repoRoot 'test-results/typing-load/fixture' }
|
||||
$OutputDirectory = [IO.Path]::GetFullPath($OutputDirectory)
|
||||
$compiler = Join-Path $env:WINDIR 'Microsoft.NET/Framework64/v4.0.30319/csc.exe'
|
||||
if (-not (Test-Path -LiteralPath $compiler)) { throw 'The installed .NET Framework x64 C# compiler is required.' }
|
||||
New-Item -ItemType Directory -Path $OutputDirectory -Force | Out-Null
|
||||
$target = Join-Path $OutputDirectory 'D3ROTypingFixture.exe'
|
||||
& $compiler /nologo /target:winexe /platform:x64 /optimize+ /r:System.Windows.Forms.dll /r:System.Drawing.dll /r:System.Web.Extensions.dll "/out:$target" (Join-Path $PSScriptRoot 'TypingFixture.cs')
|
||||
if ($LASTEXITCODE -ne 0) { throw "Fixture compilation failed: $LASTEXITCODE" }
|
||||
Write-Output $target
|
||||
95
scripts/verification/typing-race-proof.mjs
Normal file
95
scripts/verification/typing-race-proof.mjs
Normal file
|
|
@ -0,0 +1,95 @@
|
|||
// Deterministic source A/B. No GUI, input injection, checkout mutation, or personal data.
|
||||
// Run from this checkout: node scripts/verification/typing-race-proof.mjs
|
||||
import { execFileSync, spawnSync } from 'node:child_process'
|
||||
import { createHash } from 'node:crypto'
|
||||
import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'
|
||||
import { createRequire } from 'node:module'
|
||||
import path from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
|
||||
const root = process.cwd()
|
||||
const desktop = path.join(root, 'apps/desktop')
|
||||
const requiredRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../..')
|
||||
if (root.toLowerCase() !== requiredRoot.toLowerCase()) throw new Error('Run from the authoritative checkout root')
|
||||
const require = createRequire(path.join(desktop, 'package.json'))
|
||||
const vitest = path.join(path.dirname(require.resolve('vitest/package.json')), 'vitest.mjs')
|
||||
const sources = [
|
||||
'apps/desktop/src/main/services/TextInsertService.ts',
|
||||
'apps/desktop/src/main/services/TypoCorrectionService.ts',
|
||||
'apps/desktop/src/main/services/InputTelemetryService.ts'
|
||||
]
|
||||
const tests = [
|
||||
'tests/main/services/typing-race-integration.test.ts',
|
||||
'tests/main/services/input-telemetry-stale-snapshot.test.ts'
|
||||
]
|
||||
const stamp = new Date().toISOString().replaceAll(':', '-').replaceAll('.', '-')
|
||||
const output = path.join(root, 'test-results/typing-load', `race-proof-${stamp}`)
|
||||
mkdirSync(output, { recursive: true })
|
||||
const sha256 = (text) => createHash('sha256').update(text).digest('hex')
|
||||
const normalized = (file) => path.resolve(file).replaceAll('\\', '/').toLowerCase()
|
||||
const head = execFileSync('git', ['rev-parse', 'HEAD'], { cwd: root, encoding: 'utf8' }).trim()
|
||||
const overrides = {}
|
||||
const evidence = sources.map((file) => {
|
||||
const baseline = execFileSync('git', ['show', `${head}:${file}`], { cwd: root, encoding: 'utf8', maxBuffer: 4_000_000 })
|
||||
const current = readFileSync(path.join(root, file), 'utf8')
|
||||
overrides[normalized(path.join(root, file))] = baseline
|
||||
return { file, baselineSha256: sha256(baseline), currentSha256: sha256(current) }
|
||||
})
|
||||
const testEvidence = tests.map((file) => ({
|
||||
file: `apps/desktop/${file}`, currentSha256: sha256(readFileSync(path.join(desktop, file), 'utf8'))
|
||||
}))
|
||||
writeFileSync(path.join(output, 'head-source-overrides.json'), JSON.stringify(overrides))
|
||||
const configPath = path.join(output, 'vitest.proof.config.mjs')
|
||||
writeFileSync(configPath, `
|
||||
import { readFileSync } from 'node:fs';
|
||||
const overrides = process.env.D3RO_RACE_BASELINE === '1'
|
||||
? JSON.parse(readFileSync(${JSON.stringify(path.join(output, 'head-source-overrides.json'))}, 'utf8')) : {};
|
||||
export default {
|
||||
plugins: [{ name: 'read-only-head-source-overlay', enforce: 'pre', load(id) {
|
||||
return overrides[id.split('?')[0].replaceAll('\\\\', '/').toLowerCase()] ?? null;
|
||||
}}],
|
||||
test: { environment: 'node', globals: true,
|
||||
setupFiles: [${JSON.stringify(path.join(desktop, 'tests/setup.ts'))}], testTimeout: 10000 },
|
||||
resolve: { alias: {
|
||||
'@d3ro/core': ${JSON.stringify(path.join(root, 'packages/core/src'))},
|
||||
'@d3ro/ui': ${JSON.stringify(path.join(root, 'packages/ui/src'))},
|
||||
'@d3ro/i18n': ${JSON.stringify(path.join(root, 'packages/i18n/src/index.tsx'))}
|
||||
}}
|
||||
};
|
||||
`)
|
||||
const runs = []
|
||||
for (const baseline of [true, false]) {
|
||||
const label = baseline ? 'baseline-head' : 'current-fix'
|
||||
const resultPath = path.join(output, `${label}.json`)
|
||||
const result = spawnSync(process.execPath, [vitest, 'run', ...tests, '--config', configPath,
|
||||
'--reporter=json', `--outputFile=${resultPath}`], {
|
||||
cwd: desktop, env: { ...process.env, D3RO_RACE_BASELINE: baseline ? '1' : '0' },
|
||||
encoding: 'utf8', timeout: 90_000, maxBuffer: 4_000_000
|
||||
})
|
||||
writeFileSync(path.join(output, `${label}.log`), `${result.stdout ?? ''}\n${result.stderr ?? ''}`)
|
||||
if (result.error) throw result.error
|
||||
if (!existsSync(resultPath)) throw new Error(`${label} did not produce a test report; inspect ${label}.log in ${output}`)
|
||||
const report = JSON.parse(readFileSync(resultPath, 'utf8'))
|
||||
runs.push({ label, exitCode: result.status, passed: report.numPassedTests, failed: report.numFailedTests,
|
||||
tests: report.testResults.flatMap((suite) => suite.assertionResults.map((test) => ({
|
||||
title: test.fullName, status: test.status, failureMessages: test.failureMessages
|
||||
}))) })
|
||||
}
|
||||
const report = {
|
||||
capturedAt: new Date().toISOString(), kind: 'deterministic-source-ab-not-gui',
|
||||
head, node: process.version, sources: evidence, tests: testEvidence, runs,
|
||||
sourcesUnchangedDuringRun: [...evidence, ...testEvidence].every(({ file, currentSha256 }) => sha256(readFileSync(path.join(root, file), 'utf8')) === currentSha256),
|
||||
baselineScope: 'Only the three listed services are loaded from HEAD; dependencies, test doubles and tests are current in both runs. This is not a full historical release test.',
|
||||
scope: 'Actual TypoCorrectionService + TextInsertService with an in-memory text sink, and actual InputTelemetryService with delayed UIA responses. Three text-preservation cases isolate timestamp, revision and native-tick guards; one unchanged-input control must still correct. The remaining 13 cases check snapshot invalidation/metadata/coalescing contracts, not 13 independent text-loss reproductions. Native keys, OS IME, model loading and CPU stress are not exercised.'
|
||||
}
|
||||
writeFileSync(path.join(output, 'report.json'), JSON.stringify(report, null, 2) + '\n')
|
||||
process.stdout.write(JSON.stringify({ output, runs: runs.map(({label, passed, failed}) => ({label, passed, failed})) }, null, 2) + '\n')
|
||||
const baseline = runs[0]
|
||||
const fixed = runs[1]
|
||||
const integratedLosses = baseline.tests.filter((test) => test.title.includes('preserves the whole next Hangul word'))
|
||||
const control = baseline.tests.filter((test) => test.title.includes('still replaces the intended typo'))
|
||||
if (!report.sourcesUnchangedDuringRun || integratedLosses.length !== 3 ||
|
||||
integratedLosses.some((test) => test.status !== 'failed' || !test.failureMessages.some((message) => message.includes('오늘 잘 됬어요 안녕됐어요 '))) ||
|
||||
control.length !== 1 || control[0].status !== 'passed' || baseline.exitCode !== 1 ||
|
||||
fixed.exitCode !== 0 || fixed.failed !== 0 || fixed.passed !== 17 ||
|
||||
fixed.passed !== baseline.passed + baseline.failed) process.exitCode = 1
|
||||
25
scripts/verification/typing-runtime/Launch.cs
Normal file
25
scripts/verification/typing-runtime/Launch.cs
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
using System;
|
||||
using System.Diagnostics;
|
||||
using System.IO;
|
||||
using System.Windows.Forms;
|
||||
|
||||
internal static class Launch {
|
||||
[STAThread] private static void Main() {
|
||||
try {
|
||||
var root = Path.GetFullPath(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "../../.."));
|
||||
if (Process.GetProcessesByName("D3RO Voice").Length > 0) {
|
||||
MessageBox.Show("Quit the installed D3RO Voice from its tray menu first, then open this launcher again. This avoids two global input hooks during verification.", "Close installed D3RO Voice first");
|
||||
return;
|
||||
}
|
||||
if (!File.Exists(Path.Combine(root, "apps/desktop/out/main/index.js"))) {
|
||||
throw new InvalidOperationException("Build the desktop product before starting verification.");
|
||||
}
|
||||
var start = new ProcessStartInfo(Path.Combine(root, "node_modules/electron/dist/electron.exe"));
|
||||
start.Arguments = "\"" + Path.Combine(root, "scripts/verification/typing-runtime") + "\" --remote-debugging-address=127.0.0.1 --remote-debugging-port=19229";
|
||||
start.WorkingDirectory = Path.Combine(root, "apps/desktop");
|
||||
start.UseShellExecute = false;
|
||||
start.EnvironmentVariables.Remove("ELECTRON_RUN_AS_NODE");
|
||||
Process.Start(start);
|
||||
} catch (Exception error) { MessageBox.Show(error.Message, "Typing verification launch failed"); }
|
||||
}
|
||||
}
|
||||
15
scripts/verification/typing-runtime/README.md
Normal file
15
scripts/verification/typing-runtime/README.md
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
# Isolated typing verification runtime
|
||||
|
||||
This runs the built desktop product with a separate local-only profile at `test-results/typing-load/profile`. It has no account, typed-text learning, startup registration or protocol registration. The actual product bootstrap, native hook, UIA reader and correction services are used. Input telemetry starts disabled; enable it through the product settings/API when a fixture is ready.
|
||||
|
||||
Build the desktop normally, then build the launcher:
|
||||
|
||||
```powershell
|
||||
pwsh -File scripts/verification/typing-runtime/build.ps1
|
||||
```
|
||||
|
||||
Quit the installed D3RO Voice through its tray menu. Open `test-results/typing-load/runtime/D3ROTypingRuntime.exe` yourself from the interactive desktop or an external terminal. The launcher refuses to run while the installed app is present. Per repository `AGENTS.md`, an agent must not background-launch Electron from a subshell.
|
||||
|
||||
The launcher uses a localhost-only CDP port 19229 for verification. Close the test app when finished and reopen the installed app. Do not copy production account configuration into this profile. Runtime PID/path evidence is stored in `test-results/typing-load/runtime.json`. Reports and compiled executables remain in ignored `test-results`, not in source control.
|
||||
|
||||
Use with `../typing-fixture`. Record the product source hashes, configuration, exact CPU overlap and model load timings with each report. Synthetic scan codes and WinForms text readback do not establish hardware keyboard or Chromium/terminal correctness.
|
||||
10
scripts/verification/typing-runtime/build.ps1
Normal file
10
scripts/verification/typing-runtime/build.ps1
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
$ErrorActionPreference = 'Stop'
|
||||
$repoRoot = [IO.Path]::GetFullPath((Join-Path $PSScriptRoot '../../..'))
|
||||
$output = Join-Path $repoRoot 'test-results/typing-load/runtime'
|
||||
$compiler = Join-Path $env:WINDIR 'Microsoft.NET/Framework64/v4.0.30319/csc.exe'
|
||||
if (-not (Test-Path -LiteralPath $compiler)) { throw 'The installed .NET Framework x64 C# compiler is required.' }
|
||||
New-Item -ItemType Directory -Path $output -Force | Out-Null
|
||||
$target = Join-Path $output 'D3ROTypingRuntime.exe'
|
||||
& $compiler /nologo /target:winexe /platform:x64 /reference:System.Windows.Forms.dll "/out:$target" (Join-Path $PSScriptRoot 'Launch.cs')
|
||||
if ($LASTEXITCODE -ne 0) { throw "Runtime launcher compilation failed: $LASTEXITCODE" }
|
||||
Write-Output $target
|
||||
34
scripts/verification/typing-runtime/main.cjs
Normal file
34
scripts/verification/typing-runtime/main.cjs
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
// Interactive verification entry: real built product with an isolated local-only profile.
|
||||
const { app } = require('electron')
|
||||
const fs = require('node:fs')
|
||||
const path = require('node:path')
|
||||
const { createHash } = require('node:crypto')
|
||||
const root = path.resolve(__dirname, '../../..')
|
||||
const profile = path.join(root, 'test-results/typing-load/profile')
|
||||
fs.mkdirSync(profile, { recursive: true })
|
||||
app.setPath('userData', profile)
|
||||
app.setAppLogsPath(path.join(profile, 'logs'))
|
||||
app.setAppPath(path.join(root, 'apps/desktop'))
|
||||
process.env.NODE_ENV = 'test'
|
||||
// A test launch must never take ownership of the installed OAuth protocol handler.
|
||||
app.setAsDefaultProtocolClient = () => false
|
||||
app.setLoginItemSettings = () => { throw new Error('OS startup registration is disabled in the isolated verification runtime') }
|
||||
const configPath = path.join(profile, 'd3ro-voice-config.json')
|
||||
if (!fs.existsSync(configPath)) fs.writeFileSync(configPath, JSON.stringify({
|
||||
appUsageMode: 'local', onboardingCompleted: true, autoLaunch: false,
|
||||
closeToTray: false, soundEnabled: false, inputTelemetryEnabled: false,
|
||||
inputLearnTypedText: false, suggestionEnabled: false,
|
||||
typoAutoCorrectEnabled: false, typoAutoCorrectUseModel: false,
|
||||
llmModelId: 'gemma4:e4b', ollamaServerUrl: 'http://127.0.0.1:11434',
|
||||
sttModelId: 'tiny', sttProviderConfigs: {local: {modelId: 'tiny'}},
|
||||
localModelsDir: path.join(process.env.APPDATA, 'd3ro-voice/whisper-models')
|
||||
}, null, 2))
|
||||
const productBundle = path.join(root, 'apps/desktop/out/main/index.js')
|
||||
const productPackage = JSON.parse(fs.readFileSync(path.join(root, 'apps/desktop/package.json'), 'utf8'))
|
||||
fs.writeFileSync(path.join(root, 'test-results/typing-load/runtime.json'), JSON.stringify({
|
||||
pid: process.pid, startedAt: new Date().toISOString(), profile,
|
||||
productBundle, productPackageVersion: productPackage.version,
|
||||
productBundleSha256: createHash('sha256').update(fs.readFileSync(productBundle)).digest('hex'),
|
||||
isolation: 'local-only empty profile; protocol and OS startup registration disabled; real product bootstrap'
|
||||
}, null, 2))
|
||||
require(productBundle)
|
||||
1
scripts/verification/typing-runtime/package.json
Normal file
1
scripts/verification/typing-runtime/package.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"name":"d3ro-typing-verification","version":"1.0.0","private":true,"main":"main.cjs"}
|
||||
Loading…
Add table
Add a link
Reference in a new issue