Browser automation
with the limits in view.

PAIRED BROWSER PROVIDER RUN · CPU

The same DOM tasks, with both local providers.

Five seeds repeated the same eight MiniWoB tasks for each provider on Windows 10 and a Ryzen 5 5600H CPU. All 40 episodes per provider reached full task reward without a harness timeout or an outcome disagreement. Only 11 episodes per provider called the semantic model; all other selections came from exact local rules.

In this CPU run, NLI's model-call p50 was about 4.7× MiniLM's (810 ms vs 173 ms). The median paired delta for the full browser loop was only +6 ms because model inference occurred in 11 of 40 episodes and environment reset dominates total time. Every episode starts a fresh bridge, so these model calls include process-level pipeline initialization; this does not measure warm requests to a persistent MCP server, GPU latency, or broad task quality. Confidence estimates remain uncalibrated. The measurement supports no general speed claim.

EXTERNAL YES/NO VALIDATION · FULL 3,270-ROW SPLIT

Neither local model beat the majority baseline.

We ran the default local MiniLM and optional local NLI providers on every example in the pinned Google BoolQ validation split. Each item combines a passage with a human-annotated yes/no question. The three conditions ran on Windows 10, Node 22.14.0, AMD Ryzen 5 5600H; all 3,270 examples are paired. The split's majority answer is “yes” (2,033/3,270, or 62.2%).

All three conditions fell below the 62.2% majority baseline (MiniLM CPU -5.2 points; MiniLM DirectML -5.0; NLI CPU -12.4). DirectML returned six more correct answers than MiniLM CPU, while their settings disagreed on 302 predictions. CPU NLI answered 1,626 correctly versus 1,864 for MiniLM and disagreed on 1,728 predictions; MiniLM alone was correct on 983 paired cases, NLI alone on 745 paired cases. NLI's warm p95 was 145 ms versus 49 ms for MiniLM CPU. These are one run per model and do not establish statistical significance or broad quality. A separate profile on 30 evenly spaced validation rows recorded 5,340 DmlExecutionProvider and 60 CPUExecutionProvider node events (5,400 total). This confirms mixed provider execution for those inputs; event counts do not measure compute share. Confidence is uncalibrated; Brier and ECE are descriptive.

Dataset: Google BoolQ validation split, pinned revision 35b264d03638db9f4ce671b711558bf7ff0f80d5, licensed CC BY-SA 3.0; see the BoolQ paper and data license notice. Dataset passages and questions were used locally and are not included in the reports. Model overlap with pretraining data is unknown. MiniLM reports were recorded at commit ed1cbf7, NLI at 6073828; the model-cache directory existed before these runs, but individual files were not checked.

LOCAL DECISION DEVICE CHECK

CPU was faster on this small model.

Each device row summarizes five matched runs over the same 30 project-specific labels on one Windows 10 Ryzen 5 5600H laptop with a GeForce GTX 1650 (4 GB, reported by nvidia-smi). The model-cache directory existed before each process, but individual cached files were not checked. A separate DirectML profile records which ONNX execution provider handled each node.

Across these five pairs, CPU warm p95 had a 21 ms median (20–21 ms across run p95s); DirectML had a 138 ms median (137–149 ms) and was slower in every pair. The nearest-rank p95 across all 150 CPU calls was 21 ms; across all 150 DirectML-requested calls it was 148 ms. A separate 30-call ONNX profile observed 5,340 DmlExecutionProvider node events and 60 CPUExecutionProvider events (178 and 2 per call). This verifies DirectML execution with some CPU fallback, but node counts do not measure compute share. Every warm p95 in these runs stayed below 500 ms; DirectML first-call latency included initialization, with a 474 ms median and 693 ms maximum. This one machine and non-held-out fixture do not establish a general speed guarantee, full browser latency target, or GPU advantage. Confidence estimates remain uncalibrated; ECE is descriptive only.

NLI DEVICE REPEATS · WINDOWS CPU AND DIRECTML

DirectML was slower for this NLI workload.

Five separate Node processes per device ran the same 30 project-authored confirmation labels with Xenova/nli-deberta-v3-small. The cache directory existed before each process, though individual files were not checked. The host was Windows 10, Node 22.14.0, Ryzen 5 5600H, and an operator-reported GTX 1650 (4,096 MiB shown by nvidia-smi). Reports identify commit c3449c3 and an unmodified tracked tree.

CPU was faster in all five matched pairs. Its median warm p95 was 83 ms; with DirectML requested it was 654 ms, about 7.9× slower. First calls include pipeline initialization. CPU and DirectML outputs were stable across their five repeated processes; the two device settings disagreed on 11 of 30 predictions in the paired outputs. Choice was 6/10 on CPU and 7/10 with DirectML; yes/no was 8/10 for each, and Score MAE was 1.146 vs 1.089. Repeating these same project-authored labels does not create independent quality examples; this is not a held-out quality estimate.

The same device pair in the MiniWoB browser loop

Five seeds repeated the same eight DOM tasks per device (40 episodes each). Every episode earned full reward, there were no timeouts, and paired task outcomes disagreed on 0 of 40. Only 11 of 40 episodes called NLI; the other episodes used exact local rules.

DirectML-requested NLI calls had p95 1456 ms versus 827 ms on CPU. Browser-loop p95 was 1601 ms versus 1087 ms; end-to-end p95 including reset was 2916 ms versus 2405 ms. The paired median deltas were +4 ms for the browser loop and +18 ms end to end. This curated set does not show warm persistent-server latency or broad task quality; NLI remains an optional provider, and these runs do not validate the 500 ms decision-call target.

A separate instrumented 30-example DirectML run recorded 43,500 DmlExecutionProvider and 43,800 CPUExecutionProvider node-execution events (87,300 total). That verifies mixed DirectML and CPU execution; event counts do not measure compute share. Profiling adds overhead and is excluded from the latency table. The run demonstrates neither a general GPU speed advantage nor performance parity across machines.

DEFAULT MINILM PROVIDER · CPU AND DIRECTML

DirectML was slower for MiniLM on this browser run.

The current commit a9f8c46 ran the same eight selected DOM tasks across five seeds on Windows 10, with a Ryzen 5 5600H and an operator-reported GTX 1650 (4 GB). Each condition completed 40 episodes; every task reached full reward, and paired outcomes disagreed on 0 of 40 pairs. Eleven semantic model calls were recorded per condition; the other episodes used local rules.

The paired median DirectML-minus-CPU delta was +209 ms for model calls, +8.5 ms for the browser loop, and +0 ms end to end. DirectML's decision-call p95 was 387 ms versus 179 ms on CPU. This browser run did not record ONNX per-node placement; a separate decision-only profile records mixed DirectML and CPU execution. The small selected sample does not establish a general GPU speed advantage or warm persistent-server latency.