Skip to content

Benchmark report ​

Date: 2026-11-09 Frozen evidence commit: 98d8af4cd800f7720da76ab7ea6acf3b933fab03Status: harness validation, not gate results. ADR-005 keeps G0, G1, and G2 unpassed.

Next run: G0 on one vertical, with a real model and billed usage. More scripted rows do not close a gate (ADR-005, ADR-009).

This is the public-facing report. The detailed evidence is in G1-G2.md; the raw rows are in bench/results/.

1. The question ​

Does a verified map of a site beat a document-shaped browser on the metrics that matter — steps and cost per completed task — and does it survive drift?

The answer needs four arms on the same tasks:

ArmWhat it measures
Aagent-browser (or browser-use) with a model
BPortolan's view / act / read fallback with a model
CPortolan's map: open_site then call
DAn oracle: a human-written map with the same tiers

G0 compares A and a perfect D. G1 compares C against D's win. G2 breaks the site and measures repair. Without D there is no denominator, and without a real model A, B, and C are not comparable.

2. What was run ​

2.1 Compiler on the dev fixtures ​

SiteOracle opsCompiledCoverageYieldHeldTier mix (compiled)Tier mix (oracle)
store8450%100%checkouturl 3, http 1url 4, http 2, ui 2
portal4375%100%—url 2, ui 1url 3, ui 1

checkout is irreversible and was held by the publisher — the review gate working. The missing store API ops were never exercised; open_checkout is a static route and does not lift. The portal's export_csv needs download capture.

Raw: compiled-dev.md.

2.2 Held-out maps ​

SiteFrozen versionOpsTiers
library4search, get_book, borrowurl, url, ui
support5login, list_tickets, filter_tickets, paginate, create_ticketui, url, url, url, http

The seed maps author states and signatures; the compiler fills impls from fallback traces and verifies them on three fresh sessions. The freeze record carries the base commit and the model.

Raw: freeze.json, maps/heldout/.

2.3 N=3, Arm C, scripted model ​

48 rows: 16 held-out templates × 3 param sets.

ArmRunsTemplatesSuccess (95% CI)$/completed (95% CI)Steps/task (95% CI)
C4816100.0% (100.0–100.0)$0.0000 ($0.0000–$0.0000)7.8 (6.4–9.3)

The 100% and $0 are by construction: the scripted model is deterministic and makes no billed calls. The number validates the harness, maps, backend, and result schema. It says nothing about a model's success rate.

Raw: n3.jsonl, n3.md, n3-final-table.md.

2.4 Drift, scripted model ​

Opv1after switchafter repairtier
library.searchpassfailpassurl
support.loginpassfailpassui
support.createpassfailpasshttp
  • Repair rate: 100% (3/3)
  • Duplicate writes: 0
  • Drift-day success: 100%

The sample is three ops, one run, and the only write is support.create. No idempotency key is retried, so a duplicate write is not possible in this run. The semantic filter and paginate failures from the v2 fixtures are not in the sample.

Raw: drift.json, drift.md.

2.5 Latency on the store fixture ​

OpTierp50p95Budget (p50)
searchurl8.4ms9.9ms150ms
get_producturl8.5ms11.1ms150ms
list_products_apihttp1.0ms1.8ms150ms
get_product_apihttp1.8ms2.1ms150ms
add_to_cartui25.4ms27.1ms500ms

Raw: perf.json, perf.md.

2.6 Client conformance ​

ClientStatusDetail
Claude Codepasssites=3, op=search, ok=true
Gemini CLIpasssame
Codex CLIfail402 Payment Required (deactivated workspace)

The server is portolan mcp, which launches Chromium, so ok=true means the client reached a real search result through the url tier — not just the MCP surface. The fixture is the store app on 127.0.0.1:4100; the model is the client's own, so this is a client-compatibility check, not a success-rate measurement.

Raw: conformance.json, conformance.md.

3. What is missing ​

RequirementWhy it is missing
Arm A (agent-browser)No provider key; the baseline needs a model
Arm B (fallback with a model)Same
Arm D (oracle)The held-out maps exist; no oracle run with a model and billed usage
A second providerOnly scripted was configured
N=3 with billed usageEvery row is $0 by construction
Real-model failuresNo provider was called
Semantic drift (filter, paginate)Not in the three-op sample
Idempotency retry on a writeNot exercised

The local open-weights smoke test exists (bench/local/open-weights.test.ts) and is skipped without PORTOLAN_LOCAL_BASE_URL and PORTOLAN_LOCAL_MODEL. It is the smallest real-model check; it has not produced a committed row.

4. Gate status ​

GatePass conditionStatus
G0perfect ops ≥3x fewer steps and ≥5x cheaper than agent-browsernot run
G1compiled maps ≥60% of the oracle win, ≥50% yield, break-even K≤3not passed; yield is 100%, the oracle denominator is missing
G2≥80% repair, zero duplicate writesnot a valid sample; 3/3 scripted

The release gate stays closed. v0.1.0 is not released on this evidence (ADR-005). The artifacts above may be cited only as harness validations.

5. Reproduce ​

bash
pnpm install
pnpm build

pnpm --filter @abubakarsiddiik/bench pilot        # 16 rows, scripted
pnpm --filter @abubakarsiddiik/bench n3           # 48 rows, scripted
pnpm --filter @abubakarsiddiik/bench drift        # v1 -> v2 repair, scripted
pnpm --filter @abubakarsiddiik/bench conformance  # MCP clients
pnpm --filter @abubakarsiddiik/bench perf         # latency

A real run needs ANTHROPIC_API_KEY/OPENAI_API_KEY (or an OpenAI-compatible endpoint) and the same commands with a provider-backed runner. The model port is configured per role; see packages/model/src/config.ts.

6. Threats to validity ​

  • A deterministic model proves the harness, not the product. The 100% success and 7.8 steps are properties of the script.
  • No oracle denominator. G1 cannot be computed from a single arm.
  • Three drift ops is not a drift rate. A 3/3 result has a wide interval; the semantic failures are absent.
  • The fixture is local and synchronous. A real site adds network, auth, anti-bot, and dynamic content.
  • The latency sample is one machine and ten runs. It shows the tier overhead, not a production p50.
  • Conformance is two clients and one op on a local fixture. It shows the clients can drive the surface and reach a result; it is not a task sample.

7. What the next run must do ​

Per ADR-005: arms A, B, C, and D-oracle on the 16 held-out templates, two providers, a model-Y transfer run, N=3 with billed usage, oracle rows, and a drift run with real models including the semantic filter and paginate failures and an idempotency-key retry. Until then the gates stay closed.

Released under Apache-2.0 · MIT · CC-BY-4.0