QA your app
Portolan QA turns your coding agent into the test author. Cases are JSON files under qa/, reviewed in git. Runs are deterministic — no model in the run path — with typed results, evidence, and a local SQLite history under .portolan/.
The agent authors and repairs the tests; the runner replays them.
Set up
Inside your project (the app under test):
portolan qa init --base-url http://127.0.0.1:3000This writes portolan.config.json, the qa/cases and qa/suites directories, the portolan-qa skill, and the MCP entry for a detected harness. Restart the harness so the QA tools load. portolan qa doctor checks config, database, and the browser.
The loop
- After a change, call
qa_plan. It returns the cases the diff touches (touchesglobs and routes) and any that are not verified. - Run the selection with
qa_run. Results are typed:fail_codeis the contract the agent branches on (not_found,stale_ref,assertion_failed,value_lost, …). - On a locator failure (
not_found,stale_ref,not_interactive), callqa_repair. It replays to the failure, recovers the moved or renamed target by name similarity, rewrites the case as a draft, records a drift event, and confirms with one run. - Fix the app or the expectation, then rerun.
Authoring
One case per file, qa/cases/<id>.json, the id matching the path:
{
"schema": 1,
"id": "auth/login",
"title": "Login",
"suite": "smoke",
"effect": "read",
"steps": [
{ "goto": "/sign-in" },
{ "do": "fill", "target": { "role": "textbox", "name": "Email" }, "value": "${fixture.user.email}" },
{ "do": "click", "target": { "role": "button", "name": "Login" } }
],
"expect": [{ "url": "/dashboard" }]
}The agent can also hand qa_generate a loose step list (from_spec) or a recorded trace file (from_trace). Generate fills the defaults, and replaces secret-looking fill values with ${env.QA_*} references — a recorded password never lands in a case file.
Fixtures resolve at run time from qa/fixtures.json; secrets are env references. Evidence is redacted before it is written.
Trust
- A generated or repaired case is always
draft. qa_verifyreplays a case N times on fresh sessions; only then is itverified, and the record lives in the database, not the JSON.- Editing a verified case demotes it to draft. That is correct.
qa_runrefuses unverified cases unless told otherwise.
Tools
| Tool | Purpose |
|---|---|
qa_list | Case index with status, suite, tags |
qa_plan | Cases affected by the current diff |
qa_run | Execute a selection; typed per-case results |
qa_result | One case's failure: expected vs actual, evidence paths, next[] |
qa_case_read / qa_case_write | Read or write a case file |
qa_generate | Normalize a spec or a recorded trace into a draft |
qa_verify | Replay N times; the only path to verified |
qa_repair | Recover a broken locator from a failed run |
qa_coverage | Routes covered by cases, and the gaps |
qa_report | JSON, markdown, or JUnit XML for a run |
The same surface is on the CLI as portolan qa <command>.
The full design — storage model, schema, runner contract, trust lifecycle — is in docs/QA.md.