{
"markdown": "# five46\n\n[](https://www.npmjs.com/package/five46)\n[](https://www.npmjs.com/package/five46)\n[](./LICENSE)\n[](https://www.npmjs.com/package/five46)\n\n**An autonomous AI testing agent that verifies your app or API actually works while you're still building it — fully local, using your own LLM key.**\n\nYou just changed something, and you want to know — right now, against the real running thing — whether it actually works, without first writing a test yourself. Give five46 a plain-English goal — \"log in and confirm the dashboard loads,\" \"create a user via POST, then confirm it via GET\" — and an LLM, using your own OpenAI, Anthropic, Gemini, Groq, or AWS Bedrock key, drives your real app or real API, one real action at a time, and tells you honestly whether it worked, with a root-cause hypothesis if it didn't. Once it does, that exact run is captured as a real, standalone Playwright (or `node:test`) spec you keep — so the same check that helped you while you were building the feature becomes a permanent regression test afterward, with no five46 or LLM involved in ever running it again.\n\n\n\n> **Status:** early proof of concept, verified end-to-end against real live LLM keys across dozens of real-world sites and APIs.\n\nIf five46 is useful to you, a ⭐ on [GitHub](https://github.com/sekharsdet/five46) helps other people find it — much appreciated!\n\n## Why five46, and how it's different\n\nMost testing tools assume you already have a suite to run. five46 is built for the moment *before* that — mid-feature, before a test exists at all. Point it at what you're building, describe the outcome you expect in plain English, and keep re-running it as you keep changing code; once it's solid, the run it just did becomes your regression test, not a separate thing you write afterward.\n\nMost AI-driven test-generation tools also run in a cloud sandbox: your app's traffic, screenshots, and DOM leave your machine and go through a third-party service you don't control. five46 is the opposite bet — **everything runs on your laptop**, using a key you already pay for, and the *only* thing that ever leaves your machine is the text sent to your chosen LLM provider on each step (always disclosed, never hidden). If your organization can't adopt a cloud-hosted AI testing platform for compliance or trust reasons, this is built for exactly that constraint.\n\nIt's also not a black box: every run ends with a real `.spec.ts`/`.test.mjs` file you can read, diff, commit to your repo, and run in CI with plain `npx playwright test` — no vendor lock-in, no proprietary runner.\n\n## Features\n\n- **Bring your own key (BYOK)** — OpenAI, Anthropic, Gemini, Groq, or AWS\n Bedrock. Your key, your usage, your cost.\n- **Fully local** — no cloud sandbox, no tunneling for local dev servers.\n Nothing but the LLM calls ever leaves your machine.\n- **Browser and API testing** — drive a real Chromium browser, or drive\n real HTTP requests directly, from the same agentic engine.\n- **Real, standalone output** — every successful run writes a plain\n Playwright `.spec.ts` (or `node:test` script for API tests) you can\n re-run any time, with no five46 or LLM involved.\n- **Session reuse** — log in once, capture the session, reuse it across\n runs without paying the LLM cost of logging in every time.\n- **Self-healing selectors** — a stale selector gets one bounded, disclosed\n recovery attempt instead of just failing the step.\n- **Resilient generated specs** — when a real, live check confirms\n Playwright's own `getByRole()` resolves uniquely to the exact element a\n step acted on, the generated spec prefers it over a positional CSS\n selector, since it's far more resistant to future DOM changes. Falls back\n to the always-correct selector automatically wherever that check can't be\n made — never changes what the live run itself does.\n- **Root-cause hypotheses** — a failed assertion gets an LLM-generated\n hypothesis for what likely went wrong and what to check next.\n- **MCP server** — expose `five46_test`/`five46_api` as tools an\n IDE-embedded AI assistant (Claude Code, Cursor, etc.) can call directly.\n- **Safe by default** — API testing is read-only unless you explicitly\n unlock writes/deletes; destructive-looking browser clicks are blocked by\n default too.\n- **Flaky-test detection** — `--repeat N` runs the same goal N times and\n reports whether the outcome/behavior actually stayed the same.\n- **Diffing** — `five46 diff` compares two generated run files directly.\n- **Project management** — `five46.config.json` + `--project` for\n reusable, named target defaults (url, session, safety flags).\n- **Video replay** — `--record-video` records the whole session as a\n `.webm`.\n- **Structured planning** — on by default, one extra upfront LLM call plans\n the whole goal, then most steps execute directly against the real\n page/response with no further live decision needed; `--no-structured-plan`\n opts back into the fully-adaptive, live-decision-every-step loop.\n- **Fast per-step decisions** (`--fast-steps`, opt-in) — on Groq/Gemini,\n swaps in a genuinely faster model tier for the high-frequency per-step\n decision only; the upfront plan always uses your configured model. No\n effect on OpenAI/Anthropic/Bedrock, already at their fastest reliable\n tier. Opt-in, not default — see \"Fast per-step decisions\" below.\n- **Story mode** (`--story`) — splits a raw, multi-AC user story into\n independent goals and runs them with bounded concurrency, reporting a\n clear pass/fail per acceptance criterion. See \"Story mode\" below.\n- **Cross-run action cache** (`--action-cache`, opt-in) — skips the upfront\n planning LLM call on a repeat run of the same goal by reusing a\n previously-saved plan, falling back to a fresh plan automatically if the\n page has changed. See \"Cross-run action cache\" below.\n\n## five46 vs. cloud AI testing platforms\n\n| | five46 | Typical cloud AI testing platform |\n|---|---|---|\n| Where it runs | Your machine, fully local | Their cloud sandbox |\n| What leaves your machine | Only the text sent to your LLM provider per step (disclosed) | Your app's traffic, screenshots, DOM, credentials |\n| Pricing model | BYOK — you pay your LLM provider directly, at cost | Usage-based platform subscription on top of their own LLM cost |\n| Output | A real, standalone `.spec.ts`/`.test.mjs` file you own, re-runnable with plain Playwright/`node:test` | Usually tied to their own runner/dashboard |\n| Best fit | Teams that can't send app data to a third party, or want to run tests entirely offline/on-prem | Teams that want a managed, zero-setup service and don't mind the tradeoff |\n\nNot a knock on cloud platforms — it's a genuinely different tradeoff (their infra vs. your own key and your own machine), and the right choice depends on what your organization is allowed to send off-machine.\n\n## Installation\n\n```bash\nnpm install -g five46\n\nnpm install --save-dev playwright @playwright/test # one-time, if your project doesn't already have it\nnpx playwright install chromium # one-time, downloads the browser\n```\n\nOr run it without installing globally:\n\n```bash\nnpx five46 test http://localhost:3000 --goal \"log in and confirm the dashboard loads\"\n```\n\n\nBuilding from source instead (for contributing to five46 itself)
\n\n```bash\ngit clone https://github.com/sekharsdet/five46.git\ncd five46\nnpm install\nnpm run build\nnode dist/cli.js test http://localhost:3000 --goal \"...\"\n```\n\n \n\n## Configuration\n\nOne-time setup (same shape as `gh auth login`/`aws configure`):\n\n```bash\nfive46 config\n```\n\nThis prompts for an LLM provider + key, masking secret input, and saves it\nto `~/.five46/config.json` (user-only file permissions). Or set\nenvironment variables instead — these always take priority over the saved\nconfig, which is useful for CI:\n\n```bash\nexport FIVE46_LLM_PROVIDER=openai # or: anthropic, gemini, groq, bedrock\nexport FIVE46_LLM_API_KEY=sk-... # for bedrock, use your AWS region instead\n```\n\n### Getting a key\n\nDon't have a key yet? Pick whichever's easiest to get, or whichever you\nalready use — five46 calls one small, cheap model per provider on every\nstep (never a \"flagship\" model), so per-run cost is low regardless of\nwhich one you pick. **If wall-clock speed is what you care about most,\npick Groq** — its whole differentiator is LPU-based inference hardware\nbuilt specifically for fast token generation, meaningfully faster\nround-trips than typical GPU-hosted inference for an equivalent-size\nmodel. Since a run's time is dominated by LLM round-trip latency (not\nfive46's own code), the provider you pick is the single biggest lever\nyou control over how fast a run feels:\n\n| Provider | Get a key at | Notes |\n|---|---|---|\n| **Groq** | [console.groq.com/keys](https://console.groq.com/keys) → \"Create API Key\" | Free tier, no credit card required, generous rate limits — also the fastest provider here, built on inference-optimized hardware. |\n| **Gemini** | [aistudio.google.com](https://aistudio.google.com/apikey) → \"Get API key\" | Free tier, no credit card required — the fastest path to a first successful run. |\n| **OpenAI** | [platform.openai.com/api-keys](https://platform.openai.com/api-keys) → \"Create new secret key\" | Account creation is free, but a key can't make real calls until you add a payment method — no meaningful free tier. |\n| **Anthropic** | [console.anthropic.com/settings/keys](https://console.anthropic.com/settings/keys) → \"Create Key\" | Same shape as OpenAI — you can browse the console for free, but need billing set up before a key actually works. |\n| **AWS Bedrock** | No key — see below | Uses your existing AWS credentials instead of an API key. |\n\nThe model each provider calls: `gpt-4o-mini` (OpenAI), `claude-3-5-haiku-latest`\n(Anthropic), `gemini-flash-latest` (Gemini), `llama-3.3-70b-versatile` (Groq),\n`anthropic.claude-3-5-haiku-20241022-v1:0` (Bedrock).\n\n**AWS Bedrock is different — there's no key to paste in:**\n\n1. In the [Bedrock console](https://console.aws.amazon.com/bedrock/home) →\n **Model access**, request/enable access to the Claude model above, in\n the region you plan to use.\n2. Make sure AWS credentials are available the normal way — five46 relies\n on the standard AWS SDK credential chain, same as the AWS CLI:\n `aws configure`, `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` env vars, or\n an IAM role.\n3. Run `five46 config`, choose `bedrock`, and enter your **region**\n (e.g. `us-east-1`) when prompted — not a key.\n\n## Quick start\n\n```bash\nfive46 test http://localhost:3000 --goal \"log in and confirm the dashboard loads\"\n```\n\n`--goal` is required. Useful flags: `--max-steps` (default 15),\n`--headed` (watch it drive a real visible browser instead of headless),\n`--out` (spec path), `--allow-deletes` (allow clicking destructive-looking\nelements, e.g. \"Delete Account\"), `--no-root-cause` (skip the extra LLM\ncall that analyzes a failed assertion), `--repeat N` (run the goal N times\nand report whether it's flaky — see below), `--record-video` (save a\n`.webm` of the whole session), `--project name` (pull defaults from\n`five46.config.json` — see below), `--no-structured-plan` (opt out of the\ndefault upfront-plan-then-fast-path behavior and use the fully-adaptive,\nlive-decision-every-step loop instead — see below), `--fast-steps` (opt-in,\nuse a faster model for per-step decisions on Groq/Gemini — see below),\n`--story path` (split a raw multi-AC user story into independent goals and\nrun them concurrently instead of a single `--goal` — see below),\n`--action-cache` (opt-in, skip the upfront planning LLM call on a repeat\nrun of the same goal by reusing a previously-saved plan — see below).\n\nA successful run writes a real, human-readable Playwright `.spec.ts` file\ncontaining every confirmed-working step — re-runnable any time via\n`npx playwright test`. The run itself is **not deterministic** (the same\ngoal against the same page can take a different path next time); the\ngenerated spec is the frozen, repeatable artifact.\n\nA failed assertion is reported as a real finding about the app (with a\nscreenshot, DOM snapshot, and a root-cause hypothesis), clearly separated\nfrom a tooling hiccup (an unparseable LLM response, a stuck/repeating\nagent) — the two are never conflated.\n\n**Exit codes are CI-friendly**: `five46 test`/`five46 api` exit 0 only\nwhen the goal was actually reached, and non-zero for anything else\n(a failed assertion, a stuck/looping run, a missing API key, ...) —\nso `five46 test --goal \"...\" || exit 1` in a CI script works as\nexpected.\n\n## Testing behind a login\n\nCapture a session once, reuse it across runs:\n\n```bash\nexport FIVE46_LOGIN_USERNAME=...\nexport FIVE46_LOGIN_PASSWORD=...\n\nfive46 login https://your-app.example.com/login --goal \"log in\" --out session.json\nfive46 test https://your-app.example.com/dashboard --goal \"...\" --storage-state session.json\n```\n\nYour username/password are never sent to the LLM — the model only ever\nsees placeholder tokens; the real values are substituted locally at the\npoint Playwright actually types them. `session.json` is itself a live\nbearer credential — treat it like one: don't commit it (it's written with\nuser-only file permissions).\n\n## API/backend testing\n\nNo browser involved — the same agentic engine drives real HTTP requests\ntoward a goal instead, and writes a real, standalone `node:test` script\n(plain `node:test` + `node:assert` + native `fetch`, no Playwright\nneeded):\n\n```bash\nfive46 api https://api.your-app.example.com --goal \"create a user, then fetch it back and confirm the name matches\"\n```\n\nRead-only (`GET`/`HEAD`/`OPTIONS`) by default. Add `--allow-writes` to\nunlock `POST`/`PUT`/`PATCH`, and `--allow-deletes` to separately unlock\n`DELETE`. Requests are restricted to the target's own origin unless you\nname another one via repeatable `--allow-host `.\n\n## Listing past runs\n\n```bash\nfive46 list # current directory\nfive46 list ./tests # or any other directory\nfive46 list --project checkout # only runs tagged with this project\n```\n\nLists previously generated `five46-agent-*.spec.ts`/`five46-api-*.test.mjs`\nfiles with their goal and outcome, most recent first. No separate \"rerun\"\ncommand — every generated file already is a real, standalone Playwright/\n`node:test` file: `npx playwright test ` / `node --test `.\n\n## Diffing two runs\n\n```bash\nfive46 diff five46-agent-abc123.spec.ts five46-agent-def456.spec.ts\n```\n\nA plain line diff between any two generated (or other text) files, with\nthe header's run-id token ignored (the outcome half of that same line is\nstill compared). Exits 0 if identical, 1 if they differ.\n\n## Flaky-test detection\n\n```bash\nfive46 test http://localhost:3000 --goal \"...\" --repeat 5\n```\n\nRuns the same goal N times (sequentially — capped at 10) and reports\nwhether it's flaky: either the outcome differed across runs, or every run\nreached the goal but took a genuinely different path. Exits 0 only if\nevery repeat succeeded with byte-identical generated output. Works the\nsame way on `five46 api`.\n\n## Project management\n\n```json\n// five46.config.json\n{\n \"projects\": {\n \"checkout\": { \"url\": \"http://localhost:3000/checkout\", \"storageState\": \"session.json\" }\n }\n}\n```\n\n```bash\nfive46 test --goal \"...\" --project checkout\n```\n\nA CLI flag always wins over a project default; a project only fills in\nwhat you didn't pass. `--goal` is never project-configurable. The LLM API\nkey is never sourced from this file — only a provider label can be.\n\n## Video replay\n\n```bash\nfive46 test http://localhost:3000 --goal \"...\" --record-video\n```\n\nRecords the whole session as a real `.webm` (also available on\n`five46 login`). No special \"replay\" command — open the file in any video\nplayer.\n\n## Structured planning\n\nOn by default. One extra LLM call plans the whole goal upfront; most steps\nthen execute directly against the real page/response with no further live\ndecision — falling back to a normal live decision only when a step's\nprediction doesn't resolve cleanly. Same safety guarantees as an ordinary\nrun (destructive-click gating, method/host allowlisting) are enforced\nindependently at the fast path too, not skipped. This is the single biggest\nlever for cutting a run's wall-clock time, since LLM round-trip latency —\nnot five46's own code — is the dominant per-run cost.\n\nConfirming an outcome fast-paths too, not just navigating to it: a planned\n`assert_visible` step fast-paths under the same rule as clicks/fills (its\ntarget must resolve to exactly one real element, or it falls back to a live\ndecision) — the visibility check itself still runs for real against the\nlive page either way, nothing is assumed. `assert_text`/`assert_value`/\n`assert_page_text`/`assert_page_text_absent` always make a live decision,\nsince they'd also need to predict an exact expected text/value for a page\nthe plan never saw, a bigger risk than predicting an element's role/name.\nA well-formed goal can often complete\nwith a single LLM call total (the upfront plan) if every step, including\nthe final confirmation, fast-paths.\n\n```bash\nfive46 test http://localhost:3000 --goal \"...\" --no-structured-plan\n```\n\n`--no-structured-plan` opts back into the fully-adaptive loop (a live\ndecision every single step) — works the same way on `five46 api`. Note:\nthis default applies to the `test`/`api` CLI commands only — an ordinary\nMCP-driven `goal`/`story` call always uses the fully-adaptive loop instead,\nsince structured planning isn't independently exposed as an MCP tool\nparameter. The one exception: an MCP call that supplies `steps` (see below)\nforces structured planning on for that call, since that's what grounds it.\n\n## Cross-run action cache\n\n```bash\nfive46 test http://localhost:3000 --goal \"...\" --action-cache\n```\n\nOpt-in, off by default, `test`/`api` CLI commands only (no MCP equivalent).\nRequires structured planning to be on (the default) — it's an alternative\n*source* for the upfront plan, not a parallel mechanism, so it has nothing\nto do with `--no-structured-plan` set. On a `goal-reached` run, the whole\nplan is saved to `~/.five46/cache.json`, keyed by project/goal/URL; the\n*next* run of that same goal skips the upfront planning LLM call entirely\nand reuses the cached plan — worthwhile when you're re-running the same\ncheck repeatedly (e.g. in a tight edit-test loop) and want to shave off\nthat call's latency/cost every time. Every step is still resolved live\nagainst the real page/response before it runs, exactly like an ordinary\nstructured-plan run — a stale cache (the page genuinely changed since it\nwas written) is detected automatically and just falls back to a fresh live\nplan call, never a wrong guess. Disabled automatically during `--repeat`,\nsince flakiness detection needs a genuinely fresh live decision on every\nrepeat.\n\n## Fast per-step decisions\n\n```bash\nfive46 test http://localhost:3000 --goal \"...\" --fast-steps\n```\n\nOpt-in — off by default. On Groq and Gemini, swaps in a genuinely faster\nmodel tier (`llama-3.1-8b-instant`, `gemini-flash-lite-latest`) for the\nhigh-frequency per-step action-decision call only; the one-time upfront\nplan (and the root-cause hypothesis call, if triggered) always use your\nconfigured model, never the fast one. On OpenAI, Anthropic, and Bedrock\nthis flag has no effect — each is already at the fastest model tier that\nprovider offers without risking reliability on the strict JSON-only action\nschema.\n\nThis is a real tradeoff, not a free win: a smaller/faster model is a\ngenuine, currently-unquantified risk to per-step decision quality (more\nwrong-ref picks or retries), which is exactly why it's opt-in rather than\ndefault like structured planning. If wall-clock speed matters most to you,\nalso consider picking Groq as your provider in the first place — see\n\"Getting a key\" above.\n\n## Story mode\n\n```bash\nfive46 test http://localhost:3000 --story user-story.txt --concurrency 3\n```\n\nA real user story or Jira ticket often bundles several acceptance criteria\ntogether — often independent, sometimes mutually-exclusive scenarios\n(\"checkout succeeds with valid info,\" \"checkout fails with an invalid\ncoupon\") that can't coexist in one linear run. `--story ` reads the\nraw story text, splits it into independent goals with one extra upfront LLM\ncall, then runs each one exactly like an ordinary `--goal` run — same\nsafety gating, same structured planning/`--fast-steps` speed path, its own\nfresh browser session/artifact directory, its own generated spec file —\nwith up to `--concurrency` running at once (default 2 for `five46 test`,\n3 for `five46 api` — a real browser session against one origin carries a\nheavier, more bot-like footprint than a plain HTTP request, so the browser\ndefault is lower; both are hard-capped at 5). Mutually exclusive with\n`--goal` — pass exactly one. Works the same way on `five46 api`.\n\n```\nSplit into 2 scenario(s):\n AC1: log in with valid credentials and complete checkout, then confirm the order confirmation is shown\n AC2: attempt checkout with no shipping details entered, then confirm a validation error is shown\n\n=== Story mode summary ===\nAC1: PASS — log in with valid credentials and complete checkout, then confirm the order confirmation is shown\nAC2: PASS — attempt checkout with no shipping details entered, then confirm a validation error is shown\n\n2/2 acceptance criteria reached goal-reached.\n```\n\nExit code is all-or-nothing, matching `--repeat`'s own CI philosophy: 0\nonly if every acceptance criterion reached `goal-reached`. Splitting is\nitself an LLM call and could occasionally group scenarios in an unintended\nway — a malformed or unusable split response degrades safely to treating\nthe whole story as a single goal, never worse than not using `--story` at\nall.\n\n## MCP server (IDE-embedded use)\n\n```bash\nnpm install --save-dev @modelcontextprotocol/sdk zod # one-time\nfive46 mcp\n```\n\nExposes `five46_test`/`five46_api` as MCP tools an IDE-embedded AI\nassistant can call directly. Read-only by default, with no per-call way\nto unlock writes — set `FIVE46_MCP_ALLOW_WRITES=1`/\n`FIVE46_MCP_ALLOW_DELETES=1` in the server's own environment to unlock\nthem; tool arguments can never do it. `five46 login` is deliberately not\nexposed via MCP.\n\nBoth tools also accept an optional `story` field alongside `goal` (exactly\none required) — the same story-mode splitting/bounded-concurrency described\nabove, letting a coding agent hand over a whole multi-AC story it just\nimplemented a feature against and get back a per-AC pass/fail. Concurrency\nis set via `FIVE46_MCP_CONCURRENCY` in the server's own environment —\nnever a per-call tool argument, the same posture as\n`allowWrites`/`allowDeletes` — with the same per-tool defaults as the CLI\n(2 for `five46_test`, 3 for `five46_api`) when left unset, both hard-capped\nat 5.\n\nBoth tools also return `structuredContent` — a machine-parseable\n`{ passed, outcome, specPath }` (or `{ passed, acceptanceCriteria: [...] }`\nfor a `story` call) alongside the existing free-text report, so a calling\ncoding agent can branch on a real field instead of parsing prose out of the\nreport to decide whether to rework or move on.\n\nBoth tools also accept an optional `steps` field alongside `goal` — an\nordered checklist (1-50 items, `{ type: \"action\"|\"assertion\", description }`)\nof what you already know must happen, e.g. from having just read or written\nthe flow being tested:\n\n```json\n{\n \"url\": \"http://localhost:3000/cart\",\n \"goal\": \"guest can complete checkout\",\n \"steps\": [\n { \"type\": \"action\", \"description\": \"Click 'Add to cart' on the first product\" },\n { \"type\": \"action\", \"description\": \"Go to /checkout and submit the test payment\" },\n { \"type\": \"assertion\", \"description\": \"A confirmation page shows an order number\" }\n ]\n}\n```\n\nfive46 still resolves every step against the real, live page itself — a\nstep never skips discovery, it only saves five46 from having to invent the\nsequence from `goal` alone. Requires `goal` (the objective `steps`\nrefines); mutually exclusive with `story` (one checklist doesn't map onto\nseveral independent scenarios). Forces structured-plan mode on for that\ncall, so it costs the one extra upfront LLM call that mode always does.\n\n## Known limitations\n\n- **No *closed* shadow DOM traversal** — elements inside a `mode: \"closed\"`\n shadow root aren't visible to the agent's page snapshot (by design, a\n closed shadow root is deliberately inaccessible to external tooling in\n general, not just five46). An *open* shadow root (the overwhelming\n majority of real-world web components) works fine, same as `