{
  "markdown": "# qes-exploration\n\n[![engine tests](https://github.com/HiddenTrail/exploratory-testing-engine/actions/workflows/engine-tests.yml/badge.svg)](https://github.com/HiddenTrail/exploratory-testing-engine/actions/workflows/engine-tests.yml)\n\nAn LLM-based **disconfirmation engine** for exploratory API testing: instead of\nrunning a fixed, pre-scripted test plan, it drives a live system, forms a\nfalsifiable hypothesis about its behavior, and puts that hypothesis through a\ncold, adversarial review before trusting it - mirroring how real scientific\nmethod works rather than how most AI-testing tools work (which mostly confirm,\nrarely try hard to disprove themselves). See\n[`docs/exploratory-testing-engine-concept.md`](docs/exploratory-testing-engine-concept.md)\nfor the original vision this project is one deliberately narrow, implemented\nslice of.\n\n## How a run works\n\nAgainst a live system under test (SUT), each **checkpoint**:\n\n1. **Casts** a batch of real tests (an adapter-defined test-proposal schema),\n   executes them for real, and records predicted vs. actual outcomes.\n2. Forms **one hypothesis** about the system's behavior and any anomalies\n   noticed - a specific, falsifiable claim per anomaly, not a vague suspicion.\n3. Gets a **cold Skeptic review** of that hypothesis: a second LLM call that\n   never sees the raw test data, only the hypothesis itself. It checks\n   whether the cited evidence actually discriminates the claim from its own\n   named rival explanation - not just whether evidence exists - and returns a\n   `weak` (keep going) or `strong_enough` (stop) verdict.\n4. The loop continues on `weak`, informed by the Skeptic's critique, or stops\n   on `strong_enough` or a checkpoint cap.\n\nIf the final hypothesis claims anomalies, a bug report is written per claim -\nhonestly marked `inconclusive` if the checkpoint budget ran out while the\nSkeptic still had objections, `corroborated` only if it was satisfied. Output\nis a JSON result, a JSON bug list, and a self-contained HTML report.\n\n## Bootstrapping a new adapter automatically\n\nTesting a new API normally means hand-writing an *adapter* (see below). The\n`engine/bootstrap/` pipeline can generate a first draft of one instead, by\nactually pointing itself at a live system and working out the schema for real:\n\n```\nDiscover      → try the live SUT's own OpenAPI/Swagger doc first (free, exact)\nDraft (LLM)   → only if discovery found nothing: infer a schema from free text\nProbe (LLM)   → send real requests to confirm/correct the draft against\n                 the live system's actual responses - a 422 naming a missing\n                 field resolves an unknown more reliably than a guess would\nGenerate      → emit a real, runnable adapter.py from what was confirmed\n```\n\nRun all four phases end to end with:\n\n```\npython -m engine.bootstrap.cli \\\n  --name my_api --display-name \"My API\" --base-url http://localhost:8000\n```\n\nThis never auto-registers or auto-runs the result - it prints the line to add\nto `engine/adapters/registry.py` and the command to run it, keeping\nregistration a deliberate human step. A generated adapter that never achieved\na real success is refused outright (`status == \"failed\"`); one that ran out\nof probing budget while still uncertain is generated anyway, with a\nprominent warning comment carrying forward exactly what's still unconfirmed.\nSee [`docs/examples/bootstrap_demo/`](docs/examples/bootstrap_demo/) for a\nreal, unedited run of this pipeline - including the adapter it generated and\nthe bug it found.\n\nOther flags:\n- `--discover-only` - stop after Phase 1 (schema discovery) and write\n  `runs/<name>/discovered_schema.{json,html}`, then exit - no LLM calls at\n  all, so it's a free way to check what a SUT publishes before spending any\n  probing/generation budget on it.\n- `--spec-text <text>` - a schema-inference fallback, read only if Phase 1\n  discovery finds nothing at all (i.e. no OpenAPI/Swagger doc).\n- `--max-probes N` - cap on Phase 3 probing rounds (default 8).\n\n### Context-enriched bootstrap\n\nBeyond the schema itself, free-text background - what the API does, its\nnormal use cases - can be supplied and gets threaded into both Phase 3\nprobing and the generated adapter (baked into its schema doc and system\nprompt, so it persists into every future checkpoint-loop run against that\nadapter, not just the one-off bootstrap):\n\n```\n--context-source file --context-file <path>     # (default) read a plain text file\n--context-source jira --ticket <id>              # read a MOCKED ticket store\n```\n\nThis is a 4-phase roadmap, being built incrementally:\n1. ✅ Thread context into probing.\n2. ✅ Persist context into the generated adapter.\n3. ✅ Prove the source is swappable via a mocked `jira` ticket store\n   (`engine/bootstrap/jira_mock.py`) - no real JIRA calls.\n4. ⏳ Not started - real JIRA API integration (auth, live ticket fetch),\n   left as a `TODO` in `jira_mock.py` until explicitly requested.\n\n## Ontology layer (prioritization)\n\n`engine/ontology/` sits between the domain-grounded oracle claims and the\nDriver: a 4-layer flat-file stack (generic heuristic library → per-SUT\nbusiness/domain facts → context/results → a ranked, prioritized test-idea\nlist) that scores claims up or down based on what's already been tested,\nrefuted, or flagged in a ticket, instead of handing the Driver an unranked\ndump. Proven end to end on `token_purchase` (a live Driver run consumed the\nranked list, and its results were fed back into the context layer). Each\nranked idea carries a stable id the Driver can cite (`oracle_claim_id`) when\na test targets it, so feedback actually reprioritizes claims on the next run\nrather than only ever matching on the Driver's own free-text hypothesis.\nFull status and backlog: [`docs/ontology-todo.md`](docs/ontology-todo.md).\n\n```\npython -m engine.ontology.oracle_creator --sut token_purchase   # layer 4: rank\npython -m engine.ontology.website --sut token_purchase          # view all 4 layers\npython -m engine.ontology.feedback --sut token_purchase --run <output.json>  # close the loop\n```\n\n## Layout\n\n```\nengine/\n  adapter.py    # SUTAdapter interface - what a per-SUT adapter must supply\n  tools.py      # HYPOTHESIS_TOOL / SKEPTIC_TOOL / BUG_REPORT_TOOL - shared across every adapter\n  client.py     # Anthropic client + call_tool_with_retry (tool-forced calls, retried on transient errors)\n  loop.py       # the checkpoint loop itself\n  outcome.py    # the typed envelope an adapter puts on each result - a SUT's behaviour in terms the\n                #   engine can compare and count, replacing prose no generic code could read\n  diagnostics.py # domain-free detectors over those envelopes - facts about the RUN rather than the SUT:\n                #   a state whose actions are all inert, a batch that didn't start from one place, a\n                #   baseline reset that stopped working, a prior that matched nothing\n  report.py     # generic HTML rendering (prose, badges, CSS, page/checkpoint structure)\n  runner.py     # orchestrates one full run: readiness probe, loop, bug reports, file output\n  cli.py        # python -m engine.cli --adapter <name>\n  adapters/\n    registry.py           # name -> adapter module, resolved lazily at run time\n    token_purchase/        # first adapter: single request/response, decline-reason logic\n    complex_sut/            # second adapter: concurrency/rate-limiting, proves the interface generalizes\n    clash_royale/           # third adapter: a live game client - no URL, no response body, taps and frames.\n                            #   The first non-HTTP SUT, which is what moved the engine's HTTP assumption\n                            #   behind check_sut_ready / fetch_happy_day_example. Read actions.py first:\n                            #   it is the whole safety argument for driving a real account. known_screens.json\n                            #   carries eleven screens an earlier recon pass measured against this client, four\n                            #   of them classified \"abort\" - which is how a run notices it reached the shop.\n  bootstrap/\n    discovery.py  # Phase 1 - fetch and parse a live OpenAPI/Swagger document\n    freetext.py   # Phase 2 - LLM fallback: infer a schema from free-text spec text\n    schema.py     # ties discovery and the free-text fallback together\n    probe.py      # Phase 3 - active probing loop against the live SUT\n    generate.py   # Phase 4 - generate a real adapter.py from a confirmed/inconclusive result\n    report.py     # renders a DiscoveredSchema as HTML for --discover-only\n    jira_mock.py  # stubbed ticket store for context-enriched bootstrap - see above; real JIRA is a TODO\n    cli.py        # python -m engine.bootstrap.cli - chains all 4 phases end to end\n  ontology/       # prioritization layer stack (heuristics/domain/context/ranked oracle) - see above\n  tests/          # deterministic regression + parity tests (no LLM calls, runs in CI)\nexperiments/      # mostly earlier prototypes this package was hardened from, kept as a historical\n                  #   archive. Two are NOT archive: they are worked on, and the clash_royale adapter\n                  #   imports them at run time (see session.py's own note on that debt), so a rename\n                  #   in either can break the engine with nothing in CI to catch it:\n  game-ontology/  #   the general game harness - window identity, readiness, input safety, screen\n                  #   discovery. Has its own pytest suite (72 tests). Read its README's \"Things that\n                  #   bit\" before changing anything in controller.py.\n  android-bot/    #   the Clash Royale target built on that harness: attach, recon, errands, one\n                  #   authorised Training Camp battle, and the measurement tools (127 tests). Its\n                  #   README carries the safety rules and which layer holds each - read it first,\n                  #   because this one drives somebody's real account.\ndocs/\n  exploratory-testing-engine-concept.md  # the original, broader vision\n  examples/bootstrap_demo/                # a real worked example of the bootstrap pipeline's output\n```\n\n`engine/*` never imports from `engine/adapters/*` - adapters import from\n`engine`, never the reverse. `engine/adapters/registry.py` is the only place\nthat crosses that boundary, and it does so lazily (`importlib`) at CLI run\ntime. `engine/bootstrap/` follows the same rule: it depends on `engine/`, not\non any concrete adapter.\n\n## Getting started\n\n```\n# terminal 1\npip install -r engine/requirements.txt\nuvicorn engine.adapters.token_purchase.sut:app --port 8000\n\n# terminal 2\ncp engine/.env.example engine/.env   # fill in ANTHROPIC_API_KEY\npython -m engine.cli --adapter token_purchase\n```\n\nTo authenticate through Amazon Bedrock instead of an API key, set\n`ENGINE_USE_BEDROCK=1` and `AWS_REGION` (plus `AWS_PROFILE` if needed) - see\n[`engine/README.md`](engine/README.md) for the model-ID caveats, which are not\nthe same IDs `aws bedrock list-inference-profiles` reports.\n\nWrites `runs/<adapter>/output.json`, `runs/<adapter>/bugs.json` (if any\nanomalies were found), and `runs/<adapter>/report.html`. Override run\nparameters with `--model`, `--max-checkpoints`, `--first-round-budget`,\n`--default-budget`, `--out-dir`.\n\nSee [`engine/README.md`](engine/README.md) for adding a new adapter by hand,\nand the CI/testing setup.\n\n## Testing\n\n```\npip install -r engine/requirements.txt\npython -m pytest engine/tests\n```\n\nRuns automatically on every push to `master` and every PR via\n[`.github/workflows/engine-tests.yml`](.github/workflows/engine-tests.yml) -\nno Anthropic API key needed, since no test makes a real LLM call.\n\nThe game harness carries its own suites, which CI does **not** run - they are\nWindows-only (Win32 window handles, GDI capture) while CI is Linux:\n\n```\npython -m pytest experiments/game-ontology experiments/android-bot\n```\n\nThose are also LLM-free and deterministic; every real-window call is\nmonkeypatched, so no game has to be installed to run them. Run them by hand\nbefore touching `controller.py`, because what they cover is which window\nreceives input and whether the harness may close a client - the two failures\nthere are a drag sent into somebody's editor and a game shut with no way to\nreopen it, and both have happened.\n",
  "bytes": 12286,
  "sha": "697db1347a9dc45a61aa8a69f83b00cbcc5b6e142bb277030478f3ee7e2dff36",
  "repo_slug": "hiddentrail/exploratory-testing-engine",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/okf_hiddentrail_exploratory_testing_engine_e_ab6acf9d/readme"
}