{
  "markdown": "# ios-agent\n\n**Drive an iPhone or an iOS Simulator with an AI agent.** A terminal app, an\nMCP server, and the library beneath both.\n\n[![CI](https://github.com/emazaheri/ios-agent/actions/workflows/ci.yml/badge.svg)](https://github.com/emazaheri/ios-agent/actions/workflows/ci.yml)\n[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)\n[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)\n[![macOS](https://img.shields.io/badge/platform-macOS-lightgrey.svg)](#requirements)\n[![Tests](https://img.shields.io/badge/tests-618%20offline-brightgreen.svg)](#development)\n\n![ios-agent answering a question by driving Apple Maps](docs/images/demo.gif)\n\n<sub>One goal, start to finish, at 2.5x. The agent deep-links into Maps for the\ndriving time, then taps through to walking and transit and scrolls to read the\ndetail: 4 actions, 1 observation, 2,767 device tokens, 49.8s of real time. The\nterminal is the agent's own transcript; the phone is an iOS Simulator being\ndriven by it.</sub>\n\nBuilt on Apple's XCUIAutomation through WebDriverAgent. It runs on a Mac and\ndrives a simulator or a tethered phone. It is designed for **agents rather than\ntest suites**: screens arrive as a compact digest instead of raw accessibility\nXML, actions hand back the screen they produced, and anything irreversible asks\nfirst.\n\n```bash\nuv sync && uv run ios-agent quickstart\n```\n\n`quickstart` checks the toolchain, offers the repairs that are cheap enough to\nbe worth offering, builds WebDriverAgent if it is missing (about 20 seconds,\nonce), and drops you into manual mode, which drives the device by hand and\nneeds no API key. When you want the agent itself:\n\n```bash\nuv run ios-agent \"turn on bold text\"\n```\n\n## Contents\n\n- [Why it is built this way](#why-it-is-built-this-way)\n- [Requirements](#requirements) · [Setup](#setup)\n- [The terminal app](#the-terminal-app)\n- [Connecting your own agent over MCP](#connecting-your-own-agent-over-mcp)\n- [What the model sees](#what-the-model-sees)\n- [Safety](#safety)\n- [Measured on real hardware](#measured-on-real-hardware)\n- [Development](#development) · [Contributing](#contributing)\n\n## Why it is built this way\n\nRaw WebDriverAgent page source for a 200-row list runs to roughly **37,000\ntokens**. Re-reading that after every tap exhausts a context window in a handful\nof steps. Four decisions follow from that, and they are the whole design:\n\n| | |\n|---|---|\n| **Perception is budget-aware** | 251 raw nodes to 12 elements on a real third-party screen; 50–445 tokens per step |\n| **Actions return the screen they produced** | halves round-trips, and returns a *delta* when the screen is similar |\n| **Resolution runs on the host** | six tiers, so a retry costs zero model tokens where a round-trip costs a whole turn |\n| **The gate asks before acting, not after** | so the answer still means something |\n\nEverything else in the repository is downstream of those.\n\n## Requirements\n\n| For | You need |\n|---|---|\n| **Simulator** | macOS, Xcode 16.3+, an iOS runtime, Python 3.12+ |\n| **Physical iPhone** | the above plus [go-ios](https://github.com/danielpaulus/go-ios), Developer Mode, and a signing identity. Follow [docs/real-device-setup.md](docs/real-device-setup.md), which is a longer road than the simulator and has a few steps that look like bugs but are not |\n\nXcode ships **without** a simulator runtime. If `xcrun simctl list runtimes` is\nempty, `xcodebuild -downloadPlatform iOS` fetches one (around 8 GB). If a\nruntime is installed but no simulator has been created, `ios-agent` offers to\ncreate one for you: that part takes about a second.\n\n## Setup\n\n```bash\nuv sync\n./scripts/prepare_wda.sh simulator   # builds WebDriverAgent, once\nuv run ios-agent doctor              # says exactly what is still missing\n```\n\n`doctor` is the first thing to run whenever anything misbehaves. It checks the\ntoolchain, the simulator runtimes, the tunnel, WebDriverAgent's signing expiry\nand the model, and returns a **remedy for each failure** rather than letting it\nsurface later as a connection error.\n\nThe app runs those same checks before it touches a device, so a machine that is\nnot set up is told so in about a second rather than after a simulator has\nbooted.\n\n## The terminal app\n\n```bash\nuv run ios-agent                             # open it, decide later\nuv run ios-agent \"turn on bold text\"         # give it a goal\nuv run ios-agent --pick \"turn wi-fi off\"     # choose the device from a list\nuv run ios-agent manual                      # drive it by hand, no model needed\nuv run ios-agent devices                     # what is reachable\n```\n\n![the terminal app, mid-run](docs/images/ios-agent.png)\n\nIt streams the model's reasoning as it arrives, shows the digest the model is\nreading beside it, and keeps the numbers on screen while they climb: actions,\nobservations, device tokens, cost.\n\n| | |\n|---|---|\n| `/` | command menu, filtered as you type |\n| `/device` · `ctrl+o` | switch phone or simulator mid-session |\n| `esc` | stop at the next step, with a complete report; again to abort |\n| `ctrl+r` · `ctrl+s` | re-read the screen · save the audit trail |\n| `/copy` · `ctrl+y` | copy the transcript, or a selection, to the clipboard |\n| `--inline` | run in a short region under the prompt |\n| `--no-tui` | plain lines, for a pipe |\n\n**`manual` mode needs no API key.** It drives the same nine verbs by hand,\nwhich is the fastest way to debug perception on an app nobody has pointed this\nat before.\n\nThe front end is held to one rule, asserted rather than argued: **watching a\nrun may not change what it costs.** `tests/tui/test_cost.py` runs the same task\nwrapped and unwrapped and compares every counter by equality.\n\n### Choosing a model\n\nThe provider is configuration, not a dependency. The loop builds through\nLangChain's `init_chat_model`, so switching is two environment variables and an\nextra:\n\n```bash\nuv sync --extra openai\nIOS_AGENT_PROVIDER=openai IOS_AGENT_MODEL=gpt-5.6-sol uv run ios-agent \"...\"\n```\n\nAnthropic, OpenAI, Gemini, Bedrock, Groq, Mistral and a local Ollama model are\nall supported. See [agent/README.md](agent/README.md).\n\n## Connecting your own agent over MCP\n\n30 tools and 4 resources, over stdio or HTTP. Add to `.mcp.json` (already\npresent here for Claude Code):\n\n```json\n{\n  \"mcpServers\": {\n    \"ios\": { \"command\": \"uv\", \"args\": [\"run\", \"--directory\", \".\", \"ios-mcp\", \"serve\"] }\n  }\n}\n```\n\nThen ask for what you want in plain language. The server ships an `ios_operator`\nprompt that teaches the observe/act/verify loop, so clients do not have to\nreinvent it. For a remote client, `ios-mcp serve --transport http --port 8765`.\n\nOr skip the protocol and import the library:\n\n```python\noutcome = await run_goal(session, \"turn on bold text\")\n```\n\n## What the model sees\n\n```\nscreen: com.apple.Preferences / \"Display & Text Size\"  fp=3872280e\ne1   button       \"Accessibility\" id=BackButton @(38,84)\ne2   switch       \"Bold Text\" =0 id=ENHANCE_TEXT_LEGIBILITY @(336,161)\ne3   button       \"Larger Text, Off\" id=LARGER_TEXT @(190,216)\n```\n\nMeasured across eleven golden flows on a real simulator: **50 to 422 tokens per\ntool call.**\n\nThe agent passes `e2` back to an action. It never writes XPath and never\nguesses coordinates. If a ref goes stale because the screen moved, the host\nre-finds the same element by identity rather than failing.\n\n## Safety\n\nAutomating someone's real phone is not test automation. On by default:\n\n- Anything matching **Send, Pay, Buy, Delete, Confirm or Sign Out** needs\n  approval *before* it happens, via MCP elicitation or an\n  `action_requires_approval` error an external human-in-the-loop layer can\n  answer. Approval is scoped to one action: approving Send never approves\n  Delete.\n- Without an approver the run is unattended and everything destructive is\n  **refused**, because an unanswerable question is not consent.\n- `ios_type_secret` reads a value from the host keychain and sends it straight\n  to the device. It never enters a prompt, a tool result, or the audit trail.\n- Card numbers and email addresses are stripped from everything leaving the\n  server.\n- Repeated failures or a detected loop halt the session.\n- The device picker never pre-selects a physical phone. Reaching one always\n  costs a keystroke.\n\nSee [SAFETY.md](SAFETY.md). Every default is settable through an `IOS_MCP_*`\nenvironment variable, a `.env`, or an optional `ios-mcp.toml`, in that order of\nprecedence. Copy `.env.example` to `.env` for the full list.\n\n## Measured on real hardware\n\nThe eval harness was built before the agent, which is the only reason any\nof these numbers exist. Latest measurement, 13 tasks × 3 runs on\n`gpt-5.6-sol`:\n\n| | |\n|---|---|\n| success | 39/39 |\n| observations | **39, against an oracle floor of 39** |\n| refusals, unusable runs | 0, 0 |\n| cost | $2.13 over 10m28s |\n\nEvery task sits at the observation floor, including two in an app Apple did\nnot write, because every action already folds the screen it produced into\nits response.\n\n### Verified on real iOS, including a physical iPhone\n\nTier 1 runs against a scripted in-process device, so its numbers are a claim\nabout a fake. The same goal, `turn on Bold Text`, across all three tiers:\n\n| | actions | observations | digest |\n|---|---|---|---|\n| scripted fake | 3 | 1 | — |\n| iOS 26.5 simulator | 3 | 1 | 167 raw nodes → 14 elements, 261 tokens |\n| **iPhone, iOS 26.6, Wi-Fi** | **3** | **1** | 140 raw nodes → 15 elements, 243 tokens |\n\nIdentical on all three, and on the phone it took 48.6s where the simulator took\nseconds. The switch was confirmed by navigating there and reading `value=\"1\"`\nindependently of what the agent claimed, then restored.\n\nMost importantly, **a real no-op still reports `screen_changed=False` on the\nphone.** If a physical device had moved its fingerprint between settled\nsnapshots, the verification step would have been silently dead on hardware\nwhile every simulator and fake test stayed green.\n\nHardware is opt-in twice over, by the `device` marker and\n`IOS_MCP_ALLOW_DEVICE=1`, because hardware being present is not consent to\nchange settings on it.\n\n## Development\n\n```bash\nuv run pytest tests/unit          # 433 tests, no device, no model\nuv run pytest tests/tui           # 185 tests, the terminal front end\nuv run pytest tests/integration   # 13 tests, real simulator\nuv run pytest tests/evals -s      # golden flows, with cost per flow\nuv run ruff check . && uv run mypy ios_mcp agent/ios_agent tui/ios_tui\n```\n\nThe eval suite is the quality gate: it reports tokens, wall time, action count\nand resolution-tier distribution per flow. A drift from `exact` toward\n`text-fuzzy` is the leading indicator that a flow is about to become flaky.\nAgent tasks additionally declare an **action floor**, the number of actions a\nhand-written oracle needs, asserted against that oracle so it cannot drift into\nan aspiration. Failures are attributed too: a report says which of them were\nthe device, perception, the model or the policy gate, rather than only that\nsomething failed.\n\nThose numbers are kept over time in `tests/evals/history.jsonl`, one committed\nline per measured run. CI runs the one series that costs nothing (the oracle\nagainst a scripted device) and fails if any of it moves without the new line\nbeing committed alongside. Hand-run slices go in the same file:\n\n```bash\npython scripts/eval_trend.py show --suite agent-oracle\npython scripts/eval_trend.py append .artifacts/evals/agent-s5.json --suite agent-model\n```\n\nThe guard is exact rather than banded, because every number it checks is a\ncount on a fixed route with no model, no network and no clock in it. See\n[docs/adr/0009](docs/adr/0009-the-eval-trend-is-committed.md).\n\n```bash\nuv run python scripts/tui_screenshot.py   # render the front end to .artifacts\n```\n\nA passing test suite says nothing about what a terminal app looks like. That\nscript has caught eight display bugs no assertion did.\n\nThree distributions in one uv workspace, and the dependencies only point one\nway. See [ARCHITECTURE.md](ARCHITECTURE.md):\n\n```\nios-tui    terminal front end     depends on ios-agent, ios-mcp\nios-agent  goal-directed agent    depends on ios-mcp\nios-mcp    library + MCP server   depends on neither\n```\n\n## Why the automation runs on a host, not on the phone\n\nAn iOS app cannot automate other apps on the device it runs on. The sandbox\nblocks cross-process access, and the Accessibility API is unavailable to\nsandboxed apps even with user consent. XCUIAutomation only executes inside an\nXCTest runner started by `testmanagerd`, which is driven from a host. Any iOS\napp in this project's future is a client of this server, never the engine.\n\n## Contributing\n\nIssues and pull requests are welcome. [CONTRIBUTING.md](CONTRIBUTING.md) covers\nthe setup, the loop, and the five conventions that are load bearing rather than\nstylistic. CI runs ruff, mypy and the 618 offline tests on Linux and macOS.\n\n## License\n\nMIT. See [LICENSE](LICENSE).\n\n<!-- Ownership proof for the MCP registry: it checks that the server name\n     appears in the README published to PyPI. -->\n\nmcp-name: io.github.emazaheri/ios-agent\n",
  "bytes": 13138,
  "sha": "878b83c8be1a4b692be82a0425dc69db2572bb19e868a79f546b04c5269662cd",
  "repo_slug": "emazaheri/ios-agent",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_emazaheri_ios_agent_987c8312/readme"
}