{
"markdown": "# OSWright\n\n[](https://pypi.org/project/oswright/)\n[](https://github.com/Ask-812/oswright/actions/workflows/test.yml)\n[](https://pypi.org/project/oswright/)\n[](LICENSE)\n\n**Desktop automation for AI agents, without paying for a screenshot every step.**\n\nmcp-name: io.github.Ask-812/oswright\n\nAn MCP server that lets an LLM drive real desktop applications — the desktop\nequivalent of [Playwright MCP](https://github.com/microsoft/playwright-mcp). It\nkeeps a model of the screen between actions and re-reads only the parts that\nchanged, so the same work costs an order of magnitude fewer tokens.\n\n\n\nEight fields read off an invoice and typed into an expense form, verified by the\napplication itself. Same task, same result, **7.4× less context** than returning\na screenshot after every action. Every number on screen is measured during the\nrun — regenerate the whole thing with `python benchmarks/record_demo.py`.\n\n### Why this exists\n\nMost GUI agents re-perceive the entire screen on every step: screenshot, OCR,\nhand the model an image, repeat. Measured on a live desktop, **the median\nobservation changes 0.012% of the screen's pixels**. Re-reading everything does\nfar more work than the change warrants, and charges ~2,800 image tokens whether\nanything happened or not.\n\nOSWright asks the compositor what changed, rescans only that, and answers\nelement lookups from the cheapest source that can. The claims below are measured\non this machine and reproducible from [`benchmarks/`](benchmarks/) — including\nthe ones that did not come out in its favour.\n\n### Key Features\n\n- **Cross-platform.** Windows (Win32 API), Linux (pynput/X11), macOS (pynput/Quartz).\n- **Accessibility tree.** Find elements deterministically by role and name via Windows UI Automation — 100% accurate, instant, no model needed.\n- **Fast OCR.** Windows OCR (built-in, instant) with EasyOCR fallback for Linux/macOS. Results are cached automatically.\n- **Lightweight on Windows.** No PyTorch download — Windows uses the built-in OCR engine, so a full install is a few MB rather than a few GB.\n- **Image matching.** Locates elements by template image via OpenCV.\n- **Window management.** List, focus, minimize, close, and screenshot specific windows.\n- **Screenshot diffing.** Detect when the screen changes with `wait_for_change`.\n- **Clipboard access.** Read and write system clipboard for data transfer.\n- **App launcher.** Launch applications and wait for them to load.\n- **Auto-snapshot.** Every action returns a screenshot so the agent always sees current state.\n- **43 MCP tools.** Screen, OCR, UIA, mouse, keyboard, windows, clipboard, and compound actions.\n- **Incremental perception.** Rescans only the parts of the screen that changed, and can return what changed instead of a full screenshot — ~21× fewer tokens per step.\n- **Screen memory.** Recognises screens it has read before and reuses them, verified by pixels — 89× cheaper than reading again.\n- **Speculative perception.** Learns what actions do and confirms the expected result instead of re-reading — 19–23× cheaper, with a `surprise` report when the interface does something unexpected.\n- **Adaptive waiting.** Waits for the screen to actually settle rather than sleeping a fixed 300 ms — 11.9 s saved over a 50-step task.\n- **Resolution cascade.** Element lookups stop at the cheapest method that works; repeat lookups cost ~0.05 ms.\n- **DPI-correct.** Coordinates are physical pixels everywhere, so clicks land correctly on scaled displays.\n- **Test suite.** 237 automated tests; the desktop-driving ones skip themselves when no display is available.\n\n### Requirements\n\n- Python 3.10 or newer\n- VS Code, Cursor, Windsurf, Claude Desktop, or any other MCP client\n\n## Getting started\n\nFirst, install the OSWright MCP server with your client.\n\n**Standard config** works in most tools:\n\n```json\n{\n \"mcpServers\": {\n \"oswright\": {\n \"command\": \"uvx\",\n \"args\": [\"oswright\"]\n }\n }\n}\n```\n\n> **Note:** If you don't have `uvx`, you can use `pip install oswright` and then set `\"command\": \"oswright\"` directly.\n\n\nClaude Desktop
\n\nFollow the MCP install [guide](https://modelcontextprotocol.io/quickstart/user), use the standard config above.\n\n \n\n\nClaude Code
\n\n```bash\nclaude mcp add oswright uvx oswright\n```\n\n \n\n\nVS Code
\n\nAdd to your user or workspace `settings.json` under `mcp.servers`:\n\n```json\n{\n \"mcp\": {\n \"servers\": {\n \"oswright\": {\n \"command\": \"uvx\",\n \"args\": [\"oswright\"]\n }\n }\n }\n}\n```\n\nOr use the VS Code CLI:\n\n```bash\ncode --add-mcp '{\"name\":\"oswright\",\"command\":\"uvx\",\"args\":[\"oswright\"]}'\n```\n\n \n\n\nCursor
\n\nGo to `Cursor Settings` -> `MCP` -> `Add new MCP Server`. Name it `oswright`, use `command` type with the command `uvx oswright`.\n\n \n\n\nWindsurf
\n\nFollow Windsurf MCP [documentation](https://docs.windsurf.com/windsurf/cascade/mcp). Use the standard config above.\n\n \n\n\nCline
\n\nAdd to your `cline_mcp_settings.json`:\n\n```json\n{\n \"mcpServers\": {\n \"oswright\": {\n \"type\": \"stdio\",\n \"command\": \"uvx\",\n \"args\": [\"oswright\"],\n \"disabled\": false\n }\n }\n}\n```\n\n \n\n\nGoose
\n\nGo to `Advanced settings` -> `Extensions` -> `Add custom extension`. Name it `oswright`, use type `STDIO`, and set the `command` to `uvx oswright`.\n\n \n\n\nUsing pip instead of uvx
\n\nIf you prefer a standard pip install:\n\n```bash\npip install oswright\n```\n\nThen use this config:\n\n```json\n{\n \"mcpServers\": {\n \"oswright\": {\n \"command\": \"oswright\"\n }\n }\n}\n```\n\nOr run directly:\n\n```bash\npython -m oswright\n```\n\n \n\n## Incremental perception\n\nMost GUI agents re-perceive the entire screen on every step: full screenshot,\nfull OCR, then hand the model a fresh image. Measured on a live desktop, the\nmedian observation changes **0.012% of pixels** — so a full rescan does roughly\n240× more work than the change warrants, and the screenshot it returns costs\n~2,800 image tokens whether anything happened or not.\n\nOSWright keeps a model of the screen between observations and rescans only the\nregions that actually moved.\n\n```\nobserve() -> {\"changed\": true,\n \"added\": [{\"text\": \"Saved\", \"x\": 812, \"y\": 447}],\n \"removed\": [\"Unsaved changes\"],\n \"screen_fraction_scanned\": 0.015}\n```\n\nMeasured on this machine over a 14-step agent loop:\n\n| | v0.4.0 (full OCR + screenshot) | incremental |\n|---|---|---|\n| Median latency per step | 212 ms | **33 ms** |\n| Tokens per observation | ~2,764 | **~49** |\n| Tokens over 14 steps | 38,696 | **1,025** |\n| Screen re-read | 100% | **16%** |\n\nThe busier the screen, the larger the gap: full OCR scales with how much text is\non screen, whereas the incremental path scales with how much *changed*. The same\ncomparison measures 6.5× on a quiet desktop and **14.3×** with a dense web page\nopen. Re-measure with [`benchmarks/`](benchmarks/) rather than trusting these.\n\nCost is a proxy, though, and a cheaper perception path that quietly degraded\naccuracy would be worse than none. So it is checked against task completion:\nscripted tasks driving the real tool surface across four applications, graded\nagainst each application's own state — UI Automation for Calculator and\nExplorer, the window title for Chrome and VS Code — never against OCR.\n\n| configuration | Calculator | File Explorer | Chrome | tokens |\n|---|---|---|---|---|\n| v0.4-style (full screenshot) | 9/9 | 3/3 | 3/3 | 118,858 |\n| delta only | 9/9 | 3/3 | 3/3 | **5,252** |\n| delta + memory | 9/9 | 3/3 | 3/3 | 5,099 |\n| delta + memory + prediction | 9/9 | 3/3 | 3/3 | 7,981 |\n\n**Accuracy is identical across every configuration while token cost falls 23×.**\nRun it with `python benchmarks/bench_tasks.py`.\n\n### Why both pixels and accessibility\n\nThe design bets that neither perception path wins everywhere. Turning each half\noff measures that rather than asserting it:\n\n| configuration | Calculator | File Explorer | Chrome |\n|---|---|---|---|\n| full cascade | **9/9** | **3/3** | **3/3** |\n| accessibility only | 9/9 | **0/3** | **0/3** |\n| pixels only | **6/9** | 3/3 | 3/3 |\n\nAccessibility-only — the posture most Windows GUI agents take — is perfect on\nXAML and blind on a Win32 list view and on web content. Probed against VS Code\nit sees **18 elements**, the entire IDE being a single node named `Chrome Legacy\nWindow`, while OCR reads 94 including every filename.\n\nPixels-only fails Calculator's buttons, because the button a human reads as `7`\nis *named* `Seven`, and Windows OCR returns no digits from Calculator at all.\n\nThe cascade is the only configuration that passes everywhere.\n\n### The resolution cascade\n\n`find_element` and `click_element` stop at the first method that can answer,\nso cost tracks how *novel* the request is rather than how large the screen is:\n\n| Rung | Method | Typical cost |\n|---|---|---|\n| 0 | Already in the screen model | **~0.05 ms** |\n| 1 | Rescan only what changed | ~70 ms |\n| 2 | Accessibility tree (knows a Button *is* a button) | ~40 ms |\n| 3 | App's own text buffer via UIA TextPattern — exact characters | ~400 ms |\n| 4 | Full-screen OCR | ~250 ms |\n\nLooking up text the model already knows is **~5,000× cheaper** than the v0.4.0\npath (0.05 ms versus 244 ms). The response reports which rung answered, so you\ncan see what a task is actually costing.\n\nRung 3 is worth understanding: UIA's `TextRange.FindText` searches the\napplication's *own* text buffer and returns exact bounding rectangles. It is\nimmune to font, DPI, antialiasing and OCR error. It sits below the pixel rungs\nonly because scanning a window's controls for it costs a few hundred\nmilliseconds of cross-process COM — it is the accurate rung, not the fast one.\n\n> **Note on ordering.** These rungs are ordered by *measurement*, not by theory.\n> The common advice is to make the accessibility tree primary, but on real\n> applications it is not always cheaper: walking Chrome's tree took 537 ms here,\n> slower than a full-screen OCR pass, and VS Code exposed only 18 elements to it.\n> Neither pixels nor accessibility wins everywhere, which is why this is a\n> cascade rather than a choice.\n\n### Asking the compositor instead of looking\n\nOn Windows, the desktop compositor already knows which pixels changed and\nexposes them through DXGI Desktop Duplication. Asking it costs **0.14 ms and\ntransfers no pixels**, against tens of milliseconds to capture a frame and\ndiscover it was identical — so an idle observation skips the capture entirely.\n\nWhen something *has* changed, the compositor is left holding that frame, so its\npixels are read directly from the GPU rather than grabbed a second time through\na different API — **1.5–2.3× faster** than `mss` in measurements here.\n\nThe compositor is used only as a fast *negative* for change detection. When it\nreports a change, the dirty regions still come from hashing the captured frame:\nthe two are measured over slightly different intervals, so compositor rectangles\ncan under-report relative to the pixels actually captured, and an under-reported\nregion is text that never gets re-read. It degrades silently to tile hashing and\nnormal capture wherever Desktop Duplication is unavailable.\n\nEnable delta observations for action tools with `--observation-mode delta`.\nThe default remains `screenshot` for compatibility with existing clients.\n\n**Reproduce all of this yourself:** see [`benchmarks/`](benchmarks/).\nThe reasoning behind each decision, including the dead ends, is in\n[`docs/ENGINEERING_LOG.md`](docs/ENGINEERING_LOG.md).\n\n## Configuration\n\nOSWright MCP server supports the following arguments. They can be provided in the JSON configuration as part of the `\"args\"` list:\n\n| Option | Description | Env Variable |\n|--------|-------------|-------------|\n| `--port ` | Port for SSE transport. If omitted, uses stdio (default). | `FASTMCP_PORT` |\n| `--host ` | Host to bind the HTTP/SSE server to. Default: `127.0.0.1`. | `FASTMCP_HOST` |\n| `--transport ` | Transport protocol: `stdio`, `sse`, `streamable-http`. Auto-detected from `--port`. | |\n| `--ocr-languages ` | OCR languages (default: `en`). Example: `--ocr-languages en es fr` | `OSWRIGHT_OCR_LANGUAGES` |\n| `--timeout ` | Default timeout for auto-wait operations (default: `10`). | `OSWRIGHT_TIMEOUT` |\n| `--snapshot-max-width ` | Downscale the auto-snapshot returned after each action. `0` (default) keeps full resolution. Lower values cut token cost significantly. | `OSWRIGHT_SNAPSHOT_MAX_WIDTH` |\n| `--observation-mode ` | What action tools return: `screenshot` (default), `delta` (only what changed, ~30× fewer tokens), or `both`. | `OSWRIGHT_OBSERVATION_MODE` |\n| `--no-atlas` | Do not remember screens across visits. | `OSWRIGHT_NO_ATLAS` |\n| `--no-speculate` | Do not predict the outcome of actions. | `OSWRIGHT_NO_SPECULATE` |\n| `--allow-remote` | Required to bind a non-loopback address. See [Security](#security). | |\n| `--log-level ` | Logging level: `DEBUG`, `INFO`, `WARNING`, `ERROR`. Default: `INFO`. | `OSWRIGHT_LOG_LEVEL` |\n\nAn explicit command-line flag always wins over the corresponding environment variable.\n\n### Example: Multi-language OCR\n\n```json\n{\n \"mcpServers\": {\n \"oswright\": {\n \"command\": \"uvx\",\n \"args\": [\"oswright\", \"--ocr-languages\", \"en\", \"es\", \"fr\"]\n }\n }\n}\n```\n\n### Standalone MCP server (SSE)\n\nWhen running from a worker process or another machine, use SSE transport:\n\n```bash\nuvx oswright --port 8931\n```\n\nThen in your MCP client config:\n\n```json\n{\n \"mcpServers\": {\n \"oswright\": {\n \"url\": \"http://127.0.0.1:8931/sse\"\n }\n }\n}\n```\n\n## Security\n\n**OSWright has no authentication.** Anyone who can reach the port gets full\nkeyboard, mouse, screen and clipboard control of the machine — it is remote\ndesktop takeover, not a sandboxed API.\n\nThe server therefore binds to `127.0.0.1` by default and **refuses** to start on\na non-loopback address unless you pass `--allow-remote`. To reach it from\nanother machine, prefer an SSH tunnel over exposing the port:\n\n```bash\nssh -L 8931:127.0.0.1:8931 user@desktop-host\n```\n\nStdio transport (the default, used by every MCP client config above) is not\nnetwork-exposed at all and is the recommended way to run OSWright.\n\nTools that can destroy work are annotated accordingly: `close_window` is marked\ndestructive, and `launch_app` starts arbitrary programs. Screenshot tools refuse\nto overwrite an existing `save_path`.\n\n## Platform Notes\n\n| Platform | Input Backend | OCR Backend | Extra downloads |\n|----------|--------------|-------------|-------|\n| Windows | Win32 API (SendInput) | Windows OCR (instant, built-in) | **None.** No PyTorch. UI Automation included. |\n| Linux | pynput (X11) | EasyOCR | PyTorch (~2.5 GB). Requires X11; Wayland has limited support. |\n| macOS | pynput (Quartz) | EasyOCR | PyTorch (~2.5 GB). Grant Accessibility permissions in System Settings > Privacy > Accessibility. |\n\nOn Windows, EasyOCR is *not* installed, because the built-in Windows OCR engine\nis faster and needs no model download. Install it only if you need a language\nWindows OCR does not support:\n\n```bash\npip install \"oswright[easyocr]\"\n```\n\n### Coordinates\n\nAll coordinates returned by OCR, image matching and UI Automation are **absolute\nphysical screen pixels**, ready to pass straight to `mouse_click`. This holds for\nsub-regions and for multi-monitor setups where the virtual desktop starts at a\nnegative origin. `screenshot` also reports `origin_x`/`origin_y`, the absolute\nposition of the image's top-left pixel, for when you read a coordinate off the\nimage yourself.\n\n## Tools\n\n\nScreen
\n\n- **screenshot** -- Take a screenshot of the screen or a region. Returns the image as native MCP image content. Optionally saves to a file path.\n - Read-only: **true**\n\n- **get_screen_info** -- Get screen dimensions and monitor count.\n - Read-only: **true**\n\n \n\n\nOCR / Text Finding
\n\n- **find_text_on_screen** -- Find all occurrences of text on screen using OCR. Returns matches with coordinates and confidence.\n - Parameters: `text`, `exact`, region bounds, `monitor`\n - Read-only: **true**\n\n- **read_screen_text** -- Read ALL visible text on the screen using OCR. Returns every detected text element with position.\n - Parameters: region bounds, `monitor`\n - Read-only: **true**\n\n \n\n\nImage Matching
\n\n- **find_image_on_screen** -- Find all occurrences of a template image on screen using OpenCV template matching.\n - Parameters: `template_path`, `threshold`, `monitor`\n - Read-only: **true**\n\n \n\n\nMouse
\n\n- **mouse_click** -- Click the mouse at coordinates or current position. Returns screenshot.\n - Parameters: `x`, `y`, `button`, `clicks`\n\n- **mouse_double_click** -- Double-click at coordinates or current position. Returns screenshot.\n\n- **mouse_move** -- Move the mouse cursor to screen coordinates.\n\n- **mouse_scroll** -- Scroll the mouse wheel. Returns screenshot.\n - Parameters: `amount`, `x`, `y`\n\n- **mouse_drag** -- Drag from one point to another. Returns screenshot.\n - Parameters: `start_x`, `start_y`, `end_x`, `end_y`, `button`, `duration`\n\n- **get_mouse_position** -- Get the current mouse cursor position.\n - Read-only: **true**\n\n \n\n\nKeyboard
\n\n- **type_text** -- Type text character by character. Returns screenshot.\n - Parameters: `text`, `delay`\n\n- **press_key** -- Press a key or combo like `Enter`, `Ctrl+C`, `Alt+Tab`. Returns screenshot.\n - Parameters: `key`\n\n \n\n\nCompound Actions
\n\n- **click_text** -- Find text via OCR and click on it. Auto-retries until found or timeout. Returns screenshot.\n - Parameters: `text`, `exact`, `button`, `timeout`, `poll_interval`, `monitor`\n\n- **double_click_text** -- Find text via OCR and double-click on it. Returns screenshot.\n\n- **right_click_text** -- Find text via OCR and right-click on it. Returns screenshot.\n\n- **hover_text** -- Find text via OCR and hover over it. Returns screenshot.\n\n- **fill_field** -- Find a label, click it, clear, and type a value. Returns screenshot.\n - Parameters: `target_text`, `value`, `exact`, `timeout`, `monitor`\n\n- **fill_form** -- Fill multiple fields in one call. Reduces round-trips.\n - Parameters: `fields` (list of `{label, value}`), `timeout`, `monitor`\n\n- **wait_for_text** -- Wait for text to appear on screen. Polls via OCR.\n - Parameters: `text`, `exact`, `timeout`, `poll_interval`, `monitor`\n - Read-only: **true**\n\n- **wait_for_text_gone** -- Wait for text to disappear from screen.\n - Parameters: `text`, `exact`, `timeout`, `poll_interval`, `monitor`\n - Read-only: **true**\n\n- **wait_for_time** -- Wait for a specified duration (capped at 30s), then screenshot.\n\n \n\n\nWindow Management
\n\n- **list_windows** -- List all visible windows. Optionally filter by title substring.\n - Parameters: `title_filter`\n - Read-only: **true**\n\n- **focus_window** -- Bring a window to the foreground by title. Returns screenshot.\n - Parameters: `title`\n\n- **close_window** -- Close a window by title (sends WM_CLOSE). Returns screenshot.\n - Parameters: `title`\n\n- **minimize_window** -- Minimize a window by title. Returns screenshot.\n - Parameters: `title`\n\n- **screenshot_window** -- Capture a screenshot of just one window.\n - Parameters: `title`, `save_path`\n - Read-only: **true**\n\n \n\n\nClipboard
\n\n- **get_clipboard** -- Get the current text content of the system clipboard.\n - Read-only: **true**\n\n- **set_clipboard** -- Copy text to the system clipboard.\n - Parameters: `text`\n\n \n\n\nApp Management
\n\n- **launch_app** -- Launch an application and optionally wait for it to load. Runs the program directly, never through a shell.\n - Parameters: `command`, `args`, `wait_text`, `timeout`\n - Reports `wait_text_found` so you can tell whether the app actually loaded.\n\n- **get_ocr_info** -- Get info about the active OCR backend and available backends.\n - Read-only: **true**\n\n \n\n\nIncremental Perception
\n\n- **observe** -- Report what changed on screen since the last observation. Rescans only the regions that moved. Prefer this over `screenshot` for tracking state.\n - Parameters: `force_full`\n - Read-only: **true**\n\n- **find_element** -- Find on-screen text using the cheapest method that can answer. Reports which cascade rung responded.\n - Parameters: `text`, `exact`, `window_title`\n - Read-only: **true**\n\n- **click_element** -- Find text via the cascade and click it. The cheap alternative to `click_text`.\n - Parameters: `text`, `exact`, `button`, `window_title`\n\n- **read_model_text** -- Read on-screen text from the incremental model without re-OCRing the display.\n - Parameters: `query`, `limit`\n - Read-only: **true**\n\n- **perception_stats** -- Report how much perception work the model has avoided.\n - Read-only: **true**\n\n- **remember_screen** -- Remember the current screen so future visits skip reading it. Persists across sessions.\n - Read-only: **true**\n\n- **atlas_stats** -- Report what the screen atlas has remembered and how often it helped.\n - Read-only: **true**\n\n \n\n\nAccessibility / UI Automation (Windows)
\n\n- **get_ui_tree** -- Get the accessibility tree of the focused window. Returns all interactive elements with names, types, positions. Deterministic and instant.\n - Parameters: `window_title`, `max_depth`\n - Read-only: **true**\n\n- **click_ui_element** -- Click a UI element using the accessibility tree. More reliable than OCR.\n - Parameters: `name`, `control_type`, `automation_id`, `window_title`\n\n- **fill_ui_element** -- Set the value of a UI element (e.g., text box). More reliable than OCR-based fill.\n - Parameters: `value`, `name`, `automation_id`, `window_title`\n\n \n\n\nAdvanced Screen
\n\n- **get_active_window** -- Get info about the currently focused window.\n - Read-only: **true**\n\n- **wait_for_change** -- Wait for the screen to visually change. Takes a baseline screenshot, polls until different.\n - Parameters: `timeout`, `poll_interval`\n\n \n\n## Python Library\n\nOSWright also works as a standalone Python library with a Playwright-style API:\n\n```python\nfrom oswright import OSWright\n\nwith OSWright() as ow:\n screen = ow.screen()\n screen.click(text=\"Start\")\n screen.type_text(\"Hello World\")\n screen.press(\"Ctrl+S\")\n screen.screenshot(\"desktop.png\")\n```\n\nSee the [examples/](examples/) directory for more.\n\n## Architecture\n\n```\noswright/\n __init__.py # Package entry point (single source of __version__)\n core.py # OSWright class (= Browser)\n screen.py # Screen class (= Page)\n locator.py # Locator + Assertions (= Locator + expect)\n capture.py # Screen capture (mss - cross-platform, thread-safe)\n dirty.py # Change detection - which parts of the screen moved\n screenmodel.py # Persistent screen model, updated incrementally\n cascade.py # Resolution cascade - cheapest method that can answer\n atlas.py # Remembers screens across visits and sessions\n settle.py # Knowing when the screen has finished responding\n speculate.py # Predicting what an action does, instead of looking\n textprovider.py # Exact text from the app itself via UIA TextPattern\n detect.py # OCR dispatcher with caching (auto-selects best backend)\n _ocr_windows.py # Windows OCR backend (instant, built-in)\n accessibility.py # Windows UI Automation (deterministic element finding)\n cache.py # Screenshot diffing, image hashing, OCR result cache\n _dpi.py # Process DPI awareness (keeps every API in physical pixels)\n _dxgi_windows.py # Compositor dirty rectangles via DXGI Desktop Duplication\n input.py # Platform dispatcher for input backends\n _input_windows.py # Windows input backend (Win32 API)\n _input_pynput.py # Linux/macOS input backend (pynput)\n window.py # Window management (list, focus, close)\n clipboard.py # Clipboard read/write (cross-platform)\n mcp_server.py # MCP server (43 tools for AI agents)\ntests/\n conftest.py # Fixtures that skip when no display/OCR is available\n test_core.py # Unit tests (no desktop required)\n test_perception.py # Incremental perception (stubbed, runs headless)\n test_atlas.py # Screen memory and its failure modes (headless)\n test_speculate.py # Prediction, settling, and their limits (headless)\n test_e2e.py # End-to-end tests against the real desktop (marked `e2e`)\n```\n\n### Remembering screens\n\nApplications are deterministic — the same dialog has the same layout every time.\nOSWright remembers screens it has read and reuses them on the next visit, across\nsessions: **125 ms cold read → 1.4 ms warm recall (89×)**.\n\nA remembered screen is never trusted on recognition alone. A few regions are\nspot-checked *by pixels* before the layout is reused, so a screen that has\nchanged is rejected rather than acted on. Verification fails closed: a screen\nwith nothing checkable is not remembered at all.\n\nDisable with `--no-atlas`. Remembered screens live in `~/.oswright/atlas.json`.\n\n### Predicting actions instead of observing them\n\nApplications are deterministic — clicking Save produces the same dialog every\ntime — so after the first observation the outcome of an action is already known.\nOSWright learns what actions do and *confirms* the expected screen rather than\nreading it again: **19–23× cheaper than observing** (2.3 ms versus 43–50 ms).\n\nA prediction has to be seen twice before it is trusted, is retired if it proves\nwrong, and is checked the same two ways a remembered screen is. A failed\nprediction is reported to the agent as a `surprise` — the interface did\nsomething it does not normally do, which is worth knowing rather than silently\nabsorbing.\n\n**What a confirmed prediction guarantees:** the layout — the same controls in\nthe same places. Not that every character is identical. A single changed digit\nalters fewer pixels than a blinking caret, so no whole-screen check can separate\nthem at any resolution. Use `observe(force_full=True)` when exact text matters.\n\nDisable with `--no-speculate`.\n\n### Waiting only as long as needed\n\nAction tools used to sleep a fixed 300 ms, chosen for the slowest case, so every\naction paid the worst case. The compositor knows when the screen stops changing,\nso the wait now ends when the interface actually settles:\n\n| | |\n|---|---|\n| Previous fixed sleep | 300 ms |\n| Median actual wait | **61.5 ms** |\n| Saved over a 50-step task | **11.9 s** |\n\n\"Settled\" means no *large* change recently, not no change: a real desktop is\nnever still — a caret and a clock produce a change event every ~18 ms covering\nabout 32 pixels, while genuine UI changes cover tens of thousands.\n\n### Not done yet\n\n- **Wayland input injection**, and macOS `AXTextMarker` as a TextPattern\n equivalent.\n- **A vision-model rung** for surfaces that are neither accessible nor\n text-legible: games, canvases, image editors.\n- **Transitions keyed on more than the previous screen**, for actions whose\n outcome depends on state that is not visible.\n\n### What is measured, and what is not\n\nPerception **cost** and task **success** are both measured on this machine and\nreproducible via `benchmarks/` — across four applications, cheaper perception\ndoes not cost accuracy, and the pixel/accessibility split is measured rather\nthan argued.\n\n### Against Windows-MCP\n\nSame tasks, four scenarios, each graded by the application itself. Neither tool\ngrades itself, and [Windows-MCP](https://github.com/CursorTouch/Windows-MCP) runs\nat its own defaults:\n\n| | Calculator | Explorer | Chrome | Chrome, 2 steps | passed | tokens |\n|---|---|---|---|---|---|---|\n| oswright | 5/5 | 4/5 | 5/5 | 5/5 | 19/20 | **832** |\n| Windows-MCP, snapshot per action | 5/5 | 5/5 | 5/5 | 5/5 | **20/20** | 14,053 |\n| Windows-MCP, snapshot once | 5/5 | 5/5 | 5/5 | 5/5 | **20/20** | 8,214 |\n\n**Read that honestly: Windows-MCP was more reliable, and oswright was 16.9x\ncheaper.** oswright dropped one click in twenty, on a window that had just\nopened.\n\nThe cost difference is structural rather than a tuning win. Windows-MCP returns\nthe screen to the agent -- `Snapshot` renders the accessibility tree as\n`(x,y) button \"Seven\" [action: click]` -- and takes coordinates back, so a\ndescription of the screen is charged to the model's context on every action.\noswright takes the text and returns the outcome.\n\nThe reliability gap may be *caused by* the speed: oswright resolves and clicks\nin ~100 ms, sometimes before a freshly-focused window is ready for input, where\na slower loop gives the application time it never had to ask for. That is a\nhypothesis, not a finding -- adding a pre-action settle made no measurable\ndifference over ten trials, so it is recorded rather than fixed.\n\nThe `Chrome, 2 steps` scenario exists because every other task here is short\nenough that a tool can read the screen once and reuse those coordinates. There\nthe first click moves the controls 325 px down the page, and the snapshot-once\nconfiguration **had to re-read the screen** -- so on tasks whose interface\nmoves, its cheap number does not exist and its real cost is the per-action one.\n\nReproduce with `python benchmarks/bench_head_to_head.py` (setup in the file's\ndocstring).\n\n**What this does not establish:** four short tasks on one laptop. Nothing about\nlong multi-step work, recovery, or product maturity -- Windows-MCP has OAuth,\nanalytics, a watchdog and an installer; oswright has none of those. Its\naccessibility traversal also reads Chrome's page content, which oswright's own\naccessibility rung does not. \"Substantially cheaper per action, at a small\nreliability cost\" is the claim. \"Better product\" is not.\n\n## Development\n\n```bash\npip install -e \".[dev]\"\n\npytest tests/ # everything available on this machine\npytest tests/ -m \"not e2e\" # unit tests only, no desktop needed\nruff check oswright tests # lint\npython benchmarks/bench_pipeline.py # reproduce the performance numbers\npython benchmarks/bench_tasks.py # task success (opens Calculator repeatedly)\n```\n\nDesign decisions, measurements and dead ends are recorded in\n[`docs/ENGINEERING_LOG.md`](docs/ENGINEERING_LOG.md).\n\n## License\n\nMIT\n",
"bytes": 31180,
"sha": "83597683b23a6f656f9cea5d47e9201ade4666539d2638896df72b5afa172fc0",
"repo_slug": "ask-812/oswright",
"fonte": "repo",
"truncated": false,
"api": "https://agentalog.com/api/listings/mcp_io_github_ask_812_oswright_bceb52dd/readme"
}