{
  "markdown": "# glovebox-mcp\n\nA **sandboxed computer-use MCP server** — let an AI agent drive a real browser and desktop apps\n(mouse, keyboard, screenshots, vision grounding), confined to a **nested X11 window** so it can\nnever touch your real screen, files, or other apps.\n\nLike a lab **glovebox**: the agent reaches in and manipulates real applications, sealed off from\neverything else. Bring the sandbox up, log into whatever sites or apps you want to automate *inside\nthat window*, and the agent operates only there — you can watch it live and close it instantly.\n\n> Speaks the [Model Context Protocol](https://modelcontextprotocol.io), so it works with MCP clients\n> like Claude Code. Your host can run Wayland; the sandbox gives the agent a real X server to drive.\n\n![glovebox-mcp — an agent filling a sign-up form inside the sandbox](assets/demo.gif)\n\n<sub>An agent driving a real browser in the sandbox — gliding the cursor, inserting a unicode name (`Nadja Kovačič`), typing, and submitting. All confined to a nested X11 window.</sub>\n\n## Why a nested X11 sandbox?\n\n- Most desktop automation (xdotool, PyAutoGUI) is **X11-only**, but many modern desktops run **Wayland**.\n- **Xephyr** provides a real X server *inside a single window* (`DISPLAY :1`). Everything the agent\n  does — clicks, typing, screenshots — is **confined to that window**, not your real desktop.\n- You stay in control: watch it live, `pkill Xephyr` to close everything.\n\n## Requirements\n\n- **Linux** — the sandbox nests a real X server (Xephyr), so it works even on **Wayland** hosts\n  (via Xwayland). Not macOS/Windows. Developed on Ubuntu; any modern Linux with the packages below.\n- **Python 3.10+** and [`uv`](https://docs.astral.sh/uv/) (used for the virtualenv).\n- **System packages** — `xserver-xephyr` (Xephyr), `openbox`, `scrot`, `x11-utils`, `xdotool`,\n  `wmctrl`, `xclip` (+ `tesseract-ocr` for `basic`). On Debian/Ubuntu the installer **auto-installs**\n  them via `apt` (sudo); on Fedora/Arch it prints the matching `dnf`/`pacman` command. The MCP server\n  itself is distro-agnostic — any Linux with these tools works.\n- A **browser** in the sandbox (Chromium or Chrome).\n- **NVIDIA GPU (≥6 GB VRAM)** — only for the `local` vision mode.\n\n## Install\n\nPick a vision backend and run its one-liner (clone → install). Each one installs the system packages\n(auto via `apt` on Debian/Ubuntu) and the Python deps for that mode, and writes a ready-to-paste\n`mcp-config.json` with your paths.\n\n**`none`** — no local models; your agent reads screenshots itself (lightest, instant):\n\n```bash\ngit clone https://github.com/segentic-lab/glovebox-mcp && cd glovebox-mcp && ./install.sh none\n```\n\n**`basic`** — Tesseract OCR grounding (`parse_screen` → text + coordinates, CPU-only):\n\n```bash\ngit clone https://github.com/segentic-lab/glovebox-mcp && cd glovebox-mcp && ./install.sh basic\n```\n\n**`local`** — OmniParser on an NVIDIA GPU (`parse_screen` → text **+ icons**, pixel-precise; ~4 GB weights, ≥6 GB VRAM):\n\n```bash\ngit clone https://github.com/segentic-lab/glovebox-mcp && cd glovebox-mcp && ./install.sh local\n```\n\nYour choice is written to `.vision-mode` (override per run with the `GLOVEBOX_VISION` env var).\n\n### Works with any MCP client / harness\n\nClaude Code, Cursor, Codex, or your own agent — it's a standard MCP server, not tied to any one host.\nTwo compatibility notes: **`basic`/`local`** return element **coordinates as text**, so they work even\nwith text-only agents; **`none`** relies on the client passing the tool's **screenshots** to a\nmultimodal model (fine for Claude Code, Cursor, and other image-capable MCP clients).\n\n## Quickstart\n\n1. **Start the sandbox** (leave it running):\n   ```bash\n   ./start-display.sh              # 1440×900 Xephyr window with a browser\n   ./start-display.sh 1920x1080    # …or pass a screen size (or set $RES)\n   ```\n   Log into any sites or apps you want to automate **in that window**.\n2. **Register the server** with your MCP client. `install.sh` already wrote **`mcp-config.json`** with\n   your real install path — copy its `glovebox` block into your client's MCP config:\n   ```jsonc\n   { \"mcpServers\": { \"glovebox\": {\n       \"command\": \"/abs/path/to/glovebox-mcp/.venv/bin/python\",   // filled in by install.sh\n       \"args\":    [\"/abs/path/to/glovebox-mcp/server.py\"],\n       \"env\":     { \"DISPLAY\": \":1\" }\n   } } }\n   ```\n   Restart the client so it loads the server.\n3. Ask the agent to screenshot / click / type — it operates **only** on the `:1` window.\n\n> **Driving it with an AI agent?** Paste [`AGENTS.md`](AGENTS.md) into the agent's system prompt — it\n> teaches the observe → act → verify loop, grounding, the upload/unicode gotchas, and when to stop.\n\n## Tools\n\n| Tool | What |\n|---|---|\n| `status()` | Server + sandbox status in one read-only call: version, vision backend, host display, live instances, and which system deps (xdotool, xclip, Xephyr, tesseract, OmniParser weights) are present. Run it first — and paste it into bug reports. |\n| `parse_screen()` | Vision grounding → JSON of detected elements (id, type, label, interactive, pixel-center; capped at 300 per call, flagged via `\"truncated\"`) + a numbered Set-of-Mark image at `/tmp/glovebox_annotated_<N>.png`. (`local` mode: OmniParser on GPU, ~2 s.) |\n| `click_element(id)` | Click an element from the last `parse_screen` (no coordinate guessing). |\n| `screenshot()` | Screenshot of an instance. |\n| `click(x,y)` · `move_mouse` · `scroll` · `drag` · `double_click` | Pointer ops. |\n| `type_text(text)` | **Unicode-safe** typing (ASCII via xdotool; anything with č/š/ž… is inserted via the clipboard, because xdotool's synthetic unicode is silently dropped by some GTK apps). |\n| `press_keys(\"ctrl+a\"/\"Return\"/…)` | Keys/combos (xdotool syntax). |\n| `upload_file(filepath, selector?)` | Attach a local file to a page's `<input type=file>` via the **Chrome DevTools Protocol**. The nested X11 file picker is invisible to automation **and hangs the renderer**, so use this for all uploads — never click an upload button expecting a dialog. Works on Chromium started by `launch_app`/`start-display.sh` (they open a per-instance `--remote-debugging-port`, `9222+N`). **Browser file inputs only** — for native apps see `open_file`. |\n| `open_file(filepath, app?)` | Open a local file in a **native** app on the instance's display (e.g. `app=\"gimp\"`) or via `xdg-open`. GTK apps get the same X11/D-Bus handling as `launch_app`. |\n| `list_files()` | The instance's staging folder `files/<N>/` (under the install dir) + its contents. |\n| `launch_app(command, name?, size?)` · `list_instances()` · `close_instance(n)` | Multi-instance control (see below). |\n| `wait_ms(ms)` · `get_screen_size()` | Timing / sandbox size. |\n\nEvery control tool takes **`instance=N`** and optional **`observe`** / **`settle_ms`** (see below).\nIn `local` mode OmniParser is **lazy-loaded** on first `parse_screen` (~6 s once, then ~2 s/parse).\n\n### Structured, honest responses\n\nEvery tool returns JSON: `{\"ok\": true, \"action\": \"click\", \"instance\": 1, \"detail\": \"clicked (10,20) button 1\", …}`.\nFailures come back with the MCP `isError` flag set and the same JSON shape embedded in the error\ntext — `{\"ok\": false, \"error\": \"…\", \"fix\": \"…\"}` — where **`fix` names the call that unblocks you**\n(e.g. a click on a dead instance says to run `list_instances()` / `launch_app()`; an unknown element\nid says to re-run `parse_screen()`). Silent no-ops are treated as failures too: an invalid keysym,\na zero scroll, closing an instance that isn't running, or non-ASCII typing without `xclip` all\nerror instead of pretending success. `screenshot()` returns a PNG image; `observe=\"screenshot\"|\"parse\"`\nreturns `[json, image]` in one call.\n\n### Vision backend (selectable)\n\n`GLOVEBOX_VISION` env var, or the `.vision-mode` file, or default `local`:\n\n| Mode | `parse_screen` | Needs | When |\n|---|---|---|---|\n| `none` | disabled (returns a note) — use `screenshot()` + reason | nothing (mcp, mss, xdotool) | lightest; let the agent's own vision do grounding |\n| `basic` | Tesseract OCR → text elements + coords | `tesseract-ocr` + `pytesseract` | no GPU; text-only grounding |\n| `local` | OmniParser → text **+ icons** + coords | torch + CUDA + OmniParser weights | best grounding |\n\nSwitch anytime with `./install.sh <mode>` (installs only what that mode needs).\n\n### Multi-instance (a fleet of app windows)\n\nEvery control tool takes **`instance=N`** (default 1 = the `start-display.sh` sandbox). Spin up more —\neach its own Xephyr display/window on the host desktop:\n\n- `launch_app(command, name?, size?)` → starts the next free `:N` running **any** GUI app\n  (`chromium`, `gimp`, `inkscape`, `xterm`, …). Chromium auto-gets X11 flags, a per-instance profile,\n  a remote-debugging port, and D-Bus isolation. Returns the instance id.\n- `list_instances()` · `close_instance(n)`.\n\nBecause each display has its **own cursor**, multiple agents can drive different instances **in parallel** —\none window each. The only shared resource is the GPU for `local`-mode `parse_screen` (it just queues).\nThe host display for new windows is `GLOVEBOX_HOST_DISPLAY` (default `:0`); `XAUTHORITY` is auto-discovered.\n\n### One-call action + observe\n\n`click` · `click_element` · `type_text` · `press_keys` · `scroll` · `drag` · `double_click` take\n**`observe`** (`none` default · `screenshot` · `parse`) and **`settle_ms`**. With `observe=\"screenshot\"`\nthe action returns its result **and** the resulting screen in a single call (with `settle_ms` to let the\npage update first) — no separate screenshot round-trip. Default `none` keeps routine steps cheap; opt\ninto `screenshot`/`parse` on the steps that change the page (navigations, submits).\n\n### Files & uploads\n\nEach instance gets a staging folder **`files/<N>/`** inside the install dir — a stable place to drop\nfiles for that instance (readable by native apps and, since it's under `$HOME`, by snap Chromium too).\n`list_files(instance)` shows the folder and its contents.\n\n- **Browser** `<input type=file>` → **`upload_file(path, instance)`** (via CDP). The nested file\n  picker is invisible to automation and hangs snap Chromium, so never click an upload button expecting\n  a dialog.\n- **Native apps** (GIMP, Inkscape, editors) → **`open_file(path, instance, app=\"gimp\")`**, or just\n  drive the app's own **Open dialog** — unlike the browser's, it's a real visible window you can type a\n  path into (`Ctrl+L` in a GTK file chooser).\n- **Saving / downloads** → apps run as your user, so they can save anywhere you can write. `launch_app`\n  Chromium instances are pre-configured to **download and \"save as\" into `files/<N>/`**; point native\n  apps' Save dialogs there too, then `list_files(instance)` to see the results.\n\n## Maintenance (`local` mode)\n\n`install.sh` clones [OmniParser](https://github.com/microsoft/OmniParser), downloads the v2 weights, and\napplies two patches automatically:\n\n- **PaddleOCR made optional** (this uses easyocr): `OmniParser/util/utils.py`'s\n  `from paddleocr import PaddleOCR` is wrapped in try/except and the module-level `paddle_ocr = PaddleOCR(...)`\n  is guarded with `… if PaddleOCR is not None else None`.\n- **`transformers` is pinned to 4.49.0** — newer releases break Florence-2's remote config.\n\nIf you upgrade OmniParser manually, re-apply the PaddleOCR patch. Weights live in `OmniParser/weights/`.\n\n## Stop\n\n```bash\npkill Xephyr      # closes the sandbox (browser + WM + display)\n```\n\n## Safety\n\n- The agent's input and vision are **scoped to the sandbox display** — it does not see or control your real desktop.\n- The server process runs as your user (shell/file access, like any MCP server); only its **GUI control** is\n  sandboxed to the Xephyr window. For OS-level isolation from your files, run it inside a VM or container.\n- You can watch everything live and close it instantly with `pkill Xephyr`.\n- Automate responsibly — only sites and services you are authorized to use.\n\n## Files\n\n- `server.py` — the MCP server (all tools).\n- `install.sh` — mode-aware installer (`none` / `basic` / `local`).\n- `start-display.sh` — launches the Xephyr sandbox (display + window manager + browser).\n- `AGENTS.md` — drop-in tool-usage instructions for the AI agent (paste into its system prompt).\n- `mcp-config.json` — a ready-to-paste MCP client config snippet.\n\n## Credits\n\n`local` vision mode uses Microsoft's [OmniParser](https://github.com/microsoft/OmniParser)\n(cloned and weights downloaded by `install.sh`, under its own license). Screen capture uses\n[mss](https://github.com/BoboTiG/python-mss); input is driven with\n[xdotool](https://github.com/jordansissel/xdotool). Not affiliated with Microsoft.\n\n## Contributing\n\nShipped **as-is** under MIT. Issues and PRs are welcome, but this is maintained by one person —\n**no support or response time is guaranteed.** If it's useful to you, a ⭐ helps.\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n",
  "bytes": 12883,
  "sha": "fb7d600211dcfcb0f7a38553f179b72cd359b3f982fdb1c11817785a652b6318",
  "repo_slug": "segentic-lab/glovebox-mcp",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_segentic_lab_glovebox_mcp_84d02944/readme"
}