{
  "markdown": "# OpenEye\n\n[![npm version](https://img.shields.io/npm/v/@dumbspacecookie/openeye?color=blue)](https://www.npmjs.com/package/@dumbspacecookie/openeye)\n[![npm downloads](https://img.shields.io/npm/dm/@dumbspacecookie/openeye)](https://www.npmjs.com/package/@dumbspacecookie/openeye)\n[![license](https://img.shields.io/npm/l/@dumbspacecookie/openeye)](./LICENSE)\n[![CI](https://github.com/dumbspacecookie/openeye/actions/workflows/test.yml/badge.svg)](https://github.com/dumbspacecookie/openeye/actions/workflows/test.yml)\n[![github stars](https://img.shields.io/github/stars/dumbspacecookie/openeye?style=social)](https://github.com/dumbspacecookie/openeye)\n\n> **alpha software.** interfaces may change. file issues at\n> github.com/dumbspacecookie/openeye.\n\nevery AR headset maker — HoloLens, Snap Spectacles, Apple, Android — ships a device that can see. none of them ship a brain. every developer building on these platforms has to figure out the intelligence layer themselves, from scratch, every time, for every device. the work doesn't compound. what one team learns on HoloLens doesn't help the team building on WebXR. every deployment is a silo.\n\nOpenEye is the shared brain.\n\na thin piece of software sits on the device, captures what the camera sees, and turns it into a natural-language description of the scene. that description goes to OpenEye. OpenEye runs an AI agent with tools for verifying procedure steps, recalling prior sessions, and writing down what it learns. each session becomes structured memory the agent can search later, and an exportable training trajectory you can fine-tune on.\n\nOpenEye doesn't ship a vision model — **you bring your own**. drop in Claude vision, GPT-4o, Gemini, Groq Llama vision, or a local Ollama model with moondream/llava. a working reference adapter for both cloud and local is in [`examples/vision-adapter/`](examples/vision-adapter/).\n\n---\n\n## install\n\n```bash\nnpm install @dumbspacecookie/openeye\npip install -r node_modules/@dumbspacecookie/openeye/sidecar/requirements.txt\n```\n\nthe Python sidecar (FastAPI + SQLite) handles state and auto-spawns when\nyou create an agent — you don't have to start it yourself.\n\nprefer to hack on it locally? clone instead:\n\n```bash\ngit clone https://github.com/dumbspacecookie/openeye.git\ncd openeye && npm install && npm run build\npip install -r sidecar/requirements.txt\n```\n\n---\n\n## quick start\n\n```typescript\nimport { OpenEyeAgent, setupProviders, makeStreamFn, ANTHROPIC_SONNET } from \"@dumbspacecookie/openeye\";\nimport { describeFrameWithClaude } from \"./examples/vision-adapter/claude-vision-adapter.js\";\nimport * as fs from \"node:fs\";\n\nsetupProviders();\n\nconst agent = await OpenEyeAgent.create({\n  model: ANTHROPIC_SONNET,\n  streamFn: makeStreamFn(),\n  systemPrompt: \"You are a procedure assistant. Verify steps with precision.\",\n  tenantId: \"your-org\",\n});\n\nconst vsId = await agent.client.createVisualSession({\n  deviceType: \"android-tablet\",\n  procedureId: \"bolt-assembly-v1\",\n  procedureName: \"M6 Bolt Assembly\",\n});\n\n// 1. vision adapter (you bring this) turns a camera frame into text\nconst frameBytes = fs.readFileSync(\"./frame.jpg\");\nconst description = await describeFrameWithClaude(\n  frameBytes,\n  \"Operator is installing an M6 bolt. Describe hand position, tool, and bolt state.\",\n);\n\n// 2. OpenEye logs the description and lets the agent verify the step\nconst frameId = await agent.client.logFrame({\n  visualSessionId: vsId!,\n  sequenceNum: 1,\n  sceneDescription: description,\n  stepContext: \"step-1-position-bracket\",\n});\n\nawait agent.prompt(`Frame 1: ${description}\\nVerify step-1-position-bracket.`);\n\nawait agent.client.endVisualSession(vsId!, \"completed\");\nawait agent.captureAndClose({ completed: true, visualSessionId: vsId! });\n```\n\nno API key for the agent? run it locally with Ollama:\n\n```bash\nollama pull llama3.3\n```\n\n```typescript\nimport { ollamaModel } from \"@dumbspacecookie/openeye\";\nconst agent = await OpenEyeAgent.create({ model: ollamaModel(\"llama3.3\"), streamFn: makeStreamFn() });\n```\n\n---\n\n## what this gives you\n\n| capability | OpenEye |\n|---|---|\n| agent runtime + tool calling | yes |\n| persistent session memory (FTS5) | yes |\n| visual frame logging + search | yes |\n| step verification with pass/fail/uncertain outcomes | yes |\n| training data export (ShareGPT JSONL) | yes |\n| DPO preference pair export | yes |\n| HuggingFace dataset push | yes |\n| MCP server (Claude Desktop, Cursor) | yes |\n| reference vision adapters (Claude + Ollama) | yes |\n| vision model | **no — you bring this** |\n| hosted cloud | **no — bring your own ingest endpoint** ([contract](docs/cloud-sync.md)) |\n\nwhat makes this useful is the loop: a session generates pass/fail outcomes against real procedure steps, those outcomes become a reward signal, and the full conversation gets packaged as a ShareGPT trajectory ready for DPO training in TRL, LLaMA-Factory, or Axolotl. you supply the fine-tuning pipeline — OpenEye supplies the data.\n\n---\n\n## how it works\n\nwhen a frame arrives from a device, your vision adapter describes it in plain language. that description goes to the AI agent. the agent has access to a set of tools:\n\n| tool | what it does |\n|---|---|\n| `search_memory` | FTS5 search across all past agent sessions |\n| `search_frames` | FTS5 search across all past frame descriptions |\n| `recall_skill` | retrieve relevant procedural skills for the current task |\n| `write_skill` | persist a new skill doc after completing a complex task |\n| `start_visual_session` | begin a tracked AR/XR session |\n| `end_visual_session` | close a visual session |\n| `log_frame` | record a frame's scene description |\n| `verify_step` | record a step result — pass / fail / uncertain |\n\nstep verifications become the reward signal: `reward = (passes + 0.5 × uncertain) / total`. at the end of a session the whole conversation gets packaged into a ShareGPT trajectory, ready for any DPO-compatible trainer.\n\n**important**: the reward signal reflects the agent's own judgments against scene descriptions, not external ground truth. to use this as real RL data, you should periodically validate trajectories against a human-labeled subset, or use it as supervised data rather than treating it as objective truth. fine-tuning on this raw signal alone risks training the model to be confidently wrong.\n\n---\n\n## use cases\n\n**lead use cases (production-ready):**\n\n**manufacturing & assembly** — bolt installation verification, equipment pre-operation checks, assembly sequence compliance, visual QC at line stations. android tablet or AR overlay over the work area. low regulatory burden, B2B procurement appetite.\n\n**training & onboarding** — any procedure where a trainee needs a second set of eyes that remembers everything it's ever seen. apprentice mechanics, new line operators, lab technicians. mistakes during training don't cost much, which makes this the safest first deployment.\n\n**field service & inspection** — lockout/tagout compliance, PPE verification, pre-job safety checklists, equipment inspection. technicians already carry phones. a sample skill file for LOTO ships in `skills/field-service/`.\n\n**not yet recommended:**\n\n**medical / surgical** — the technical pieces work, but anything influencing surgical decisions is subject to FDA 510(k)/De Novo review. the example skills in `skills/medical/` are illustrative starting points written by an engineer, **not validated clinical protocols**. don't use them as compliance baselines without independent medical and regulatory review.\n\n---\n\n## training data and HuggingFace\n\n```typescript\n// export training trajectories\nconst count = await agent.exportTrajectories(\"./trajectories.jsonl\");\n\n// export DPO preference pairs (TRL/Axolotl-compatible)\nconst pairs = await agent.exportDPOPairs(\"./dpo_pairs.jsonl\");\n\n// push directly to HuggingFace\nconst result = await agent.pushToHub(\"myuser/my-procedure-runs\", {\n  tags: [\"procedure-verification\", \"bolt-assembly\"],\n});\nconsole.log(`published ${result.pushed} trajectories to ${result.url}`);\n```\n\n---\n\n## MCP server\n\nuse OpenEye's tools from Claude Desktop, Cursor, Windsurf, or any MCP-compatible client:\n\n```json\n{\n  \"mcpServers\": {\n    \"openeye\": {\n      \"command\": \"python3\",\n      \"args\": [\"sidecar/mcp_server.py\"],\n      \"env\": { \"ANTHROPIC_API_KEY\": \"sk-ant-...\" }\n    }\n  }\n}\n```\n\nall 8 tools are available immediately.\n\n---\n\n## supported agent models\n\nswap with one line, no other code changes:\n\n| provider | env var |\n|---|---|\n| Anthropic (Claude Opus, Sonnet, Haiku) | `ANTHROPIC_API_KEY` |\n| Groq (Llama 3.3 — fastest, free tier) | `GROQ_API_KEY` |\n| Ollama (local, no key, no cost) | just `ollama pull llama3.3` |\n| OpenAI, Google, Mistral, Bedrock, OpenRouter | see `src/models.ts` |\n| Any OpenAI-compatible endpoint | pass `baseUrl` + `apiKey` to `customModel()` |\n\n---\n\n## data and privacy\n\nraw frame pixels never leave the device **only if your vision adapter runs on-device**. the Ollama adapter in `examples/vision-adapter/` keeps pixels local; the Claude/OpenAI/Gemini cloud adapters do not. **pick the right one for your deployment**.\n\nwhat OpenEye itself stores is the natural-language description of what your vision adapter saw — not the image. opt-in cloud sync is off by default and configurable per data type. every record is scoped to a tenant ID so a single deployment can serve multiple organisations with complete data isolation between them.\n\nopting into cloud sync? OpenEye doesn't run a hosted backend — you operate the receiving endpoint. see [`docs/cloud-sync.md`](docs/cloud-sync.md) for the HTTP contract, retry semantics, idempotency requirements, and row schemas.\n\n### sharing data back to Context (loud opt-in)\n\nOpenEye is built by [Context](https://getcontext.info). To make\nprocedure-verification models better over time, OpenEye can ship\nopted-in trajectory data — completed sessions with their reward signals —\nto Context for training. **This is off by default.** Nothing leaves your\nmachine until you set:\n\n```bash\nexport OPENEYE_CONTEXT_OPTIN=true\nexport OPENEYE_CONTEXT_API_KEY=ctx-...\n```\n\nWhat gets shipped: trajectory ID, model used, reward signal, procedure\ntag, and the agent conversation (with system prompts stripped). What\n**never** gets shipped: tenant IDs, user IDs, system prompts, visual\nsession IDs, skill files, or any raw frame descriptions.\n\nFull disclosure including EU/GDPR notes and revocation procedure:\n[`docs/context-data.md`](docs/context-data.md).\n\nCheck status anytime:\n```bash\ncurl http://127.0.0.1:7770/context/status\n```\n\n---\n\n## deploy\n\n**local** (default — sidecar starts automatically as a subprocess):\n\n```bash\ngit clone https://github.com/dumbspacecookie/openeye.git\ncd openeye && npm install && npm run build\npip install -r sidecar/requirements.txt\n```\n\n**Docker**:\n\n```bash\ndocker run -d -p 7770:7770 -v openeye-data:/data \\\n  -e ANTHROPIC_API_KEY=sk-ant-... \\\n  --name openeye openeye/sidecar\n```\n\n---\n\n## configuration\n\n| env var | default | description |\n|---|---|---|\n| `OPENEYE_HOME` | `~/.openeye` | data directory (SQLite DB + skills + consent marker) |\n| `OPENEYE_PORT` | `7770` | sidecar HTTP port |\n| `OPENEYE_BIND_HOST` | `127.0.0.1` | sidecar bind address (don't change unless you read [docs/security.md](docs/security.md)) |\n| `OPENEYE_SIDECAR_TOKEN` | — | optional shared secret for sidecar HTTP auth — required on shared hosts |\n| `OPENEYE_PYTHON` | `python3` | Python executable |\n| `OPENEYE_WORKERS` | `1` | uvicorn workers — leave at 1 unless you've moved off SQLite |\n| `OPENEYE_LOG_LEVEL` | `INFO` | sidecar log level |\n| `OPENEYE_CLOUD_URL` | — | your cloud endpoint for opt-in sync ([cloud-sync.md](docs/cloud-sync.md)) |\n| `OPENEYE_CLOUD_KEY` | — | bearer token for cloud sync |\n| `OPENEYE_SYNC_INTERVAL` | `60` | cloud sync interval in seconds |\n| `OPENEYE_SYNC_MAX_RETRIES` | `4` | retry attempts per batch on transient failure |\n| `OPENEYE_CONTEXT_OPTIN` | — | set `true` to enable Context training-data sharing ([context-data.md](docs/context-data.md)) |\n| `OPENEYE_CONTEXT_API_KEY` | — | your Context API key |\n| `OPENEYE_CONTEXT_CONSENT_CONFIRMED` | — | set `true` (CI only) to skip the consent attestation prompt |\n| `OPENEYE_CONTEXT_SYNC_INTERVAL` | `300` | Context sync interval in seconds |\n| `OLLAMA_BASE_URL` | `http://localhost:11434` | Ollama endpoint for local vision/agent |\n\n---\n\n## tests\n\n```bash\n# Python sidecar tests (56 passing)\npython -m pytest tests/ -v\n\n# TypeScript integration tests\nnpm test\n\n# TypeScript typecheck\nnpm run typecheck\n\n# Validate community skills\npython sidecar/validate_skill.py skills/\n```\n\n---\n\n## project structure\n\n```\nopeneye/\n├── src/                  # TypeScript source (agent, client, models, tools)\n├── sidecar/              # Python backend (FastAPI server, SQLite state, skills)\n├── skills/               # Example skill protocols (manufacturing, field-service, medical*)\n├── examples/             # Reference adapters and end-to-end demos\n│   └── vision-adapter/   # Claude + Ollama vision adapters\n├── eval/                 # Evaluation benchmark\n├── docs/                 # Operator docs (cloud-sync contract, etc.)\n├── schemas/              # JSON Schema for skill front matter\n├── tests/                # Python + TypeScript tests\n├── dist/                 # Compiled TypeScript output\n├── package.json\n├── tsconfig.json\n└── Dockerfile\n```\n\n*medical skills are illustrative engineering examples, not validated clinical protocols. see \"use cases\" above.\n\n---\n\n## license\n\nMIT — built on [pi-mono](https://github.com/badlogic/pi-mono) (MIT) and [hermes-agent](https://github.com/NousResearch/hermes-agent) (MIT).\n\n---\n\n*— dumbspacecookie*\n",
  "bytes": 13628,
  "sha": "c7b1825802ce87de97f1458e29396836a7be1322da84e64d53743d415c149da7",
  "repo_slug": "dumbspacecookie/openeye",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_dumbspacecookie_openeye_d80edd94/readme"
}