agent-web — URL to LLM-ready markdown: a polite, robots-respecting web page read
URL to clean markdown for LLMs: a polite, robots.txt-respecting web reader. Free, no API key
Open source Repository Open in the app JSON README (API)
About
URL to clean markdown for LLMs: a polite, robots.txt-respecting web reader. Free, no API key
Details
- Kind
- MCP servers
- Topic
- Files & documents
- Publisher
- foomworks
- Origin
- official
- Category
- ferramentas
- Transport
- http
- Version
- 0.2.1
- Last push
- 2026-06-23T21:44:32Z
- Repository state
- ativo
- License
- MIT
- Added
- 2026-08-29 03:02:48
- Updated
- 2026-08-29 03:02:48
- Origin id
io.github.foomworks/agent-web
README
# agent-web **URL → LLM-ready markdown.** A charter-clean, *polite, robots-respecting* web reader for AI agents — honors `robots.txt`, identifies honestly, and **never** bypasses anti-bot/CAPTCHA or paywalls. No account, no API key — free per-call access. 🌐 Live: **https://agent-web.foomworks.workers.dev** · 🔌 MCP: **`https://agent-web.foomworks.workers.dev/mcp`** --- ## What it gives an agent Give it one publicly reachable URL; get back the page as clean, **LLM-ready markdown** — title, word count, and the text — without parsing raw HTML yourself. Server-rendered (static) HTML pages. Ideal for summarizing an article, extracting docs, or feeding a page into an LLM. ## Connect as an MCP server (recommended) Streamable HTTP, JSON-RPC 2.0: ``` https://agent-web.foomworks.workers.dev/mcp ``` | Tool | Cost | Returns | |---|---|---| | `read_url` | free | fetch a URL → full LLM-ready markdown (title, word count, markdown) | | `read_url_preview` | free | the first ~600 characters of the markdown — a cheap look before the full pull | ## Or call it over HTTP ```bash BASE=https://agent-web.foomworks.workers.dev curl -s "$BASE/read?url=https://example.com/&ref=readme" # full markdown curl -s "$BASE/read/preview?url=https://example.com/&ref=readme" # first ~600 chars ``` Discovery: [`/.well-known/mcp.json`](https://agent-web.foomworks.workers.dev/.well-known/mcp.json) (MCP) · [`/.well-known/agent-card.json`](https://agent-web.foomworks.workers.dev/.well-known/agent-card.json) (A2A) · [`/openapi.json`](https://agent-web.foomworks.workers.dev/openapi.json) · [`/policy`](https://agent-web.foomworks.workers.dev/policy) (Acceptable-Use Policy) ## Charter-clean by design (the bright line) This is a *polite* reader, not a scraper: - Honors the origin's `robots.txt` for our user-agent — a disallowed path is **refused without fetching**; an unconfirmable `robots.txt` (5xx/error) is treated conservatively as disallowed. - Identifies honestly with a descriptive user-agent — **never** spoofs a browser. - Read-only `GET` of the **single** URL you supply — no crawl, no proxy rotation. - **Never** bypasses anti-bot/CAPTCHA, paywalls, or login walls. - SSRF-guarded: private/loopback/link-local/internal hosts are blocked and every redirect hop is re-validated. You are responsible for your right to fetch the URL you submit. ## Limits - **Static HTML only** for now — JavaScript-rendered pages, screenshots, and PDF are a later increment. - Body cap ~2 MB, fetch timeout ~12 s. - The service holds no keys and never initiates payments. ## Install the AgentSkill [`SKILL.md`](./SKILL.md) is a portable AgentSkill (ClawHub / skills.sh / Claude Skills). ## About agent-web is operated by **FOOM** — an AI-operated, human-supervised autonomous service. License: [MIT](./LICENSE).