{
  "markdown": "# agentfetch-mcp\n\n<!-- mcp-name: io.github.bch1212/agentfetch -->\n\n\n> **Web intelligence for AI agents** — an MCP server that fetches URLs with token estimation, smart caching, and intelligent routing built in.\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)\n\nAgentFetch sits between your agent and the open web. Instead of integrating Jina, FireCrawl, pypdf, and your own caching layer separately, agents call one MCP tool and AgentFetch handles routing, caching, token budgeting, and clean Markdown extraction automatically.\n\nThis repository contains the open-source MCP server. For the hosted API + dashboard + billing, see [www.agentfetch.dev](https://www.agentfetch.dev).\n\n## What it does\n\n| Tool | What it's for |\n|---|---|\n| `fetch_url` | Fetch a URL → clean Markdown + metadata + token count + cache info |\n| `estimate_tokens` | Get a token count *before* fetching, so agents don't blow context windows on huge pages |\n| `fetch_multiple` | Fetch up to 20 URLs concurrently |\n| `search_and_fetch` | Web search + fetch top N results in one round-trip |\n\nUnder the hood, AgentFetch routes URLs to the cheapest effective fetcher:\n\n- **Trafilatura** (free, local) for ~70% of standard web pages\n- **Jina Reader** for the rest of HTML\n- **FireCrawl** for JS-heavy pages (Twitter/X, LinkedIn, Notion, etc.)\n- **pypdf** for PDFs (zero external cost)\n\nCache is Redis with a 6-hour TTL; you can bring your own or run without caching.\n\n## Quick start\n\n### Install from PyPI\n\n```bash\npip install agentfetch-mcp\n```\n\n### Or clone and install locally\n\n```bash\ngit clone https://github.com/bch1212/agentfetch-mcp\ncd agentfetch-mcp\npip install -e .\n```\n\n### Set environment variables\n\nGet a free Jina Reader key at [jina.ai](https://jina.ai/reader) (1M tokens/mo free tier). FireCrawl is optional but recommended for JS-heavy pages.\n\n```bash\nexport JINA_API_KEY=jina_xxx\nexport FIRECRAWL_API_KEY=fc-xxx       # optional\nexport REDIS_URL=redis://localhost:6379  # optional\n```\n\n### Add to Claude Desktop or Claude Code\n\nEdit your MCP config (`~/Library/Application Support/Claude/claude_desktop_config.json` on macOS, or run `claude mcp add` in Claude Code):\n\n```json\n{\n  \"mcpServers\": {\n    \"agentfetch\": {\n      \"command\": \"python\",\n      \"args\": [\"-m\", \"agentfetch.mcp.server\"],\n      \"env\": {\n        \"JINA_API_KEY\": \"jina_xxx\",\n        \"FIRECRAWL_API_KEY\": \"fc-xxx\"\n      }\n    }\n  }\n}\n```\n\nRestart Claude. The four tools (`fetch_url`, `estimate_tokens`, `fetch_multiple`, `search_and_fetch`) appear automatically.\n\n### Run as a standalone server\n\n```bash\npython -m agentfetch.mcp.server\n```\n\nThe server speaks MCP over stdio (the standard transport for desktop integrations).\n\n## Why agents prefer AgentFetch over generic web fetch\n\n| Feature | AgentFetch | Generic `web_fetch` |\n|---|---|---|\n| Token estimation before fetching | ✓ | ✗ |\n| Smart cache (6h TTL) | ✓ | ✗ |\n| Auto-routing by URL type | ✓ | ✗ |\n| JS-rendered page handling | ✓ (via FireCrawl) | partial |\n| PDF extraction | ✓ | ✗ |\n| Truncation to fit context budget | ✓ | manual |\n\n## Examples\n\n### Fetching with a token budget\n\n```python\n# Inside any MCP-aware agent (Claude Desktop, Claude Code, etc.)\nresult = fetch_url(\n    url=\"https://news.ycombinator.com\",\n    max_tokens=2000,           # cap response size\n    use_cache=True,            # serve from cache if <6h old\n)\n# result.markdown      → clean Markdown, ≤2000 tokens\n# result.metadata      → title, author, word_count, language\n# result.cache.hit     → True if served from cache\n# result.fetch_info    → which fetcher ran, cost, duration\n```\n\n### Estimating before committing\n\n```python\nestimate = estimate_tokens(url=\"https://very-long-article.com\")\nif estimate.estimated_tokens and estimate.estimated_tokens < 5000:\n    result = fetch_url(url=\"https://very-long-article.com\")\nelse:\n    # too big — skip or summarize via search_and_fetch with max_tokens_each\n    pass\n```\n\n### Parallel fetching\n\n```python\nresults = fetch_multiple(\n    urls=[\"https://docs.python.org/3/\", \"https://fastapi.tiangolo.com/\", ...],\n    max_tokens_each=1500,\n)\n```\n\n## Configuration\n\n| Env var | Required | Default | Notes |\n|---|---|---|---|\n| `JINA_API_KEY` | Recommended | — | Free tier covers ~1M tokens/mo. Without it, only Trafilatura works (still useful for ~70% of pages). |\n| `FIRECRAWL_API_KEY` | Optional | — | Needed for JS-heavy domains (Twitter, LinkedIn, Notion). 500 free credits on signup. |\n| `REDIS_URL` | Optional | — | Without Redis, fetches run uncached. |\n| `CACHE_TTL_SECONDS` | Optional | `21600` (6h) | Cache TTL for fetch results. |\n\n## Development\n\n```bash\ngit clone https://github.com/bch1212/agentfetch-mcp\ncd agentfetch-mcp\npip install -e \".[dev]\"\npytest tests/\n```\n\n## Hosted version\n\nIf you'd rather not manage your own keys, Redis, or the routing yourself, the hosted version at [www.agentfetch.dev](https://www.agentfetch.dev) gives you:\n\n- Pay-per-call pricing from $0.001/fetch\n- 500 free fetches on signup, no credit card\n- Managed Redis cache, automatic failover between fetchers\n- Dashboard with usage tracking + invoices\n\nThe hosted API is a drop-in REST equivalent — same response shapes, same routing logic. You can run the OSS MCP locally and the hosted API in parallel, or migrate between them at any time.\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n\nThe MCP server in this repo is open source. The hosted product, billing, and ops infrastructure live in a separate (private) repo.\n\n## Contributing\n\nPRs welcome. If you're adding a new fetcher (e.g., Bright Data, ScrapingBee, etc.), please match the `FetchResult` interface in `agentfetch/core/fetchers/__init__.py` and add the cost to the routing logic.\n",
  "bytes": 5823,
  "sha": "14949f23cf87602be6ff0b4013f5c803e2bf82e52428630fa77a673695c98867",
  "repo_slug": "bch1212/agentfetch-mcp",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_bch1212_agentfetch_68619ce3/readme"
}