{
  "markdown": "# fetchkit\n\nAI-friendly web content fetching tool designed for LLM consumption. Rust library with CLI, MCP server, and Python bindings.\n\n## Features\n\n- **HTTP fetching** - GET and HEAD methods with streaming support\n- **Pluggable fetchers** - URL-aware dispatch to specialized handlers for repos, docs, feeds, videos, papers, and more\n- **HTML-to-Markdown** - Built-in conversion optimized for LLMs, with fetched relative links/images resolved to absolute URLs\n- **Agent content focus** - Optional low-noise extraction mode for AI agents\n- **Crawl discovery** - Optional bounded same-origin page discovery for AI agents\n- **HTML-to-Text** - Plain text extraction with clean formatting\n- **Content processors** - Post-download extraction for text PDFs, with an extensible registry\n- **Binary detection** - Returns metadata only for unsupported binary formats\n- **Timeout handling** - 1s first-byte, 30s body with partial content on timeout\n- **Safety limits** - 10 MB default decompressed body cap with truncation\n- **URL filtering** - URL-aware allow/block lists for controlled access\n- **SSRF protection** - Resolve-then-check blocks private IPs by default\n- **MCP server** - Model Context Protocol support for AI tool integration\n\n## Built-in Fetchers\n\nFetchkit routes each request through an ordered fetcher registry. Specialized\nfetchers match first; the default fetcher handles everything else.\n\n- `GitHubCodeFetcher` - GitHub source file URLs (`/blob/...`)\n- `GitHubIssueFetcher` - GitHub issue and pull request URLs\n- `GitHubRepoFetcher` - GitHub repository home pages\n- `TwitterFetcher` - X/Twitter status URLs\n- `StackOverflowFetcher` - Stack Overflow and Stack Exchange question URLs\n- `PackageRegistryFetcher` - PyPI, crates.io, and npm package pages\n- `WikipediaFetcher` - Wikipedia article URLs\n- `YouTubeFetcher` - YouTube watch and `youtu.be` URLs\n- `ArXivFetcher` - arXiv abstract and PDF URLs\n- `HackerNewsFetcher` - Hacker News item threads\n- `RSSFeedFetcher` - RSS and Atom feed URLs\n- `DocsSiteFetcher` - docs sites with `llms.txt`/`llms-full.txt` support\n- `DefaultFetcher` - all remaining HTTP/HTTPS URLs with HTML conversion, streaming, timeout handling, and binary detection\n\n## Built-in Content Processors\n\nContent processors run after a fetcher retrieves a bounded response body. They\nselect by final URL, response media type, and requested output, then turn\ndocuments into LLM-friendly content without performing their own network\nrequests.\n\n- `HtmlProcessor` - metadata and focused-content extraction followed by native\n  Markdown or text conversion; accepts a custom `HtmlToMarkdownConverter`\n- `PdfProcessor` - text-based PDF classification and Markdown extraction via\n  [`pdf-inspector`](https://github.com/firecrawl/pdf-inspector); scanned or\n  image-only pages are reported as requiring OCR\n\nCustom processors implement `ContentProcessor` and can be registered in a\n`ContentProcessorRegistry`. Pass that registry to\n`FetcherRegistry::with_content_processors` to retain the built-in fetchers while\ncustomizing post-download processing.\n\n## Installation\n\n### From crates.io (recommended)\n\n```bash\ncargo install fetchkit-cli\n```\n\n### From Git\n\n```bash\ncargo install --git https://github.com/everruns/fetchkit fetchkit-cli\n```\n\n### From Source\n\n```bash\ngit clone https://github.com/everruns/fetchkit\ncd fetchkit\ncargo install --path crates/fetchkit-cli\n```\n\n## CLI Usage\n\n```bash\n# Fetch URL (outputs markdown with frontmatter)\nfetchkit fetch https://example.com\n\n# Output as JSON instead\nfetchkit fetch https://example.com -o json\n\n# Custom user agent\nfetchkit fetch https://example.com --user-agent \"MyBot/1.0\"\n\n# Hardened outbound policy for cluster/data-plane use\nfetchkit fetch https://example.com --hardened\n\n# Discover a small same-origin page map for an agent\nfetchkit fetch https://example.com --content-focus agent --crawl --max-pages 5\n\n# Optional JS/DOM rendering for simple SPAs/docs (requires render-rakers feature)\nfetchkit fetch https://example.com/app --render-rakers\n\n# Show full documentation\nfetchkit --llmtxt\n```\n\nDefault output is markdown with YAML frontmatter:\n\n```markdown\n---\nurl: https://example.com\nstatus_code: 200\nsource_content_type: text/html; charset=UTF-8\nsource_size: 1256\nquality_score: 1.00\nextraction_method: \"full\"\n---\n# Example Domain\n\nThis domain is for use in illustrative examples in documents...\n```\n\nJSON output (`-o json`):\n\n```json\n{\n  \"url\": \"https://example.com\",\n  \"status_code\": 200,\n  \"content_type\": \"text/html\",\n  \"size\": 1256,\n  \"format\": \"markdown\",\n  \"content\": \"# Example Domain\\n\\nThis domain is for use in illustrative examples...\"\n}\n```\n\n## MCP Server\n\nRun as a Model Context Protocol server:\n\n```bash\nfetchkit mcp\n\n# Hardened profile for cluster/data-plane use\nfetchkit mcp --hardened\n```\n\nExposes `fetchkit` tool over JSON-RPC 2.0 stdio transport. Returns markdown with frontmatter (same format as CLI). Compatible with Claude Desktop and other MCP clients.\n\n## Library Usage\n\nAdd to `Cargo.toml`:\n\n```toml\n[dependencies]\nfetchkit = \"0.2\"\n```\n\nOptional rendered fetching:\n\n```toml\n[dependencies]\nfetchkit = { version = \"0.2\", features = [\"render-rakers\"] }\n```\n\n`render-rakers` is not enabled by default. It is lightweight partial rendering:\ninline JavaScript can update the DOM before markdown/text conversion, but it is\nnot a full browser engine. FetchKit blocks rakers-initiated subresource network\naccess in this mode; the initial page still uses FetchKit's normal URL, DNS,\nproxy, timeout, and size policies.\n\n### Basic Fetch\n\n```rust\nuse fetchkit::{fetch, FetchRequest};\n\n#[tokio::main]\nasync fn main() {\n    let request = FetchRequest::new(\"https://example.com\").as_markdown();\n\n    let response = fetch(request).await.unwrap();\n    println!(\"{}\", response.content.unwrap_or_default());\n}\n```\n\n### With Tool Builder\n\n```rust\nuse fetchkit::{FetchRequest, ToolBuilder};\n\nlet tool = ToolBuilder::new()\n    .enable_markdown(true)\n    .enable_text(false)\n    .user_agent(\"MyBot/1.0\")\n    .allow_prefix(\"https://docs.example.com\")\n    .block_prefix(\"https://internal.example.com\")\n    .build();\n\nlet request = FetchRequest::new(\"https://example.com\");\nlet response = tool.execute(request).await.unwrap();\n```\n\n### Toolkit Contract Surface\n\n```rust\nuse fetchkit::ToolBuilder;\n\nlet builder = ToolBuilder::new().enable_save_to_file(true);\nlet tool = builder.build();\n\nassert_eq!(tool.name(), \"web_fetch\");\nassert_eq!(tool.display_name(), \"Web Fetch\");\n\nlet definition = builder.build_tool_definition();\nlet mut service = builder.build_service();\n```\n\n### Hardened Tool Profile\n\n```rust\nuse fetchkit::Tool;\n\nlet tool = Tool::builder()\n    .hardened()\n    .allow_prefix(\"https://docs.example.com\")\n    .build();\n```\n\n## Python Bindings\n\n```bash\npip install fetchkit\n```\n\n```python\nfrom fetchkit_py import fetch, FetchRequest, FetchkitTool\n\n# Simple fetch\nresponse = fetch(\"https://example.com\", as_markdown=True)\nprint(response.content)\n\n# With configuration\ntool = FetchkitTool(\n    enable_markdown=True,\n    user_agent=\"MyBot/1.0\",\n    allow_prefixes=[\"https://docs.example.com\"]\n)\nresponse = tool.fetch(\"https://example.com\")\n```\n\n## Request Fields\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `url` | string | URL to fetch (required, `http://` or `https://`) |\n| `method` | enum? | `GET` (default) or `HEAD` |\n| `as_markdown` | bool? | Convert HTML to markdown |\n| `as_text` | bool? | Convert HTML to plain text |\n| `save_to_file` | string? | Non-blank destination path; validated by `FileSaver` before fetching |\n| `content_focus` | string? | `\"full\"`/unset returns everything; `\"main\"` strips semantic boilerplate; `\"readable\"` selects article-like content; `\"agent\"` selects the best low-noise strategy for AI agents |\n| `crawl` | bool? | Fetch the seed URL, then discover and fetch bounded same-origin pages |\n| `max_pages` | int? | Maximum crawl pages, including the seed; default 5, max 20 |\n| `if_none_match` | string? | ETag for conditional `If-None-Match` |\n| `if_modified_since` | string? | Timestamp for conditional `If-Modified-Since` |\n| `render` | string? | `\"rakers\"` to opt into rendered fetch when enabled |\n\n## Response Fields\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `url` | string | Fetched URL |\n| `status_code` | int | HTTP status code |\n| `content_type` | string? | Content-Type header |\n| `size` | int? | Content size in bytes |\n| `last_modified` | string? | Last-Modified header |\n| `etag` | string? | ETag header (use for next conditional request) |\n| `filename` | string? | From Content-Disposition |\n| `format` | string? | `\"markdown\"`, `\"text\"`, `\"raw\"`, or a fetcher-specific format |\n| `content` | string? | Page content |\n| `truncated` | bool? | True if content was cut off |\n| `method` | string? | `\"HEAD\"` for HEAD requests |\n| `error` | string? | Error message if failed |\n| `saved_path` | string? | Filesystem path when `save_to_file` succeeded |\n| `bytes_written` | int? | Bytes saved to file |\n| `metadata` | object? | Structured `PageMetadata` (title, description, links, headings, extraction method, …) |\n| `quality` | object? | Agent-facing `PageQuality` (score, warnings, link density, suggested next action) |\n| `crawl` | object? | Bounded crawl discovery result with visited page summaries |\n| `word_count` | int? | Word count of returned content |\n| `redirect_chain` | string[] | URLs visited during redirects (empty if none) |\n| `is_paywall` | bool? | Heuristic paywall signal (soft, not guaranteed) |\n| `rendered_by` | string? | Rendering backend used before conversion, e.g. `\"rakers\"` |\n\n## Error Handling\n\nErrors are returned in the `error` field:\n\n- `InvalidUrl` - Malformed URL\n- `UrlBlocked` - URL blocked by filter\n- `NetworkError` - Connection failed\n- `Timeout` - Request timed out\n- `HttpError` - 4xx/5xx response\n- `ContentError` - Failed to read body\n- `BinaryContent` - Binary content has no registered processor\n\n## Security\n\nFetchkit blocks connections to private/reserved IP ranges by default, preventing SSRF attacks when used in server-side or AI agent contexts.\n\n**Blocked by default:** loopback, private networks (10.x, 172.16-31.x, 192.168.x), link-local (169.254.x including cloud metadata), IPv6 equivalents, multicast, and other reserved ranges.\n\n```rust\n// Default: private IPs blocked (safe for production)\nlet tool = Tool::default();\n\n// Explicit opt-out for local development only\nlet tool = Tool::builder()\n    .block_private_ips(false)\n    .build();\n```\n\nDNS pinning prevents DNS rebinding attacks. IPv6-mapped IPv4 addresses are canonicalized before validation.\nRedirects are followed manually in the default fetcher so each hop is revalidated against scheme and DNS policy. Allow/block prefixes are matched against parsed URLs rather than raw strings, which prevents lookalike host overmatches such as `allowed.example.com.evil.test`.\nProxy environment variables are ignored by default. Use the hardened profile for cluster-facing deployments and opt in with `ToolBuilder::respect_proxy_env(true)` only when it is part of an intentional egress design.\n\nSee the [`knowledge/security/threat-model.md`](knowledge/security/threat-model.md) concept for the full threat model.\nSee [`docs/hardening.md`](docs/hardening.md) for deployment guidance.\n\n## Configuration\n\n### Timeouts And Limits\n\n- **First-byte**: 1 second (connect + initial response)\n- **Body**: 30 seconds total\n- **Body size**: 10 MB decompressed content by default\n\nPartial content is returned on body timeout or body-size limit with `truncated: true`.\n\n### PDF And Binary Content\n\nWhen Markdown is requested, text-based PDFs are downloaded within the configured\nbody limit and converted to Markdown. PDF parsing runs locally; no OCR service or\nadditional network request is used. Scanned/image-only PDFs report `use_ocr` as the\nsuggested next action.\n\nOther binary content returns metadata only:\n- Images, audio, video, fonts\n- Archives (zip, tar, rar, 7z)\n- Office documents\n\n### HTML Conversion\n\nHTML is automatically converted to markdown:\n- Headers: `h1-h6` → `#` to `######`\n- Lists: Proper nesting with 2-space indent\n- Code: Language-aware, collision-safe fences and inline backticks\n- Links/images: Titles, relative-URL resolution, and highest-resolution `srcset`\n- Tables: Valid Markdown with formatted cells and escaped pipes\n- Rich content: Footnotes, LaTeX math, callouts, figures, highlights,\n  strikethrough, details, and task checkboxes\n- Strips: document head, scripts, styles, templates, iframes, SVGs\n- Adds a bounded [Agent resources](docs/agent-discoverability.md) navigation appendix\n  when discoverable resources are available\n\n## License\n\nMIT. See [Third-Party Notices](THIRD_PARTY_NOTICES.md) for adapted components.\n",
  "bytes": 12717,
  "sha": "0a6ca7435f29cf9e99f3ec739ea3046660536afcc6323009ba7a1845f0d890f8",
  "repo_slug": "everruns/fetchkit",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/okf_everruns_fetchkit_knowledge_index_md_2e3588b6/readme"
}