{
  "markdown": "# SmartFetch V1.10.6 — accurate x402 payment onboarding\n\nSmartFetch takes a public web URL and returns clean agent-ready text, Markdown, links, metadata, and the retrieval method used. It tries cheap HTTP retrieval first and falls back to a real Chromium browser when needed.\n\nSmartFetch V1.10.5 is live in production and is the currently published\nOfficial MCP Registry version. This branch prepares V1.10.6 for review; it\ndoes not itself deploy the service or publish a Registry release.\n\n## V1.10.6 payment onboarding\n\n- OpenAPI models the real request-ID-only HTTP 402 body and documents the\n  base64-encoded x402 v2 `PAYMENT-REQUIRED` challenge header.\n- The paid retry request uses `PAYMENT-SIGNATURE`; a successfully settled\n  response carries `PAYMENT-RESPONSE`. The documented payment requirement is\n  generated from the same runtime scheme, network, amount, asset, and public\n  payee as the server.\n- `/docs` and `/llms.txt` display payment details from the active generated\n  requirement and link to the tested Python HTTP buyer plus the official x402\n  TypeScript/Python buyer guide.\n- SmartFetch verifies authorization before retrieval and settles only after\n  successful delivery. Retrieval responses with status 400 or higher,\n  including HTTP 502 failures, are returned without settlement.\n\n## V1.10.5 method guidance and MCP logging\n\n- Free `GET /fetch` and `HEAD /fetch` requests return HTTP 405 with\n  `Allow: POST`; GET includes concise links to OpenAPI, documentation, and\n  x402 discovery while HEAD has no response body.\n- OpenAPI continues to expose only paid `POST /fetch`. The guidance methods do\n  not invoke payment verification, retrieval, or MCP execution.\n- MCP SDK DEBUG/INFO records are routed to stdout, while\n  WARNING/ERROR/CRITICAL records and tracebacks remain on stderr. SmartFetch's\n  structured activity logger and query-string redaction remain unchanged.\n\n## V1.10.4 retrieval diagnostics\n\n- Failed HTTP retrievals now carry finite, typed diagnostic codes through the\n  existing browser fallback and final HTTP 502 path without changing retrieval\n  strategy, retries, timeouts, fallback, or client response behavior.\n- The existing final `tool_failed` activity event can include only a normalized\n  target hostname, strategy, phase, finite failure code, attempt flags, and an\n  optional bounded upstream HTTP status. IP literals are recorded only as the\n  categorical value `ip-literal`.\n- URL credentials, ports, paths, queries, fragments, exception messages,\n  response content, and payment material never enter these diagnostic fields.\n\n## V1.10.3 access-log privacy patch\n\n- Uvicorn application access logs retain the request method, route path, HTTP\n  version, and response status while omitting the entire query string.\n- Query parameters remain available to routing and request handlers; the\n  change is confined to access-log formatting and does not alter MCP, x402,\n  payment, retrieval, or discovery behavior.\n- SmartFetch's structured activity events remain allowlisted and continue to\n  provide request IDs, client categories, statuses, and durations where those\n  fields are currently available.\n- Hosting-provider infrastructure logs are outside the application logging\n  boundary and may require separate provider controls if they record full\n  request targets independently.\n\n## V1.10.2 AgentCash discovery\n\n- `/openapi.json` now includes the canonical AgentCash `info.x-guidance` and\n  operation-level `x-payment-info` fields for paid `POST /fetch` discovery.\n- AgentCash pricing metadata is derived from SmartFetch's configured x402\n  price. The live x402 challenge remains authoritative for the exact scheme,\n  network, asset contract, atomic amount, public payee, and settlement.\n- V1.10.2 does not change retrieval, rendering, security, payment verification,\n  settlement, facilitator selection, or any MCP tool contract.\n\n## V1.10.1 compatibility patch\n\n- Activity events are emitted as one compact JSON line through a dedicated\n  plain stdout handler, with explicit `message` and `level` fields.\n- The community x402 manifest identifies the paid HTTP resource and structured\n  MCP endpoint without an ambiguous bare tool-name array. The four tools remain\n  discoverable through MCP `tools/list` and `/meta`.\n- `/openapi.json` describes the active x402 v2 exact network, USDC on Base mainnet\n  asset, configured price, and atomic amount for `POST /fetch`.\n\n## What changed in V1.10\n\n- V1.10 adds privacy-safe structured activity events for MCP discovery, tool\n  attempts, x402 challenges, verified payments, execution, and settlement.\n- `GET /.well-known/x402` provides a free, proxy-aware community discovery\n  manifest for buyers such as Agent402. It is a community convention rather\n  than a finalized x402 Foundation protocol endpoint.\n- Runnable Python and TypeScript MCP buyers list tools for free, enforce a\n  `$0.005` maximum payment, handle challenge → pay → retry, and print the\n  result plus settlement receipt.\n- The production MCP server exposes four live paid tools backed by the same\n  retrieval engine: `fetch_webpage`, `webpage_to_markdown`,\n  `extract_webpage_text`, and `render_webpage`.\n- Every tool has its own official x402 MCP Bazaar declaration and unique\n  `mcp://tool/<tool-name>` payment resource while sharing the existing exact\n  `$0.005` payment requirements.\n- HTTP `POST /fetch` and MCP tool execution are live with x402 `exact`\n  payments on Base mainnet at `$0.005` per execution.\n- Free `/.well-known/x402`, `/docs`, `/openapi.json`, `/llms.txt`,\n  `/robots.txt`, `/sitemap.xml`, and `/meta` routes serve humans, crawlers,\n  and agents.\n- Runtime discovery links use FastAPI/Starlette's proxy-aware request scheme\n  and host. The Railway hostname is not embedded in runtime discovery output.\n- SmartFetch V1.10.5 is the currently published Official MCP Registry release;\n  V1.10.6 requires a separate publication after release.\n\nAll V1.8 HTTP payment, Bazaar, Registry, and native MCP behavior remains\nunchanged:\n\n- Base Sepolia remains the default x402 network and continues to use `https://x402.org/facilitator` without CDP credentials.\n- Base mainnet can use Coinbase's authenticated CDP facilitator when explicitly selected.\n- Mainnet startup requires both CDP API credentials and fails closed for invalid credentials or missing facilitator support.\n- `/meta` distinguishes enabled testnet and mainnet payment modes.\n- `cdp-sdk`, FastAPI, Uvicorn, and x402 are exactly pinned so infrastructure upgrades are explicit.\n\nThe retrieval engine, extraction/browser behavior, SSRF protections, request limits, rate limiting, and concurrency controls are unchanged.\n\n## API\n\n### GET /health\n\nReturns service health/version.\n\n### GET /meta\n\nMachine-readable service description.\n\n### POST /fetch\n\n```json\n{\n  \"url\": \"https://example.com/article\",\n  \"max_chars\": 20000,\n  \"force_browser\": false\n}\n```\n\n`max_chars` is optional (minimum 1,000; default 20,000; maximum 50,000).\n\nExample response fields:\n\n```json\n{\n  \"success\": true,\n  \"requested_url\": \"https://example.com/article\",\n  \"final_url\": \"https://example.com/article\",\n  \"status_code\": 200,\n  \"render_method\": \"http\",\n  \"title\": \"…\",\n  \"content\": \"…\",\n  \"markdown\": \"…\",\n  \"links\": [],\n  \"word_count\": 1000,\n  \"content_hash\": \"…\",\n  \"truncated\": false,\n  \"elapsed_ms\": 350,\n  \"request_id\": \"…\",\n  \"service_version\": \"1.10.6\"\n}\n```\n\nWhen x402 payment protection is enabled, the `PAYMENT-REQUIRED` header for\n`POST /fetch` includes a Bazaar declaration with the `url`, `max_chars`, and\n`force_browser` input contract plus a representative successful response.\n`/`, `/health`, `/meta`, `/.well-known/x402`, `/docs`, `/openapi.json`,\n`/llms.txt`, `/robots.txt`, and `/sitemap.xml` remain free and do not advertise\npaid Bazaar metadata.\n\n### HTTP x402 buyer flow\n\n1. Send the JSON request without payment.\n2. Decode `PAYMENT-REQUIRED` and validate its scheme, network, asset, amount,\n   and public payee before signing.\n3. Sign through an official capped x402 client and retry with\n   `PAYMENT-SIGNATURE`.\n4. After successful settlement, read `PAYMENT-RESPONSE`.\n\nSmartFetch verifies authorization before retrieval and settles only after\nsuccessful delivery. Retrieval responses with status 400 or higher, including\n502 failures, are returned without settlement. See the tested\n[Python HTTP buyer](scripts/paid_fetch_mainnet_test.py) and the official\n[x402 TypeScript/Python buyer guide](https://docs.x402.org/getting-started/quickstart-for-buyers).\n\nNever place a private key or recovery phrase in a URL, request body, log,\nexample, command-line argument, or repository file. Never commit secret-bearing\n`.env` files. Use hidden interactive input, a platform-injected secret, or an\napproved wallet/secret-management service.\n\n## Free discovery routes\n\n- `/.well-known/x402` is a community buyer manifest with the paid HTTP\n  resource, structured MCP endpoint, configured price/network, and dynamic\n  machine-readable links.\n- `/docs` is a concise human- and crawler-readable service guide.\n- `/openapi.json` is the explicit OpenAPI 3.1 contract for `POST /fetch`.\n- `/llms.txt` summarizes endpoints, tools, price, and source for agents.\n- `/robots.txt` allows public crawling and points to `/sitemap.xml`.\n- `/sitemap.xml` lists only free discovery content, not `/fetch` or `/mcp`.\n\nAll absolute links in these responses and `/meta` are generated from the\nframework-resolved public request URL. The existing Uvicorn/Railway trusted\nproxy configuration supplies the external HTTPS scheme; application code does\nnot parse raw forwarded headers.\n\n## AgentCash discovery\n\nAgentCash-compatible agents can inspect SmartFetch without making a paid\nrequest:\n\n```bash\nnpx agentcash@latest discover https://smartfetch-production-ea53.up.railway.app\nnpx agentcash@latest check https://smartfetch-production-ea53.up.railway.app/fetch\n```\n\nThe current official AgentCash CLI also supports this optional free command:\n\n```bash\nnpx agentcash@latest add https://smartfetch-production-ea53.up.railway.app\n```\n\n`add` discovers the origin, installs its generated skill into supported agent\nclients, and adds the origin to the AgentCash MCP server description. It is an\noptional persistence/install step; it does not invoke SmartFetch `POST /fetch`.\n\n`agentcash fetch` is a paid action. Do not run it without explicit approval and\na buyer-controlled spending policy. A future approved call would use this\nplaceholder form:\n\n```bash\n# PAID: do not run without explicit approval.\nnpx agentcash@latest fetch \\\n  https://smartfetch-production-ea53.up.railway.app/fetch \\\n  -m POST \\\n  -b '<EXPLICITLY_APPROVED_SMARTFETCH_REQUEST>'\n```\n\nUse SmartFetch as a paid fallback when ordinary access to a public HTTP(S)\nwebpage fails, returns incomplete content, requires JavaScript rendering, or\nneeds clean text or Markdown. SmartFetch does not bypass CAPTCHAs,\nauthentication, paywalls, robots restrictions, or access controls, and cannot\nguarantee access to every website.\n\n## MCP\n\nRemote MCP clients connect to `/mcp` using Streamable HTTP. The production\nserver exposes exactly four live tools:\n\n```text\nfetch_webpage\n  url: required string\n  max_chars: optional integer (default 20000, minimum 1000, maximum 50000)\n  force_browser: optional boolean (default false)\n\nwebpage_to_markdown\n  url: required string\n  max_chars: optional integer (default 20000, minimum 1000, maximum 50000)\n  force_browser: optional boolean (default false)\n\nextract_webpage_text\n  url: required string\n  max_chars: optional integer (default 20000, minimum 1000, maximum 50000)\n  force_browser: optional boolean (default false)\n\nrender_webpage\n  url: required string\n  max_chars: optional integer (default 20000, minimum 1000, maximum 50000)\n```\n\n`fetch_webpage` returns the complete existing SmartFetch result.\n`webpage_to_markdown` returns Markdown and core retrieval metadata without\nduplicating the full text. `extract_webpage_text` returns clean text and core\nmetadata without Markdown. `render_webpage` always starts with browser\nrendering and returns the complete result. Every tool uses the same SSRF\nvalidation, executor, request timeout, concurrency cap, output cap, HTTP\nretrieval, and browser behavior as `POST /fetch`; none makes an HTTP request\nback to the public API.\n\nWhen x402 is enabled, an unpaid `tools/call` returns the native MCP x402 payment\nchallenge. A valid payment is verified before retrieval and settled once after\nsuccessful tool execution. The network, public payee, and price come from\n`X402_NETWORK`, `X402_PAY_TO`, and `X402_PRICE`. No seller private key or wallet\nsecret is accepted or required.\n\nEach unpaid MCP payment challenge includes a matching Bazaar declaration with\ntransport `streamable-http`, an accurate input/output contract, and its unique\nresource: `mcp://tool/fetch_webpage`, `mcp://tool/webpage_to_markdown`,\n`mcp://tool/extract_webpage_text`, or `mcp://tool/render_webpage`. The existing\nHTTP Bazaar resource for `POST /fetch` is separate and unchanged.\n\nThe root `server.json` describes the public remote endpoint registered with the\nOfficial MCP Registry. V1.10.5 remains the currently published Registry\nversion; V1.10.6 requires a separate publication after release.\nRegistry metadata keeps its required fixed remote URL; runtime discovery routes\nderive their URLs from each proxy-aware request. The MCP Bazaar declarations do\nnot assert indexing by Coinbase Bazaar or downstream MCP directories.\n\nUseful natural discovery phrases include: read a webpage, fetch a webpage,\nscrape a URL, retrieve website content, convert webpage to Markdown, extract\nclean text from a website, read a JavaScript website, browser render a webpage,\nweb scraping for an AI agent, get webpage content for an agent, website to\nMarkdown, and fetch public URL.\n\n## Paying MCP client examples\n\n- [Python client](examples/python/paid_mcp_client.py)\n- [TypeScript client](examples/typescript/paid-mcp-client.ts)\n- [Setup and funding guide](examples/README.md)\n\nBoth clients connect over Streamable HTTP, run free `tools/list`, enforce a\nmaximum `$0.005` payment, call `fetch_webpage`, and print the settlement\nreceipt. **Running either client can spend real USDC on Base mainnet.** Use a\nseparate low-balance CDP-managed wallet and keep `CDP_API_KEY_ID`,\n`CDP_API_KEY_SECRET`, and `CDP_WALLET_SECRET` in secret storage.\n\n## Activity events\n\nSmartFetch writes compact JSON events named `mcp_initialized`, `tools_listed`,\n`tool_call_attempted`, `payment_challenged`, `payment_verified`,\n`tool_started`, `tool_completed`, `tool_failed`, and `payment_settled`.\n\nThe event schema is deliberately allowlisted. It can contain only timestamp,\nmessage/level, opaque request ID, transport, tool, route, stage/outcome,\nstatus, duration, payment presence/stage/network/asset/amount, a finite safe\nfailure reason, and a coarse client category. A final failed retrieval may also\ninclude the normalized target hostname, retrieval strategy/phase, a finite\nfailure code, attempt flags, and a bounded upstream status. It never includes\ncomplete URLs, URL credentials/ports/paths/queries/fragments, webpage content,\nrequest bodies, headers, payment signatures/payloads, wallet or payee\naddresses, IP literals, CDP credentials, transaction hashes, or exception\nmessages.\n\n## Run locally\n\n```bash\npython -m pip install -r requirements.txt\npython -m smartfetch.server\n```\n\nThen:\n\n```bash\ncurl http://127.0.0.1:8787/health\ncurl -X POST http://127.0.0.1:8787/fetch \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"url\":\"https://example.com\"}'\n```\n\nWindows users can run `setup_windows.bat`, then `start_windows.bat`.\n\nWith `X402_ENABLED` unset or false, startup requires no payment settings and `/fetch` remains free for local and remote testing.\n\n## x402 payment modes\n\nPayment remains disabled unless explicitly enabled. Base Sepolia is the default and requires only the public receiving address:\n\n```text\nX402_ENABLED=true\nX402_PAY_TO=0xYOUR_PUBLIC_RECEIVING_ADDRESS\nX402_PRICE=$0.005\nX402_NETWORK=eip155:84532\n```\n\n`X402_PRICE` and `X402_NETWORK` use the values shown when omitted. The default price remains `$0.005`.\n\nBase mainnet must be selected explicitly and uses Coinbase's authenticated CDP facilitator:\n\n```text\nX402_ENABLED=true\nX402_PAY_TO=0xYOUR_PUBLIC_RECEIVING_ADDRESS\nX402_PRICE=$0.005\nX402_NETWORK=eip155:8453\nCDP_API_KEY_ID=YOUR_CDP_API_KEY_ID\nCDP_API_KEY_SECRET=YOUR_CDP_API_KEY_SECRET\n```\n\nMainnet requires both CDP credentials. Missing, invalid, or unusable credentials, unsupported Base-mainnet exact payments, or any facilitator/middleware initialization failure aborts startup. SmartFetch never falls back to free access or the testnet facilitator when mainnet is selected. Every network other than Base Sepolia and Base mainnet is rejected.\n\nAmong the HTTP API routes, only `POST /fetch` is protected; MCP tool execution\nis paid separately at the MCP layer. `/health`, `/`, `/meta`,\n`/.well-known/x402`, `/docs`, `/openapi.json`, `/llms.txt`, `/robots.txt`, and\n`/sitemap.xml` always remain free. Enabled `/meta` responses report\n`x402-enabled-testnet` or\n`x402-enabled-mainnet` without exposing configuration values.\n\nSmartFetch uses `X402_PAY_TO` as the public receiving address and does not need `CDP_WALLET_SECRET`. Never provide a seller MetaMask private key or recovery phrase: SmartFetch does not require, read, accept, log, or store either one.\n\n## Local validation\n\n```bash\npython tests/security_smoke.py\npython tests/api_local_smoke.py\n```\n\n`api_local_smoke.py` intentionally enables private-network access only inside the test process so it can use a localhost fixture.\n\n## Container\n\n```bash\ndocker build -t smartfetch:v1.10.6 .\ndocker run --rm -p 8787:8787 smartfetch:v1.10.6\n```\n\nThe container installs Chromium automatically.\n\n## Public deployment\n\nThis package is Docker-ready. Railway can deploy the included `Dockerfile` and `railway.json`; any Docker host that provides outbound HTTPS and enough RAM for one Chromium process should also work.\n\nRecommended initial environment variables:\n\n```text\nMAX_CONCURRENT_BROWSERS=1\nMAX_CONCURRENT_FETCHES=4\nDEFAULT_MAX_OUTPUT_CHARS=20000\nMAX_OUTPUT_CHARS=50000\nFETCH_TIMEOUT_SECONDS=12\nBROWSER_TIMEOUT_SECONDS=15\nTOTAL_REQUEST_TIMEOUT_SECONDS=25\nRATE_LIMIT_PER_MINUTE=30\nRATE_LIMIT_BURST=10\nX402_ENABLED=false\n```\n\nDo **not** set `ALLOW_PRIVATE_NETWORK=1` in production.\n\n### Important SSRF deployment note\n\nApplication-level URL validation is included, but browser rendering can execute page subresources. For a paid public deployment, the browser container should additionally be isolated by host/network egress policy from cloud metadata and private RFC1918 networks. Production should retain this defense in depth alongside the application controls and rate limits.\n\n## Test the public deployment\n\nOn Windows run `test_remote_windows.bat`, paste the public URL, and upload `tests/remote_20_results.json`.\n\nOr:\n\n```bash\npython tests/remote_20.py https://YOUR-PUBLIC-URL\n```\n\nOur deployment gate remains **18/20 minimum**, including all five forced-browser requests.\n\n## Production release status\n\nSmartFetch V1.10.5 is live in production. The production MCP server still\nexposes exactly four tools: `fetch_webpage`, `webpage_to_markdown`,\n`extract_webpage_text`, and `render_webpage`. HTTP `POST /fetch` and all four MCP\ntools use x402 `exact` payments on Base mainnet at `$0.005` per execution.\n\nFree `/.well-known/x402` discovery, privacy-safe structured activity logging,\nand the guarded buyer examples are live V1.10 additions. V1.10.5 is the\ncurrently published Official MCP Registry version; V1.10.6 is not published or\ndeployed by this repository change. Base Sepolia remains supported for testnet use, and the\nactive payment network remains controlled by `X402_NETWORK`: `eip155:84532` for\nBase Sepolia or `eip155:8453` for Base mainnet.\n",
  "bytes": 19839,
  "sha": "10974600722359e958cdfc36c6b95694c68f05a2da9bbd4c3ae241ef3cd454ad",
  "repo_slug": "friezaaaa/smartfetch",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_friezaaaa_smartfetch_70a631a0/readme"
}