{
"markdown": "# searchts\n\n\n\n**The missing layer between AI and the web.** A Python CLI and library that lets an AI agent read and search the internet, fronted by a fully open-source \"unlocker\" that gets through common bot-walls with no paid proxy and no API key.\n\n[](https://github.com/capad-xyz/searchts/actions/workflows/pytest.yml)\n[](https://pypi.org/project/searchts/)\n[](https://pypi.org/project/searchts/)\n[](https://pepy.tech/projects/searchts)\n[](https://github.com/capad-xyz/searchts/blob/main/LICENSE)\n\n
\n\n## Why searchts?\n\n- Reads pages behind common bot walls\n- Reads complete ChatGPT / Claude / Gemini / Grok / Poe / DeepSeek / Perplexity / Copilot shared conversations\n- Works with Claude, Codex, and MCP agents\n- Extracts clean Markdown, ready to feed a model\n- Searches the web without API keys\n- Downloads a page's assets (images, fonts, palette)\n- Transcribes videos, subtitles-first\n\n## Why it's free\n\nAI agents constantly need to read web pages, but the naive way they fetch is trivially blocked by modern anti-bot systems (Cloudflare, PerimeterX, DataDome). Paid unlocker services solve this, but the thing they really charge for is a large pool of clean residential IP addresses. `searchts` runs on your own machine, from your own connection, at personal volume, so it sidesteps that cost and gets through most of those walls for free.\n\n## The unlocker\n\n`searchts` reads any URL through an escalating ladder and stops at the first tier that returns real content:\n\n1. **curl_cffi**: a fetch that impersonates a real Chrome's TLS/JA3 and HTTP2 fingerprint. Beats user-agent and fingerprint filters. Fast, local, private.\n2. **Jina Reader**: a JavaScript-rendering relay (`r.jina.ai`), for pages that only fill in content after running JS. **Default on** — the target URL is sent to Jina's servers on this rung. Opt out with `SEARCHTS_NO_JINA=1` or config `jina: false` (local curl + stealth only).\n3. **stealth browser**: an undetected headless Chromium (patchright), launched lazily only when the cheaper tiers fail, for live JS / Cloudflare managed challenges.\n\nIf no tier comes back with real content, an optional human-in-the-loop step opens a real browser so you can clear the page once and continue. That covers interactive CAPTCHAs and soft walls alike: a login page served as HTTP 200 is not a challenge, but it is still a page only a human gets past. Block detection is phrase-based (not vendor-name based), so legitimate pages that merely embed a bot-sensor script are not falsely rejected. Content is extracted to clean Markdown with `trafilatura`.\n\n## AI-chat share links\n\nShare links from AI chat apps are a special kind of hard: the conversation never appears in the page HTML as extractable text, so generic readers (and most AI agents' built-in fetch) return an empty shell or a fragment cut off mid-chat. `searchts read` recognizes these URLs and decodes each provider's own data channel instead, returning the **complete conversation** as role-labeled Markdown — keyless, no login:\n\n| Provider | Share URL | How it's read |\n|----------|-----------|---------------|\n| ChatGPT | `chatgpt.com/share/…`, `chatgpt.com/s/…` | turbo-stream payload embedded in the page |\n| Claude | `claude.ai/share/…` | keyless snapshot API (behind Cloudflare) |\n| Gemini | `gemini.google.com/share/…` | keyless batchexecute RPC |\n| Grok | `grok.com/share/…`, `x.com/i/grok/share/…` | keyless share-links API |\n| Poe | `poe.com/s/…` | `__NEXT_DATA__` payload embedded in the page |\n| DeepSeek | `chat.deepseek.com/share/…` | stealth render, scrolled to the end |\n| Perplexity | `perplexity.ai/search/…`, `perplexity.ai/page/…` | stealth render, scrolled to the end |\n| Copilot | `copilot.microsoft.com/shares/…`, `…/shares/pages/…` | stealth render, scrolled to the end |\n\nThe first five need no browser. The last three are JavaScript shells with\nnothing in the initial HTML, so those reuse the stealth tier: wait for the\nconversation to render, auto-scroll until the page height stops changing (list\nvirtualization will otherwise truncate a long chat), then expand the collapsed\nsections before reading. The benchmark currently covers the five that read\nwithout a browser and passes all five; the three that need one are not in it\nyet.\n\nChatGPT issues two shapes: `/share/` for a whole conversation, and the\nnewer `/s/_` short links for a single shared turn (`t_` thread,\n`m_` message, `dr_` deep research, `cd_` Codex). Both are read.\n\nEach provider is a drop-in plugin module (`searchts/share_extractors/`); if a provider changes its format, extraction falls back to the normal unlocker ladder instead of failing.\n\n## Install\n\nKeep it (global isolated CLI, MCP extra included):\n\n```bash\npipx install \"searchts[mcp]\"\n```\n\nTry it without installing (one-shot, copy-paste):\n\n```bash\nuvx --from \"searchts[mcp]\" searchts \n```\n\nvenv / packaging only (not the recommended path for the CLI):\n\n```bash\npython -m venv .venv\nsource .venv/bin/activate # Windows: .venv\\Scripts\\activate\npip install \"searchts[mcp]\"\n# optional stealth-browser extra, still venv-only:\n# pip install \"searchts[browser]\" && patchright install chromium\n```\n\n## Quickstart\n\n```bash\nsearchts read https://en.wikipedia.org/wiki/Ada_Lovelace # fetch a page as clean Markdown\nsearchts search \"open source vector db\" # multi-provider web search (keyless by default)\nsearchts transcribe https://youtu.be/... # transcript of a YouTube/TikTok/Instagram/Reddit video\nsearchts grab https://example.com # download a page's assets + extract palette/fonts\nsearchts get https://example.com/logo.png # download one asset (image/PDF/font/file)\nsearchts doctor # see what is configured and working\n```\n\n`read` flags: `--json`, `--backend `, `--human` (hand off a CAPTCHA or login wall to a real browser), `--scrub` (redact injection).\n`search` flags: `-n `, `--json`, `--provider `. Content goes to stdout (pipeable); status to stderr.\n`grab` flags: `--out `, `--kinds `, `--read` (also save page.md), `--max `, `--json`.\n\n## Use it from your AI agent\n\nAdd searchts to your agent in one line - as an MCP server, or as a Claude Code slash command:\n\n
\n \n
\n\nTwo ways, both one command:\n\n```bash\n# 1) MCP: always-on read_url + web_search + fetch_asset + grab_site + get_status\n# Try / no install / Claude cannot see PATH:\nclaude mcp add searchts -- uvx --from \"searchts[mcp]\" searchts mcp serve\n# Keep (after pipx install \"searchts[mcp]\"):\n# claude mcp add searchts -- searchts mcp serve\n# Desktop / Cursor JSON: `searchts mcp install` (or uvx the same serve command)\n# First read: Wikipedia — example.com is thinner than _MIN_CHARS and looks like a failed install.\n\n# 2) Slash command: type /searchts in Claude Code\nsearchts skill install # writes ~/.claude/commands/searchts.md\n```\n\nSee the [MCP server reference](https://github.com/capad-xyz/searchts/blob/main/docs/mcp.md) for all five tools (`read_url`, `web_search`, `fetch_asset`, `grab_site`, `get_status`), their inputs and outputs, and when to use each.\n\n## Features\n\n- **Escalating open-source unlocker**: curl_cffi, then Jina Reader, then a stealth browser.\n- **Multi-provider search with rank fusion**: DuckDuckGo (keyless default), plus SearXNG, Exa, Brave, and Tavily when configured; results merged with reciprocal rank fusion and de-duplicated.\n- **Video transcription**: yt-dlp audio plus Whisper for YouTube, TikTok, Instagram, and Reddit videos.\n- **Asset + design grabber**: `searchts grab ` downloads a page's images/icons/css/fonts and extracts a color palette plus the fonts in use; `searchts get ` pulls a single asset. Both go through the same escalating unlock ladder, so they work on fingerprint-gated CDNs, not just open ones.\n- **Prompt-injection scrubbing**: strips invisible/bidi characters, flags injection indicators, optional redaction, so untrusted page content is safer to feed a model.\n- **Per-domain backend memory**: remembers which tier worked per domain and tries it first (`SEARCHTS_NO_MEMORY=1` to disable).\n- **Jina opt-out**: the JS-render relay is on by default; `SEARCHTS_NO_JINA=1` (or `jina: false` in `~/.searchts` config) skips it so URLs never hit `r.jina.ai`.\n- **Surfaces**: a CLI, an MCP server (`read_url`, `web_search`, `fetch_asset`, `grab_site`, `get_status`), and a Python library.\n\n## Use as a library\n\n```python\nfrom searchts import unlocker\nr = unlocker.fetch(\"https://example.com\")\nprint(r.backend, r.status, r.text)\n\nfrom searchts.search import search\nfor hit in search(\"open source vector db\", max_results=5):\n print(hit.title, hit.url)\n```\n\n## Does it actually work?\n\nRather than take our word for it, searchts ships a reproducible **two-suite** benchmark: it runs the unlocker over two page sets and reports how many it read — keyless — and which tier carried each.\n\n- **Smoke** — a small public page set (control, open docs, AI-chat share links). A regression canary, *not* evidence about hard bot-walls.\n- **Walled** — real vendors that restrict bots (Reddit, LinkedIn login wall, a Cloudflare/DataDome-class site, X, Booking). A short body under the unlocker's minimum-content threshold is a fail, not a pass; expected walled failures are reported honestly, not papered over with 100%.\n\nThese two are reported **separately** on purpose — the smoke number is not \"does it work on walls.\" See [benchmarks/README.md](https://github.com/capad-xyz/searchts/blob/main/benchmarks/README.md).\n\n```bash\npython -m benchmarks.run # both suites, print a scorecard\npython -m benchmarks.run --suite walled # the real walled pass rate only\npython -m benchmarks.run --out docs/ # write docs/scorecard.md + results.json\n```\n\nLatest run: [docs/scorecard.md](https://github.com/capad-xyz/searchts/blob/main/docs/scorecard.md). Add your own targets — see [benchmarks/README.md](https://github.com/capad-xyz/searchts/blob/main/benchmarks/README.md).\n\n> The numbers only mean something from a **residential** connection: a datacenter IP (or a VPN that reshapes your TLS fingerprint) blocks the fast curl_cffi tier more than a real user sees.\n\n## How it works, and its limits\n\n- It runs from your own residential IP at personal volume, which is why it needs no paid proxy pool. It is a personal-grade research tool, not a mass-scraping system.\n- Interactive CAPTCHAs (DataDome / Turnstile press-and-hold) and login walls are the honest ceiling. Use `--human` for those.\n- Some platforms (notably Instagram, and YouTube in 2026) may need your browser cookies or fail intermittently; that is platform-side.\n- Anti-bot systems evolve; this is an arms race and the techniques may need occasional updates. Respect each site's terms of service and use responsibly.\n\n## Configuration\n\nSearch works with no keys (DuckDuckGo). Everything else is optional, via `searchts configure` or a `.env` (see `.env.example`):\n\n- **Search providers**: Exa, Brave, Tavily API keys, or a self-hosted `SEARXNG_URL`, for more and better results.\n- **Transcription**: a Groq or OpenAI (Whisper) key, plus `ffmpeg` and `yt-dlp`.\n- **GitHub token** for higher rate limits.\n\nRun `searchts doctor` to check what is configured and working.\n\n## Optional integrations\n\nThe core is `read` / `search` / `transcribe`. Every `searchts read` goes through\n`unlocker.fetch` — there is no per-platform router. `searchts doctor` only probes\nwhether optional CLIs (`gh`, `twitter-cli`, `opencli`, `mcporter`) are on PATH\nand authenticated. Presence is not a claim that searchts reads those sites\nthrough those CLIs.\n\n## Roadmap\n\nSee [ROADMAP.md](https://github.com/capad-xyz/searchts/blob/main/ROADMAP.md) for where searchts is headed — and what's deliberately out of scope.\n\n## Credits\n\n`searchts` builds on and extends [Agent-Reach](https://github.com/Panniantong/Agent-Reach) (MIT), reusing its channel, installer, and diagnostics architecture. The escalating open-source unlocker, multi-provider search with rank fusion, prompt-injection scrubbing, per-domain backend memory, the human-in-the-loop CAPTCHA flow, the video transcript channels, the `read_url` / `web_search` MCP tools, and the `read` / `search` CLI commands are additions in `searchts`. Thanks to the original authors.\n\n## License\n\nMIT. See [LICENSE](https://github.com/capad-xyz/searchts/blob/main/LICENSE). Original portions Copyright (c) 2025 Agent Eyes; modifications and additions Copyright (c) 2026 capad-xyz.\n\n---\n\nBuilt by [capad](https://github.com/capad-xyz). Questions or feedback: open an issue or email oss@capad.fyi.\n\n> *Fun fact: \"searchts\" doesn't officially abbreviate anything. Off the record, it stands for \"search this shit\".*\n",
"bytes": 13706,
"sha": "b522b60ac3cdde2c70a3c12300c46113c50e85f6f31e2bfc4502b7837ff4cda2",
"repo_slug": "capad-xyz/searchts",
"fonte": "repo",
"truncated": false,
"api": "https://agentalog.com/api/listings/mcp_io_github_capad_xyz_searchts_eb58948f/readme"
}
![\"A](\"https://raw.githubusercontent.com/capad-xyz/searchts/main/demo/demo1.gif\")
\n ![\"Installing](\"https://raw.githubusercontent.com/capad-xyz/searchts/main/demo/demo2.gif\")
\n