{
  "markdown": "# Web Search Plus\n\n<p align=\"center\">\n  <img src=\"docs/assets/web-search-plus-logo.png\" alt=\"web search plus logo\" width=\"180\">\n</p>\n\nUnified multi-provider web search and URL extraction for OpenClaw-style agent workflows.\n\nCurrent version: **4.0.0**\n\n> **Status: stable OpenClaw skill — compatibility path, not the main engine-development path.**\n> This OpenClaw skill is kept usable and periodically synced for OpenClaw users, but active engine development happens on **[hermes-web-search-plus](https://github.com/robbyczgw-cla/hermes-web-search-plus)** and the **[web-search-plus-mcp](https://github.com/robbyczgw-cla/web-search-plus-mcp)** server. Version 4.0.0 keeps that CLI compatibility path and makes it **source-only**: Perplexity/Kilo answer synthesis is gone. The native OpenClaw plugin remains `web-search-plus-plugin-v2` (currently 4.0.3).\n\n## ⚠️ Data handling & privacy\n\n- **Search queries and extraction URLs are sent to the configured third-party providers** (Serper, Brave, Tavily, Linkup, Querit, Exa, Firecrawl, SerpBase, Keenable, You.com, or your SearXNG instance). Each provider's privacy policy and retention rules apply to what you send.\n- **For sensitive queries, pick the provider explicitly** with `--provider <name>` so you control which third party receives them; self-hosted SearXNG keeps queries on your own infrastructure.\n- **Avoid submitting internal/private URLs for extraction** — extraction URLs are forwarded to external services. Private/loopback/link-local targets and cloud metadata endpoints are blocked by default (opt out with `--allow-private-urls` / `WSP_ALLOW_PRIVATE_URLS=1` for trusted private networks).\n- **Local caching is on by default**: queries, results, provider failure history (`provider_health.json`), and provider performance samples for adaptive routing (`provider_stats.json`) are persisted under `.cache` (or `WSP_CACHE_DIR`) with owner-only permissions (dir `0700`, files `0600`). Use `--no-cache` per call, `WSP_DISABLE_CACHE=1` globally, `--clear-cache` to wipe, `--cache-stats` to inspect.\n- **API keys are never logged or cached.**\n\n## What changed in 4.0.0\n\n- **Source-only.** Perplexity via Kilo (`PERPLEXITY_API_KEY` / `KILOCODE_API_KEY`) is removed from routing, CLI, config, and docs. Leftover keys and `provider_priority` entries are ignored.\n- This is **not** a copy of plugin 4.0.3: no DonSeTch, Parallel, Octen, or TinyFish in this skill runtime.\n- Ranked source URLs and extracted page text only — no model-written answers.\n\n## What changed in 3.3.0\n\nFeature sync with `web-search-plus-plugin` v3.2.0 (hermes-web-search-plus v2.5.0–v2.9.0), adapted for the skill's CLI runtime:\n\n- **Keenable** search + extraction provider — independent web index, keyed (`KEENABLE_API_KEY`) or keyless via the **opt-in** shared public tier (`WSP_KEENABLE_ALLOW_PUBLIC=1`, ~1000 req/hour, no SLA). Lowest priority everywhere: it never displaces a configured keyed provider.\n- **Unified `--freshness`** (`day`/`week`/`month`/`year`): native date filters where supported, `freshness.applied=false` in metadata otherwise.\n- **News vertical** (`--type news`): Serper serves Google News natively with correct parsing (date, source, thumbnail, position); other providers report `search_type.applied=false`.\n- **Serper extraction** via the `scrape.serper.dev` webpage scraper — last position in the fallback chain, which is now **Tavily-first**: tavily → exa → linkup → firecrawl → you → keenable → serper.\n- **Locale-aware defaults** with lightweight query language detection (`locale.country`/`locale.language` in config, `WSP_LOCALE_COUNTRY`/`WSP_LOCALE_LANGUAGE`, `\"auto\"` language inference, curated location hints — \"mejores restaurantes Madrid\" → `es`).\n- **Spam/mirror result filtering** (strict domain matching, `quality.blocked_domains`/`quality.allowed_domains`), **domain-diversity reranking** (max 2 head slots per domain), `site:`/`--include-domains` bypass, reported via `metadata.result_filter`.\n- **Adaptive provider performance memory**: rolling latency/result/error window (persisted as `provider_stats.json`) feeding bounded ±1.0 routing-score adjustments (`routing.adaptive_adjustments`).\n- **Robustness**: `Retry-After` parsing with a single bounded inline retry for 429s, cooldown ladder decay after 30 minutes, configuration errors no longer trigger cooldowns, JSON decode failures surface as provider errors; transient HTTP codes now include 408/425/500/502/504.\n- **Security**: extraction SSRF guard extended (CGNAT `100.64/10`, IPv4-mapped IPv6); look-alike domains no longer inherit authority boosts; inline base64 images replaced with `[IMAGE: alt]` placeholders; oversized extractions truncate to a head/tail window (`WSP_EXTRACT_CHAR_LIMIT`, default 15000).\n\n## What changed in 3.2.0\n\n- **Research mode** (`--mode research`): up to three providers queried **concurrently** with deterministic result ordering, cross-provider dedup, and top-source extraction for grounding; `--research-time-budget` gates provider launches and extraction (port of hermes-web-search-plus v2.4.0).\n- **Canonical-source intent reranking** + `--quality-report` authority signals (`canonical_domain_hits`, `demoted_domain_hits`, `canonical_top_result`) for official-release/docs/policy/finance/security queries (port of hermes v2.3.0).\n- **ProviderSpec registry** (`scripts/provider_registry.py`) as the single source of truth for provider metadata.\n- **Reliability**: retry backoff jitter (`RETRY_JITTER_FRACTION`), locked + atomic provider-health writes (port of hermes v2.4.0).\n- **Security hardening** (SkillSpector findings): narrowly scoped skill triggers, SSRF guard for user-supplied URLs, owner-only cache permissions, `WSP_DISABLE_CACHE`, declared skill permissions, and the privacy disclosures above.\n\n## What changed in 3.1.0\n\n- Add **SerpBase** as the 11th search provider — low-cost Google SERP API with prepaid credits (https://serpbase.dev)\n- Explicit/fallback-only by design: NOT in default auto-routing priority. Use `--provider serpbase` or opt-in via `config.json`.\n- Closes [#4](https://github.com/robbyczgw-cla/web-search-plus/issues/4); brings skill to parity with `web-search-plus-plugin` v3.0.0\n\n## What changed in 3.0.x\n\n- Add **Brave**, **Linkup**, and **Firecrawl** search providers to the Python skill\n- Add **URL extraction** via `scripts/extract.py` with auto fallback across Firecrawl, Linkup, Tavily, Exa, and You.com\n- Align routing/fallback behavior and docs more closely with the OpenClaw plugin and Hermes port\n- Keep existing **Exa deep** / **deep-reasoning**, cooldown, retry, cache, and SearXNG protections\n\n## Search providers\n\n- **Serper** — shopping, local, broad Google-style web results\n- **Brave** — independent general web index, good broad fallback\n- **Tavily** — research and explanation queries\n- **Querit** — multilingual and international AI search\n- **Linkup** — citation/source-grounded search\n- **Exa** — semantic discovery with source URLs\n- **Firecrawl** — search with scrape-ready metadata\n- **You.com** — current-web / RAG-ish queries\n- **SearXNG** — privacy-first self-hosted metasearch\n- **SerpBase** — low-cost Google SERP, prepaid credits, **explicit/fallback-only**\n- **Keenable** — independent web index, keyed or opt-in keyless public tier, lowest priority\n\n## Extraction providers\n\n`scripts/extract.py` supports (auto order, Tavily-first):\n\n- Tavily\n- Exa\n- Linkup\n- Firecrawl\n- You.com\n- Keenable\n- Serper (webpage scraper)\n\nAuto extraction tries them in that order and falls back when a provider is unconfigured or fails. Oversized pages are truncated to a head/tail window (default 15,000 chars, `WSP_EXTRACT_CHAR_LIMIT`); inline base64 images become `[IMAGE: alt]` placeholders.\n\n## Quick start\n\n```bash\ncp .env.example .env\n# fill in at least one key or SEARXNG_INSTANCE_URL\n\npython3 scripts/search.py -q \"latest OpenClaw release\"\npython3 scripts/extract.py --url https://example.com\n```\n\nOr use the interactive wizard:\n\n```bash\npython3 scripts/setup.py\n```\n\n## Search examples\n\n```bash\npython3 scripts/search.py -q \"weather in Vienna today\"\n# auto-routes to Brave or Serper for broad current-web intent\n\npython3 scripts/search.py -q \"find credible sources for AI tutoring outcomes\"\n# auto-routes to Linkup when available\n\npython3 scripts/search.py -q \"latest AI policy updates in Germany\"\n# often Querit / Tavily depending on configured providers\n\npython3 scripts/search.py -p exa --exa-depth deep -q \"LLM scaling laws research\"\npython3 scripts/search.py -p firecrawl -q \"YC startups web scraping\"\npython3 scripts/search.py -p serpbase -q \"best laptop 2026\"\n# explicit SerpBase call — not used by auto-routing unless added to provider_priority\n```\n\n## Extraction examples\n\n```bash\npython3 scripts/extract.py --url https://example.com\npython3 scripts/extract.py --url https://docs.linkup.so --provider linkup\npython3 scripts/extract.py --url https://example.com --url https://example.org --include-images\n```\n\n## Research mode & quality reports\n\n```bash\n# Concurrent multi-provider search + dedup + top-source extraction\npython3 scripts/search.py --mode research -q \"EU AI Act obligations for foundation models\"\npython3 scripts/search.py --mode research -q \"...\" --research-providers tavily linkup exa --research-time-budget 30\n\n# Transparent routing/result diagnostics with authority signals\npython3 scripts/search.py -q \"official anthropic claude release notes\" --quality-report\n```\n\nResearch providers run concurrently (wall-clock ≈ slowest provider); result ordering stays deterministic by submission order. For canonical-source query classes (vendor releases, official docs, policy PDFs, finance/IR, security advisories) results are reranked so primary sources beat mirrors — see `metadata.intent_rerank` and `quality_report.authority_signals`.\n\n## Caching\n\nResults are cached under `.cache` (override with `WSP_CACHE_DIR`) for 1 hour by default; provider failure history lives in `.cache/provider_health.json` and adaptive-routing performance samples in `.cache/provider_stats.json`. The directory is created `0700` and files `0600`.\n\n```bash\npython3 scripts/search.py -q \"...\" --no-cache    # bypass for one call\nWSP_DISABLE_CACHE=1 python3 scripts/search.py -q \"...\"   # disable globally\npython3 scripts/search.py --clear-cache          # wipe cached results\npython3 scripts/search.py --cache-stats          # inspect\n```\n\n## Routing notes\n\nProvider priority now defaults to:\n\n```text\ntavily -> linkup -> querit -> exa -> firecrawl -> brave -> serper -> you -> searxng\n```\n\nNotable behavior:\n\n- Brave and Serper share generic web/current-info intent and use deterministic tie-breaking\n- Linkup gets explicit boosts for citation/source/evidence-style queries\n- Firecrawl can win discovery/research-ish queries when configured\n- Exa can auto-upgrade to `deep` or `deep-reasoning` based on query signals\n- Failing or cooling-down providers are skipped by fallback routing\n\n## Config files\n\n- `.env.example` — provider credentials template\n- `config.example.json` — routing and provider settings template\n- `config.json` — your live local config (created/edited locally)\n\n## Verification\n\nSuggested local checks:\n\n```bash\npython3 -m unittest discover -s tests -p 'test_*.py'\npython3 scripts/search.py --explain-routing -q \"find credible sources for climate change impacts\"\npython3 scripts/extract.py --url https://example.com --provider auto --compact\n```\n\n## Related references\n\n- OpenClaw plugin: `../projects/web-search-plus-plugin`\n- Hermes port: `../projects/hermes-web-search-plus`\n",
  "bytes": 11526,
  "sha": "1940fda552a259bcd3bc21f56be22003a1ee057dcc2201588bc3c28706a47360",
  "repo_slug": "robbyczgw-cla/web-search-plus",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/skl_robbyczgw_cla_web_search_plus_web_search_72a261ba/readme"
}