web-search-plus
robbyczgw-cla/web-search-plus · skills.sh
Open source Repository Open in the app JSON README (API)
About
Skill publicada por robbyczgw-cla/web-search-plus no skills.sh. Instale com: npx skills add robbyczgw-cla/web-search-plus@web-search-plus
Details
- Kind
- Agent skills
- Topic
- No topic detected
- Publisher
- robbyczgw-cla
- Origin
- skillssh
- Category
- ferramentas
- Stars
- 26
- Forks
- 4
- Open pull requests
- 1
- Last push
- 2026-08-31T10:17:33Z
- Repository state
- ativo
- Language
- Python
- License
- MIT
- Added
- 2026-08-30 15:22:39
- Updated
- 2026-09-08 15:05:58
- Origin id
robbyczgw-cla/web-search-plus/web-search-plus
README
# Web Search Plus <p align="center"> <img src="docs/assets/web-search-plus-logo.png" alt="web search plus logo" width="180"> </p> Unified multi-provider web search and URL extraction for OpenClaw-style agent workflows. Current version: **4.0.0** > **Status: stable OpenClaw skill — compatibility path, not the main engine-development path.** > This OpenClaw skill is kept usable and periodically synced for OpenClaw users, but active engine development happens on **[hermes-web-search-plus](https://github.com/robbyczgw-cla/hermes-web-search-plus)** and the **[web-search-plus-mcp](https://github.com/robbyczgw-cla/web-search-plus-mcp)** server. Version 4.0.0 keeps that CLI compatibility path and makes it **source-only**: Perplexity/Kilo answer synthesis is gone. The native OpenClaw plugin remains `web-search-plus-plugin-v2` (currently 4.0.3). ## ⚠️ Data handling & privacy - **Search queries and extraction URLs are sent to the configured third-party providers** (Serper, Brave, Tavily, Linkup, Querit, Exa, Firecrawl, SerpBase, Keenable, You.com, or your SearXNG instance). Each provider's privacy policy and retention rules apply to what you send. - **For sensitive queries, pick the provider explicitly** with `--provider <name>` so you control which third party receives them; self-hosted SearXNG keeps queries on your own infrastructure. - **Avoid submitting internal/private URLs for extraction** — extraction URLs are forwarded to external services. Private/loopback/link-local targets and cloud metadata endpoints are blocked by default (opt out with `--allow-private-urls` / `WSP_ALLOW_PRIVATE_URLS=1` for trusted private networks). - **Local caching is on by default**: queries, results, provider failure history (`provider_health.json`), and provider performance samples for adaptive routing (`provider_stats.json`) are persisted under `.cache` (or `WSP_CACHE_DIR`) with owner-only permissions (dir `0700`, files `0600`). Use `--no-cache` per call, `WSP_DISABLE_CACHE=1` globally, `--clear-cache` to wipe, `--cache-stats` to inspect. - **API keys are never logged or cached.** ## What changed in 4.0.0 - **Source-only.** Perplexity via Kilo (`PERPLEXITY_API_KEY` / `KILOCODE_API_KEY`) is removed from routing, CLI, config, and docs. Leftover keys and `provider_priority` entries are ignored. - This is **not** a copy of plugin 4.0.3: no DonSeTch, Parallel, Octen, or TinyFish in this skill runtime. - Ranked source URLs and extracted page text only — no model-written answers. ## What changed in 3.3.0 Feature sync with `web-search-plus-plugin` v3.2.0 (hermes-web-search-plus v2.5.0–v2.9.0), adapted for the skill's CLI runtime: - **Keenable** search + extraction provider — independent web index, keyed (`KEENABLE_API_KEY`) or keyless via the **opt-in** shared public tier (`WSP_KEENABLE_ALLOW_PUBLIC=1`, ~1000 req/hour, no SLA). Lowest priority everywhere: it never displaces a configured keyed provider. - **Unified `--freshness`** (`day`/`week`/`month`/`year`): native date filters where supported, `freshness.applied=false` in metadata otherwise. - **News vertical** (`--type news`): Serper serves Google News natively with correct parsing (date, source, thumbnail, position); other providers report `search_type.applied=false`. - **Serper extraction** via the `scrape.serper.dev` webpage scraper — last position in the fallback chain, which is now **Tavily-first**: tavily → exa → linkup → firecrawl → you → keenable → serper. - **Locale-aware defaults** with lightweight query language detection (`locale.country`/`locale.language` in config, `WSP_LOCALE_COUNTRY`/`WSP_LOCALE_LANGUAGE`, `"auto"` language inference, curated location hints — "mejores restaurantes Madrid" → `es`). - **Spam/mirror result filtering** (strict domain matching, `quality.blocked_domains`/`quality.allowed_domains`), **domain-diversity reranking** (max 2 head slots per domain), `site:`/`--include-domains` bypass, reported via `metadata.result_filter`. - **Adaptive provider performance memory**: rolling latency/result/error window (persisted as `provider_stats.json`) feeding bounded ±1.0 routing-score adjustments (`routing.adaptive_adjustments`). - **Robustness**: `Retry-After` parsing with a single bounded inline retry for 429s, cooldown ladder decay after 30 minutes, configuration errors no longer trigger cooldowns, JSON decode failures surface as provider errors; transient HTTP codes now include 408/425/500/502/504. - **Security**: extraction SSRF guard extended (CGNAT `100.64/10`, IPv4-mapped IPv6); look-alike domains no longer inherit authority boosts; inline base64 images replaced with `[IMAGE: alt]` placeholders; oversized extractions truncate to a head/tail window (`WSP_EXTRACT_CHAR_LIMIT`, default 15000). ## What changed in 3.2.0 - **Research mode** (`--mode research`): up to three providers queried **concurrently** with deterministic result ordering, cross-provider dedup, and top-source extraction for grounding; `--research-time-budget` gates provider launches and extraction (port of hermes-web-search-plus v2.4.0). - **Canonical-source intent reranking** + `--quality-report` authority signals (`canonical_domain_hits`, `demoted_domain_hits`, `canonical_top_result`) for official-release/docs/policy/finance/security queries (port of hermes v2.3.0). - **ProviderSpec registry** (`scripts/provider_registry.py`) as the single source of truth for provider metadata. - **Reliability**: retry backoff jitter (`RETRY_JITTER_FRACTION`), locked + atomic provider-health writes (port of hermes v2.4.0). - **Security hardening** (SkillSpector findings): narrowly scoped skill triggers, SSRF guard for user-supplied URLs, owner-only cache permissions, `WSP_DISABLE_CACHE`, declared skill permissions, and the privacy disclosures above. ## What changed in 3.1.0 - Add **SerpBase** as the 11th search provider — low-cost Google SERP API with prepaid credits (https://serpbase.dev) - Explicit/fallback-only by design: NOT in default auto-routing priority. Use `--provider serpbase` or opt-in via `config.json`. - Closes [#4](https://github.com/robbyczgw-cla/web-search-plus/issues/4); brings skill to parity with `web-search-plus-plugin` v3.0.0 ## What changed in 3.0.x - Add **Brave**, **Linkup**, and **Firecrawl** search providers to the Python skill - Add **URL extraction** via `scripts/extract.py` with auto fallback across Firecrawl, Linkup, Tavily, Exa, and You.com - Align routing/fallback behavior and docs more closely with the OpenClaw plugin and Hermes port - Keep existing **Exa deep** / **deep-reasoning**, cooldown, retry, cache, and SearXNG protections ## Search providers - **Serper** — shopping, local, broad Google-style web results - **Brave** — independent general web index, good broad fallback - **Tavily** — research and explanation queries - **Querit** — multilingual and international AI search - **Linkup** — citation/source-grounded search - **Exa** — semantic discovery with source URLs - **Firecrawl** — search with scrape-ready metadata - **You.com** — current-web / RAG-ish queries - **SearXNG** — privacy-first self-hosted metasearch - **SerpBase** — low-cost Google SERP, prepaid credits, **explicit/fallback-only** - **Keenable** — independent web index, keyed or opt-in keyless public tier, lowest priority ## Extraction providers `scripts/extract.py` supports (auto order, Tavily-first): - Tavily - Exa - Linkup - Firecrawl - You.com - Keenable - Serper (webpage scraper) Auto extraction tries them in that order and falls back when a provider is unconfigured or fails. Oversized pages are truncated to a head/tail window (default 15,000 chars, `WSP_EXTRACT_CHAR_LIMIT`); inline base64 images become `[IMAGE: alt]` placeholders. ## Quick start ```bash cp .env.example .env # fill in at least one key or SEARXNG_INSTANCE_URL python3 scripts/search.py -q "latest OpenClaw release" python3 scripts/extract.py --url https://example.com ``` Or use the interactive wizard: ```bash python3 scripts/setup.py ``` ## Search examples ```bash python3 scripts/search.py -q "weather in Vienna today" # auto-routes to Brave or Serper for broad current-web intent python3 scripts/search.py -q "find credible sources for AI tutoring outcomes" # auto-routes to Linkup when available python3 scripts/search.py -q "latest AI policy updates in Germany" # often Querit / Tavily depending on configured providers python3 scripts/search.py -p exa --exa-depth deep -q "LLM scaling laws research" python3 scripts/search.py -p firecrawl -q "YC startups web scraping" python3 scripts/search.py -p serpbase -q "best laptop 2026" # explicit SerpBase call — not used by auto-routing unless added to provider_priority ``` ## Extraction examples ```bash python3 scripts/extract.py --url https://example.com python3 scripts/extract.py --url https://docs.linkup.so --provider linkup python3 scripts/extract.py --url https://example.com --url https://example.org --include-images ``` ## Research mode & quality reports ```bash # Concurrent multi-provider search + dedup + top-source extraction python3 scripts/search.py --mode research -q "EU AI Act obligations for foundation models" python3 scripts/search.py --mode research -q "..." --research-providers tavily linkup exa --research-time-budget 30 # Transparent routing/result diagnostics with authority signals python3 scripts/search.py -q "official anthropic claude release notes" --quality-report ``` Research providers run concurrently (wall-clock ≈ slowest provider); result ordering stays deterministic by submission order. For canonical-source query classes (vendor releases, official docs, policy PDFs, finance/IR, security advisories) results are reranked so primary sources beat mirrors — see `metadata.intent_rerank` and `quality_report.authority_signals`. ## Caching Results are cached under `.cache` (override with `WSP_CACHE_DIR`) for 1 hour by default; provider failure history lives in `.cache/provider_health.json` and adaptive-routing performance samples in `.cache/provider_stats.json`. The directory is created `0700` and files `0600`. ```bash python3 scripts/search.py -q "..." --no-cache # bypass for one call WSP_DISABLE_CACHE=1 python3 scripts/search.py -q "..." # disable globally python3 scripts/search.py --clear-cache # wipe cached results python3 scripts/search.py --cache-stats # inspect ``` ## Routing notes Provider priority now defaults to: ```text tavily -> linkup -> querit -> exa -> firecrawl -> brave -> serper -> you -> searxng ``` Notable behavior: - Brave and Serper share generic web/current-info intent and use deterministic tie-breaking - Linkup gets explicit boosts for citation/source/evidence-style queries - Firecrawl can win discovery/research-ish queries when configured - Exa can auto-upgrade to `deep` or `deep-reasoning` based on query signals - Failing or cooling-down providers are skipped by fallback routing ## Config files - `.env.example` — provider credentials template - `config.example.json` — routing and provider settings template - `config.json` — your live local config (created/edited locally) ## Verification Suggested local checks: ```bash python3 -m unittest discover -s tests -p 'test_*.py' python3 scripts/search.py --explain-routing -q "find credible sources for climate change impacts" python3 scripts/extract.py --url https://example.com --provider auto --compact ``` ## Related references - OpenClaw plugin: `../projects/web-search-plus-plugin` - Hermes port: `../projects/hermes-web-search-plus`