webfetch
License-first federated image search with attribution, dimensions, and download tooling.
Open source Open in the app JSON README (API)
About
License-first federated image search with attribution, dimensions, and download tooling.
Details
- Kind
- MCP servers
- Topic
- Marketing & analytics
- Publisher
- ashlrai
- Origin
- official
- Category
- ferramentas
- Transport
- local
- Version
- 0.1.5
- Stars
- 3
- Last push
- 2026-07-10T03:28:25Z
- Repository state
- ativo
- Language
- JavaScript
- License
- MIT
- Added
- 2026-08-29 03:02:26
- Updated
- 2026-08-29 03:02:26
- Origin id
io.github.ashlrai/webfetch
README
# webfetch
[](https://www.npmjs.com/package/getwebfetch)
[](https://github.com/ashlrai/webfetch/actions/workflows/ci.yml)
[](./LICENSE)
[](https://glama.ai/mcp/servers/ashlrai/webfetch)
[](https://getwebfetch.com/discord)
[](https://github.com/ashlrai/webfetch)
**The license-first image layer for AI agents and humans.**
One MCP server, one CLI, and one HTTP server that federate across 25 image
providers, rank results license-first, and reject `UNKNOWN` results by default.
Any agent that speaks MCP (Claude Code, Cursor, Cline,
Continue, Roo Code, Codex) wires up from one config line. Landing page,
pricing, and hosted usage live at **[getwebfetch.com](https://getwebfetch.com)**.
## Install
| Surface | One-liner |
| ------------- | --------- |
| npm | `npm i -g getwebfetch` |
| Homebrew | `brew tap ashlrai/webfetch && brew install webfetch` |
| Docker | `docker run --rm ghcr.io/ashlrai/webfetch cli help` |
| curl \| bash | `curl -fsSL https://raw.githubusercontent.com/ashlrai/webfetch/main/install/install.sh \| bash` |
The `curl | bash` installer also wires webfetch into Claude Code's
`~/.claude/settings.json` idempotently. Re-run any time to update.
## Surfaces
| Surface | Best for | Entry point |
| ------- | -------- | ----------- |
| CLI | scripts, shell work, agent handoff | `webfetch search ...` |
| MCP server | Claude Code, Cursor, Cline, Continue, Roo Code, Codex | `npx -y getwebfetch-mcp` |
| HTTP server | local integrations and extensions | `npx -y webfetch-server` |
| Core library | TypeScript apps and custom tooling | `npm i webfetch-core` |
| Browser layer | fallback extraction and managed-browser flows | `npm i webfetch-browser` |
| Hosted cloud | pooled keys, usage tracking, team controls | [app.getwebfetch.com](https://app.getwebfetch.com) |
Package-level API notes live in [`packages/core/README.md`](./packages/core/README.md),
[`packages/browser/README.md`](./packages/browser/README.md), and the other
package READMEs under `packages/`.
## 30-second usage
CLI:
```bash
webfetch search "drake portrait" --limit 5
webfetch artist "Taylor Swift" --kind portrait --min-width 1200
webfetch download <url> --out ./portrait.jpg
printf "drake portrait\nradiohead album\n" | webfetch batch --jsonl --continue-on-error
```
MCP (from inside any MCP-speaking agent):
```
search_images({ query: "drake portrait", limit: 5 })
search_artist_images({ artist: "Taylor Swift", kind: "portrait" })
download_image({ url: "..." })
```
TypeScript library:
```ts
import { searchArtistImages, pickBest, downloadImage } from "webfetch-core";
const { candidates } = await searchArtistImages("Drake", "portrait");
const best = pickBest(candidates, { minWidth: 1200 });
if (best) {
const { cachedPath, sha256 } = await downloadImage(best.url);
console.log(best.attributionLine, "->", cachedPath);
}
```
## What problem this solves
Manually sourcing an image has four failure modes:
1. **You don't know the license**, so you can't safely ship the result.
2. **You can't script it** — every new site means another afternoon.
3. **Google's Image Search API is retired**; scraping is brittle and ToS-grey.
4. **No shared cache** — you re-download the same file dozens of times.
webfetch fixes all four by federating across direct-source APIs that have
stable terms and structured license metadata, ranking candidates
license-first, and exposing the result as a single MCP tool.
## Providers
| Provider | Covers | License default | Auth | Opt-in |
| ---------------- | ---------------------------------------- | -------------------- | ---------------------------- | ------ |
| wikimedia | portraits, events, logos, history | CC_BY_SA (metadata) | — | no |
| openverse | any CC-licensed content | CC_BY (metadata) | — | no |
| unsplash | high-quality photography | `UNSPLASH_LICENSE` | `UNSPLASH_ACCESS_KEY` | no |
| pexels | stock photography | `PEXELS_LICENSE` | `PEXELS_API_KEY` | no |
| pixabay | stock photos + illustrations | `PIXABAY_LICENSE` | `PIXABAY_API_KEY` | no |
| itunes | album covers, artist portraits | EDITORIAL_LICENSED | — | no |
| musicbrainz-caa | canonical album art | EDITORIAL_LICENSED | — | no |
| spotify | artist + album images | EDITORIAL_LICENSED | `SPOTIFY_CLIENT_ID/SECRET` | no |
| youtube-thumb | video thumbnails | EDITORIAL_LICENSED | — | yes |
| brave | general web image search | UNKNOWN (+heuristic) | `BRAVE_API_KEY` | no |
| bing | general web image search | UNKNOWN (+heuristic) | `BING_API_KEY` | yes |
| serpapi | Google Images + reverse lookup | UNKNOWN (+heuristic) | `SERPAPI_KEY` | yes |
| browser | headless fallback vs images.google.com | UNKNOWN | — | yes |
| managed-browser | Bright Data managed browser fallback | UNKNOWN | `BRIGHTDATA_API_TOKEN` | yes |
| flickr | CC / public-domain photography | CC_BY (metadata) | `FLICKR_API_KEY` | no |
| internet-archive | public-domain / CC archive media | PUBLIC_DOMAIN | — | no |
| smithsonian | Open Access museum media | CC0 | `SMITHSONIAN_API_KEY` | no |
| nasa | NASA imagery | PUBLIC_DOMAIN | — | no |
| met-museum | The Met Open Access | CC0 | — | no |
| europeana | European cultural heritage | CC_BY (metadata) | `EUROPEANA_API_KEY` | no |
| library-of-congress | US historical archive | PUBLIC_DOMAIN | — | no |
| wellcome-collection | medical/historical imagery | CC_BY (metadata) | — | no |
| rawpixel | CC0 stock slice | CC0 | `RAWPIXEL_API_KEY` optional | no |
| burst | Shopify Burst stock photos | CC0 | — | no |
| europeana-archival | Europeana text/manuscript records | CC_BY (metadata) | `EUROPEANA_API_KEY` | yes |
See [`docs/PROVIDERS.md`](./docs/PROVIDERS.md) for gotchas, rate limits, and
[`docs/PROVIDER_TUNING.md`](./docs/PROVIDER_TUNING.md) for per-use-case picks.
## Local and cloud modes
The CLI is local-first: by default `webfetch search`, `artist`, `album`,
`download`, `probe`, `license`, and `batch` call `webfetch-core` in-process
and use provider API keys from your environment. Pass `--cloud` or set
`WEBFETCH_MODE=cloud` to call `https://api.getwebfetch.com/v1/*` with
`WEBFETCH_API_KEY` or `webfetch config set apiKey wf_live_...`.
Use local mode when you want direct provider calls and a local cache. Use cloud
mode when you want hosted auth, pooled provider keys, managed browser fallback,
usage accounting, or team controls.
## Why license-first
The only outcome we reject by default is an image we can't justify. A
marginally-better photo under an unknown license is worthless to a pipeline
that needs to ship without human review. Relevance ties are easy to break;
provenance is not.
The ranker sorts by: **license tag -> metadata confidence -> resolution ->
provider priority**. `UNKNOWN` is rejected by default (Berne Convention:
most of the web is all-rights-reserved unless proven otherwise). See
[`docs/LICENSE_POLICY.md`](./docs/LICENSE_POLICY.md).
### Migration: CC0 stock providers
Older webfetch builds treated Unsplash, Pexels, and Pixabay as `CC0`. Current
builds expose their platform terms explicitly:
| Old tag | New tag | What to check |
| --- | --- | --- |
| `CC0` from Unsplash | `UNSPLASH_LICENSE` | Unsplash terms; not Creative Commons |
| `CC0` from Pexels | `PEXELS_LICENSE` | Pexels terms; not Creative Commons |
| `CC0` from Pixabay | `PIXABAY_LICENSE` | Pixabay terms; not Creative Commons |
Most callers should keep `licensePolicy: "safe-only"` because it still allows
open, platform, editorial, and press-kit categories while rejecting `UNKNOWN`.
Pipelines that require only Creative Commons or public-domain assets should use
`licensePolicy: "open-only"` and update type guards to handle the three
platform tags separately.
## webfetch vs alternatives
| Capability | webfetch | Raw Google Images | Unsplash-only | Bing CSE |
| ------------------------------------- | -------- | ----------------- | ------------- | -------- |
| Scriptable via API | yes | no (retired) | yes | yes |
| License metadata per result | yes | no | yes (one lic) | partial |
| Covers editorial music art | yes | partial | no | partial |
| Covers CC / public-domain | yes | no | no | no |
| Safe-by-default (rejects UNKNOWN) | yes | n/a | n/a | no |
| Shared content-addressed cache | yes | no | no | no |
| Attribution line pre-built | yes | no | no | no |
| One MCP config line across all IDEs | yes | no | no | no |
| No per-query cost on defaults | yes | n/a | yes | no |
## Architecture
```
+------------------+
| webfetch-core |
| (ranker, cache, |
| license coerce)|
+---------+--------+
|
+----------------+-----------+-----------+----------------+
| | | |
+-------v------+ +------v-------+ +-------v------+ +------v-------+
| webfetch | | webfetch-mcp | | webfetch- | | browser |
| CLI | | (stdio) | | server (HTTP)| | extensions |
+-------+------+ +------+-------+ +-------+------+ +------+-------+
| | | |
| | | |
+----------------+-----------+-----------+----------------+
|
+---------------------v---------------------+
| provider adapters |
| wikimedia openverse unsplash pexels |
| pixabay itunes mb-caa spotify |
| youtube brave bing serpapi |
| flickr nasa met europeana |
| loc wellcome rawpixel burst |
| browser + managed-browser + archival opt-in|
+-------------------------------------------+
```
Every surface shares `~/.webfetch/cache/` keyed by SHA-256, so a download
from the CLI is instantly available to the MCP server and vice versa.
## Safety defaults
- `licensePolicy: "safe-only"` — open, platform-license, and editorial/press categories are allowed; `UNKNOWN` is rejected.
- `safeSearch: "strict"`.
- Opt-in providers (`youtube-thumb`, `bing`, `serpapi`, `browser`, `managed-browser`, `europeana-archival`) off by default.
- 20 MB per-download cap, content-type guard, host blocklist.
- `robots.txt` respected on generic page probes.
## Roadmap
- `webfetch watch` — daemon mode for repeated queries / incremental refresh.
- Bring-your-own-provider plugin API.
- Hosted tier at [getwebfetch.com](https://getwebfetch.com) — pooled provider keys, managed browser fallback, team usage dashboard.
## Contributing
Issues and PRs welcome. Run `bun install && bun test` to get started. See
[`docs/`](./docs/) for per-area reference docs.
## License
MIT.