{
  "markdown": "# ossfind — safety-ranked open-source component discovery\n\nGiven a query like *\"http client\"*, ossfind returns open-source components **ranked by whether you\ncan actually ship a product on them** — a blended, explainable score of **fit · license · security ·\nmaintenance health · integration effort** — served through **both a web UI and an MCP tool** over one\nranking engine.\n\nIts core promise: **never recommend (\"ship\") a component whose safety evidence is unsafe, missing, or\nambiguous.** The engine fails *closed*.\n\n**New here?** → [`GETTING_STARTED.md`](GETTING_STARTED.md) — install, try the offline demo, go live,\nand connect it to an AI agent over MCP (Claude Code / Claude Desktop / Cursor config included), in\nabout five minutes.\n\n## Command line\n\n```bash\nnpm install -g ossfind\n\nossfind search \"markdown parser\"\nossfind search \"http client\" -e cargo -n 5\nossfind search \"web framework\" -e pypi --json     # machine-readable\nossfind inspect marked                            # verified exports + import line\n```\n\n```\n 1. marked  SHIP     92/100\n    MIT  ·  0 CVEs  ·  OpenSSF 7.4\n```\n\n`-e/--ecosystem` npm · pypi · github · huggingface · cargo · rubygems · all — `-l/--license`\ndeclares your project's license so incompatible results are ranked `AVOID` — `--json` for\nscripting — `--no-color` (also honours `NO_COLOR`). Colour is disabled automatically when\nstdout is not a terminal.\n\n## Quick start (from source)\n\n```bash\nnpm install\nnpm run typecheck && npm test     # 586 tests, fully offline\nnpm run gates                     # 17 safety gates, each proven able to fail\nnpm run eval                      # relevance against the labelled query set (live)\n```\n\nRun the web app (offline demo mode, uses frozen fixtures):\n\n```bash\nOSSFIND_FIXTURES=1 npm run web    # http://127.0.0.1:8787\n```\n\nBy default, the web server binds exclusively to loopback (`127.0.0.1`).\n- `HOST` — bind host (default `127.0.0.1`). Non-loopback hosts (e.g. `HOST=0.0.0.0`) require `OSSFIND_WEB_TOKEN` to be set; starting wide-open without a token is refused.\n- `PORT` — server port (default `8787`).\n- `OSSFIND_WEB_TOKEN` — optional Bearer token requiring `Authorization: Bearer <token>` on `/api/*` endpoints.\n\nRun the MCP server (for AI agents):\n\n```bash\nOSSFIND_FIXTURES=1 npm run mcp    # stdio MCP server exposing `search_components`\n```\n\nDrop `OSSFIND_FIXTURES=1` to hit live suppliers (npm registry, ecosyste.ms, deps.dev, OSV).\n\n## Ecosystems (npm · PyPI · crates.io · RubyGems · GitHub · Hugging Face)\n\nossfind searches **npm** (default), **PyPI**, **crates.io** (Rust), **RubyGems**, **GitHub** repositories, **Hugging Face** models, or\n**all six at once** (`ecosystem: \"all\"`) — one query, results from every ecosystem merged and\nsafety-ranked together, so you don't have to guess where the answer lives (e.g. \"video generation\" →\nPyPI's `decord`, a GitHub AI-model repo, and a Hugging Face model in the same result set). Pick the\necosystem with the web/MCP selector, the `ecosystem` MCP tool argument, or `&ecosystem=all` on\n`/api/search`.\n\nDiscovery is **federated**: a `FederatedDiscoverer` composes multiple source adapters per query\n(parallel, per-source error isolation + timeouts, results merged and deduped by id). Enrichment routes\neach candidate by its own id prefix (`npm:`/`pypi:`/`cargo:`/`rubygems:`/`github:`/`huggingface:`), so a mixed batch is\nenriched correctly per-source. The safety-ranking layer is the same for every source — ossfind owns\nthe ranking, not the corpus. GitHub and Hugging Face are what surface AI-model repos/models (diffusers,\nCogVideo, …) that aren't on any package registry.\n\n- **npm** needs no key — discovery uses the npm registry search API, with query expansion\n  (progressively shorter slices of the query, unioned) to recover the recall a conjunctive text\n  match loses. Optionally federate it with a local semantic index to bridge vocabulary the\n  registry cannot — `marked` says *parser* when you asked for a *renderer*:\n  ```\n  INDEX_MAX=8000 INDEX_DB_PATH=.cache/index/npm.db npm run index:build npm\n  ```\n  The same optional index federates crates.io, RubyGems and PyPI. When an index has not been\n  built, that ecosystem's search behaves exactly as before.\n\n  **It does not help everywhere, and the eval says where.** Measured on the labelled set:\n  npm and crates.io improve substantially (crates.io MRR 0.000 → 0.675, since crates.io's own\n  search ranks by name similarity and never returns `serde` for \"serialization\"). RubyGems is\n  neutral on MRR and slightly positive on recall. Rebuilding PyPI's index concentrated on the\n  top 8,000 packages measured slightly *worse* (0.611 → 0.597) than the broader 25,000-package\n  index, so the wider corpus stays — a hypothesis the harness rejected.\n\n  A RubyGems index originally measured much worse (MRR 0.500 → 0.250) by pushing `rails` out of\n  the enrichment shortlist. That was a shortlisting defect, not an index one, and is fixed: a\n  complete lexical match now counts as relevance evidence, so an adopted package whose\n  description contains every query word earns a slot regardless of its embedding score.\n  Measured on the labelled set, adding the index moved MRR 0.561 → 0.636, hit@3 60.5% → 67.4%\n  and noise@3 2.6% → 0.0%, with no per-query regressions — and made `marked` the top result for\n  *\"markdown to html renderer\"*, which no lexical probe can reach. Note that `npm run eval`\n  therefore depends on a locally built index; without one the numbers are the registry-only ones.\n- **GitHub** uses the repo search API. Set an optional `GITHUB_TOKEN` in `.env.local` for higher rate\n  limits.\n- **Hugging Face** needs no key — discovery uses the public models search API.\n- **crates.io** (Rust) and **RubyGems** need no key — discovery uses their public search APIs,\n  with licence/vulnerability/health enrichment from ecosyste.ms, OSV and deps.dev like any package\n  ecosystem. crates.io ranks by name similarity, so \"serialization\" never returns `serde` from the\n  registry alone; federate a local index (`INDEX_MAX=6000 INDEX_DB_PATH=.cache/index/cargo.db npm\n  run index:build cargo`) to fix that.\n- **Licence expressions**: an SPDX expression whose operands are *all* permissive resolves to a\n  permissive licence — Rust's near-universal `MIT OR Apache-2.0` is a real choice, not an audit\n  item. Any copyleft operand keeps the conservative treatment `G4` requires: `GPL-3.0 OR MIT`\n  never ships into a permissive project, and a `WITH` exception or unrecognized operand is left\n  for manual audit.\n- **Health evidence is attributed only when the repository claim is corroborated.** A package's\n  repository URL is self-declared, and typosquats name the real project's repo to inherit its\n  OpenSSF score — five PyPI packages claiming `github.com/psf/requests` were reported SHIP 92/100\n  on the real project's 8.1. The claim is now checked against the package name and fails closed\n  (`G17`).\n- **GitHub and Hugging Face components fail-closed to at most \"caution\"** (never \"ship\") — a raw repo's\n  or model's dependency CVEs can't be verified the way a published package's can; Hugging Face also has\n  no OpenSSF-style health score, so it relies on the existing missing-scorecard cap. License (SPDX) is\n  still enriched and gated for both.\n- **PyPI** discovery uses a **self-hosted local index** by default (no key, no third-party service).\n  Build/refresh it once:\n  ```\n  INDEX_MAX=50000 npm run index:build          # top-N PyPI packages by downloads → .cache/index/pypi.db\n  ```\n  The index is `node:sqlite` FTS5 (BM25) over name/description/keywords, semantically reranked by the\n  embedding model. Select the discovery source with `OSSFIND_PYPI_DISCOVERY=index|libraries|auto`\n  (default `auto`: local index if built, else libraries.io).\n- **libraries.io is the fallback** for PyPI (used when no local index exists). It needs a free key in\n  a gitignored `.env.local` (`LIBRARY_IO_API_KEY=…`, `LIBRARIES_IO_API_KEY` also accepted), loaded via\n  `node --env-file=.env.local …`. Without index or key, PyPI discovery degrades to empty (never crashes).\n\n## Live mode & caching\n\nLive mode stores successful supplier responses on disk to reduce repeat requests and avoid supplier\nrate limits. Fixture mode remains local and does not use this cache.\n\n- `OSSFIND_CACHE_DIR` — cache directory (default `.cache/http/`).\n- `OSSFIND_CACHE_TTL` — cache lifetime in seconds for discovery, license, and health data (default `3600`).\n- `OSSFIND_SECURITY_TTL` — cache lifetime in seconds for OSV vulnerability data (default `300`).\n- `OSSFIND_CONCURRENCY` — maximum concurrent upstream enrichment requests (default `4`).\n- `OSSFIND_NO_CACHE=1` — disable the live-response cache.\n\nSecurity responses may be up to `OSSFIND_SECURITY_TTL` seconds stale; tune this value down when\nstricter vulnerability-data freshness is required.\n\nSupplier APIs are free but rate-limited; review each supplier's terms before commercial use.\n\n## Telemetry & Usage Metrics\n\nossfind includes an in-memory, privacy-preserving usage collector that tracks aggregate operational health and supplier rate limits.\n\n### Local Inspection (Read-Only)\n\nYou can inspect usage metrics at any time without sending data anywhere:\n- **MCP Tool:** Call `usage_stats` to receive the metrics snapshot and a formatted summary of top suppliers, cache hit rates, rate-limit headroom, and latency percentiles (p50/p95).\n- **Web API:** Send `GET /api/usage` to retrieve the JSON snapshot. When `OSSFIND_WEB_TOKEN` is set, `/api/usage` requires the same `Authorization: Bearer <token>` header as `/api/search`.\n\n### What Is Collected\n- **Aggregate Supplier Counters:** Total requests, cache hits, cache misses, HTTP status class counts (`2xx`, `4xx`, `5xx`), 429 counts, error counts, and latest rate-limit headroom (`remaining`, `limit`, `reset`, `retryAfter`) per approved supplier host.\n- **Search Operations:** Total searches served, ecosystem distribution (`npm`, `pypi`, `github`, `huggingface`), verdict distribution (`ship`, `caution`, `avoid`), result count summary (min, max, mean), error counts, and latency percentiles (p50, p95).\n- **Anonymous Install ID:** A random UUID v4 generated once and stored locally in `.cache/telemetry/install-id`.\n- **Metadata:** Tool version (`0.1.0`) and ISO 8601 timestamp.\n\n### What Is Explicitly NOT Collected\n- **NO** raw query strings or search phrases.\n- **NO** package names, repository names, or model identifiers.\n- **NO** file paths, local paths, or directory names.\n- **NO** auth tokens, API keys, credentials, or environment secrets.\n- **NO** full URLs, request payloads, or response bodies.\n- **NO** IP addresses, hostnames, usernames, MAC addresses, or hardware fingerprints.\n\n### Opt-In Remote Telemetry (Client-Side)\n\nRemote telemetry is **off by default**. Absolutely no network calls are made unless **both** switches are explicitly set:\n\n```bash\n# Enable remote telemetry by setting BOTH switches:\nexport OSSFIND_TELEMETRY=1\nexport OSSFIND_TELEMETRY_URL=\"https://your-telemetry-collector.example.com/v1/metrics\"\n```\n\n- **Two-switch requirement:** If either `OSSFIND_TELEMETRY=1` or `OSSFIND_TELEMETRY_URL` is omitted, telemetry is completely inert.\n- **HTTPS required:** Ingestion URLs must use `https://`; unencrypted `http://` URLs are rejected.\n- **Batched & Non-blocking:** Telemetry flushes asynchronously in the background and never blocks search or user requests.\n- **Fail-open & silent:** Any network failure, DNS error, timeout, or HTTP error is swallowed silently. It will never break, slow, or alter search results.\n- **Inert in fixture/test mode:** Telemetry never executes when `OSSFIND_FIXTURES=1` or during automated test runs.\n- **To disable:** Unset `OSSFIND_TELEMETRY` (or set `OSSFIND_TELEMETRY=0`) or unset `OSSFIND_TELEMETRY_URL`.\n\n### Telemetry Payload Shape\n\n```json\n{\n  \"installId\": \"c3e98db2-5b94-4f27-9c98-1092e4ab78ef\",\n  \"version\": \"0.1.0\",\n  \"timestamp\": \"2026-08-30T06:30:00.000Z\",\n  \"snapshot\": {\n    \"suppliers\": {\n      \"registry.npmjs.org\": {\n        \"requests\": 14,\n        \"cacheHits\": 12,\n        \"cacheMisses\": 2,\n        \"statusClasses\": { \"1xx\": 0, \"2xx\": 2, \"3xx\": 0, \"4xx\": 0, \"5xx\": 0 },\n        \"rateLimited429\": 0,\n        \"errors\": 0,\n        \"rateLimit\": { \"remaining\": 980, \"limit\": 1000 }\n      },\n      \"api.github.com\": {\n        \"requests\": 4,\n        \"cacheHits\": 3,\n        \"cacheMisses\": 1,\n        \"statusClasses\": { \"1xx\": 0, \"2xx\": 1, \"3xx\": 0, \"4xx\": 0, \"5xx\": 0 },\n        \"rateLimited429\": 0,\n        \"errors\": 0,\n        \"rateLimit\": { \"remaining\": 58, \"limit\": 60, \"reset\": 1725000000 }\n      }\n    },\n    \"operations\": {\n      \"searchesServed\": 3,\n      \"ecosystems\": { \"npm\": 2, \"pypi\": 1, \"github\": 0, \"huggingface\": 0 },\n      \"verdicts\": { \"ship\": 2, \"caution\": 1, \"avoid\": 0 },\n      \"results\": { \"count\": 3, \"total\": 24, \"min\": 5, \"max\": 10, \"mean\": 8.0 },\n      \"errors\": 0,\n      \"latency\": { \"count\": 3, \"p50\": 18, \"p95\": 42, \"reservoirSize\": 3 }\n    }\n  }\n}\n```\n\n## How it works\n\n`discover → enrich → fit → rank`, wired in `src/pipeline/orchestrator.ts`:\n\n| Stage | Module | Source |\n|---|---|---|\n| **Discover** | `src/adapters/discovery.ts` | npm registry search |\n| **Enrich** | `src/adapters/enrichment.ts` | ecosyste.ms (license/repo), deps.dev + OpenSSF Scorecard (health), OSV (vulns) |\n| **Fit** | `src/fit/embeddings.ts` · `src/fit/tfidf.ts` | local embedding model (live) / deterministic TF-IDF (tests), both with a keyword+coverage guard |\n| **Rank** ★ | `src/ranking/rank.ts` | pure, deterministic, explainable blend + verdict |\n\nThe HTTP layer is injectable (`src/http/client.ts`), so every test replays frozen fixtures in\n`fixtures/raw/` — **no test touches the network.**\n\n## The safety model (the moat)\n\n`rank()` is pure and deterministic. It computes five 0–1 sub-scores, an overall 0–100, and a verdict\n(`ship` / `caution` / `avoid`) with **hard, fail-closed rules** that override any high score:\n\n- A vulnerability that **affects the selected version** (version-aware, prerelease-aware) forces\n  `avoid` if critical — regardless of a *future* `fixedIn`.\n- **GPL/AGPL** into a permissive project — across SPDX expression forms (`-or-later`, `+`, `AND`/`OR`,\n  case) — can never `ship`.\n- **Unknown / unverified** license, **failed OSV** retrieval, or **unparseable severity** → capped at\n  `caution`; the engine will not claim \"no vulnerabilities\" when OSV data is missing.\n\nEvery result carries a non-empty, human-readable `reasons[]` explaining the drivers.\n\n## Quality gates (`npm run gates`)\n\nThirteen executable gates, each mapped to the defect/decision that spawned it (see `PIPELINE_LOG.md`),\neach proven to **reject a known-bad input** (not just accept a good one):\n\n`G1` contract · `G2` determinism · `G3` critical-CVSS fact (v3.0/v3.1/v4) · `G4` license SPDX fact ·\n`G5` offline · `G6` version-relevance fact · `G7` evidence completeness · `G8` federation provenance ·\n`G9` Python project-context honesty · `G10` scaffold snippet integrity · `G11` Python stub structural\nhonesty · `G12` recipe resolution honesty · `G13` adoption cannot override safety · `G14` cache\npreserves response bodies · `G15` suggested ESM import matches declared exports · `G16` recall survives\ndiscovery · `G17` health evidence belongs to the package.\n\nEvery gate after `G7` exists because a real bug got past a green test suite — found by an independent\nadversarial audit, by building a real project against the published package (`G14`, `G15`), or by the\nrelevance eval (`G16`).\n\n## Audit trail\n\nEvery component has been independently audited — each audit by an agent that did not write the code —\nand **every one found real bugs the green test suite had missed**. All are fixed and gated.\n\n- `AUDIT_REPORT.md` — the safety layer: 5 blockers (all fixed).\n- `REAUDIT_REPORT.md` — independent re-audit confirming all 5 closed.\n- `CACHE_AUDIT.md` — the live cache: key-collision + stale safety signal (both fixed).\n- `FEDERATION_AUDIT.md` — federation/GitHub/all-ecosystem: 2 structural holes (→ `G8`).\n- `AUDIT_AGENT_LAYER_A.md` — npm/PyPI manifest, compat, MCP, ZIP/range: false \"compatible\" blocker (→ `G9`).\n- `AUDIT_AGENT_LAYER_B.md` — scaffold + stub parser: **code injection** into generated snippets and\n  fabricated exports from docstrings (→ `G10`, `G11`).\n- `AUDIT_AGENT_LAYER_C.md` — recipes + typeshed: fail-open on supplier errors, false \"ready\" (→ `G12`).\n\n## Status & limitations (MVP)\n\n- **Live mode works** against real suppliers, with a persistent disk cache (category-aware TTL:\n  security/OSV data defaults to 300s, everything else 3600s) and a concurrency cap. Fixture mode\n  stays offline and deterministic for tests/demos.\n- **Fit is semantic in live mode** — a local embedding model (`Xenova/all-MiniLM-L6-v2` via\n  `@huggingface/transformers`, mean-pooled, cached per package under `.cache/embeddings/`) ranks by\n  meaning. Fixture/test mode uses deterministic TF-IDF so tests stay offline and exact. Force either\n  with `OSSFIND_FIT=embeddings|tfidf`; live falls back to TF-IDF if the model can't load.\n- Ecosystems: **npm, PyPI, crates.io, RubyGems, GitHub and Hugging Face**, via a federated discovery\n  layer. Discovery composes existing search sources (registry APIs, self-hosted indexes, libraries.io,\n  GitHub) — ossfind owns the safety ranking, not the corpus. Go and Maven remain unbuilt: Go has no\n  free search API, and Maven's `groupId:artifactId` collides with the `ecosystem:name` id convention.\n- **Relevance is measured, not asserted.** `npm run eval` scores a labelled set of 43 queries\n  (MRR, hit@k, recall, noise@3) and diffs against a saved baseline. Registry search matches\n  conjunctively, so a natural-language query excludes terse-description packages — discovery probes\n  with progressively shorter slices of the query and unions the results, and fit is scored before the\n  budgeted enrichment step so a wide pool costs no more than a narrow one. The harness has already\n  rejected one plausible change that measured worse, and caught a defect where a rate-limited search\n  was reported as \"no results\".\n- **Verified signatures cover class members**, so a default-exported class reports its methods and\n  constructor rather than only its name.\n- GitHub repo components fail-closed to \"caution\" (dependency-CVE data isn't available for a raw repo).\n- The self-hosted PyPI index is optional (one source in the federation): FTS5/BM25 recall + semantic\n  rerank, plus a stored-vector hybrid recall (`searchHybrid`). It does not scale to the full ~928k\n  corpus by API crawl — use ecosyste.ms bulk dumps + an ANN index for that (deliberately not built).\n- Bundled fixtures cover ~15 npm + 12 PyPI packages for offline tests/demo; live mode enriches any package.\n- License output is **guidance, not legal advice.**\n",
  "bytes": 18765,
  "sha": "9aaaee661adfbf42ed6aa4aa58b3752c2c77c4188fa6a01a1b1a0c9dfc769a24",
  "repo_slug": "aniket-kr1030/ossfind",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_aniket_kr1030_ossfind_642be70f/readme"
}