{
  "markdown": "# WSO2 Docs MCP Server\n\n[![npm version](https://img.shields.io/npm/v/wso2-docs-mcp-server.svg)](https://www.npmjs.com/package/wso2-docs-mcp-server)\n[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)\n\n>\"This is an unofficial community project. Not affiliated with or endorsed by WSO2.\"\n\nA production-ready **Model Context Protocol (MCP)** server that provides AI assistants (Claude Desktop, Claude Code, Cursor, VS Code) with semantic search over WSO2 documentation via Retrieval-Augmented Generation (RAG).\n\nUnder the hood, it uses a blazing-fast dual-ingestion engine:\n- **GitHub Native:** Fetches raw Markdown directly from WSO2's public GitHub repositories via the Git Trees API (avoids web-scraping noise and rate limits)\n- **Web Crawl Fallback:** For products without dedicated GitHub docs repos (like the WSO2 Library)\n\n## Architecture\n\n![System Architecture](https://raw.githubusercontent.com/iamvirul/wso2-docs-mcp-server/main/docs/architecture.svg)\n\n## Documentation Sources\n\n| Product | ID | URL |\n|---|---|---|\n| API Manager | `apim` | https://apim.docs.wso2.com |\n| Micro Integrator | `mi` | https://mi.docs.wso2.com/en/4.4.0 |\n| Ballerina Integrator | `bi` | https://bi.docs.wso2.com |\n| Choreo | `choreo` | https://wso2.com/choreo/docs |\n| Identity Server | `is` | https://is.docs.wso2.com/en/latest |\n| Ballerina | `ballerina` | https://ballerina.io/learn |\n| WSO2 Library | `library` | https://wso2.com/library |\n\n## Prerequisites\n\n- **Node.js** ≥ 20\n- **Docker** (for pgvector)\n- **Embeddings** - no API key required by default:\n  - **[Ollama](https://ollama.com)** (recommended) - runs locally, model auto-downloaded on first run\n  - If Ollama is not running, the server automatically falls back to **HuggingFace ONNX** (in-process, also downloads automatically)\n  - Cloud providers are also supported: OpenAI, Google Gemini, Voyage AI\n\n---\n\n## Quick Start\n\nChoose the setup path that fits your use case:\n\n- **[Install from npm](#install-from-npm)** - simplest, no cloning required\n- **[Clone and build](#clone-and-build)** - for development or contributions\n\n---\n\n### Install from npm\n\nInstall the package globally to get the `wso2-docs-mcp-server`, `wso2-docs-crawl`, and `wso2-docs-migrate` commands available system-wide:\n\n```bash\nnpm install -g wso2-docs-mcp-server\n```\n\n> **Prefer no global install?** You can use `npx wso2-docs-mcp-server`, `npx wso2-docs-crawl`, and `npx wso2-docs-migrate` in every step below - just replace the bare command with its `npx` equivalent.\n\n#### 1. Start pgvector\n\nDownload the `docker-compose.yml` and start the database:\n\n```bash\ncurl -O https://raw.githubusercontent.com/iamvirul/wso2-docs-mcp-server/main/docker-compose.yml\ndocker compose up -d\n```\n\n#### 2. Start Ollama (optional but recommended)\n\n[Install Ollama](https://ollama.com) and pull the default embedding model:\n\n```bash\nollama pull nomic-embed-text\nollama serve\n```\n\n> **No Ollama?** Skip this step. The server automatically falls back to HuggingFace ONNX - model downloads on first use with no extra setup.\n\n#### 3. Run database migration\n\n```bash\nDATABASE_URL=\"postgresql://wso2mcp:wso2mcp@localhost:5432/wso2docs\" \\\n  wso2-docs-migrate\n```\n\n> Run migration again whenever you change `EMBEDDING_DIMENSIONS` (i.e. switch embedding provider). The script detects and handles dimension changes automatically.\n\n#### 4. Index WSO2 documentation\n\n```bash\n# Index all products (first run downloads the embedding model automatically)\nDATABASE_URL=\"postgresql://wso2mcp:wso2mcp@localhost:5432/wso2docs\" \\\n  wso2-docs-crawl\n\n# Index a single product (faster, great for testing)\nDATABASE_URL=\"postgresql://wso2mcp:wso2mcp@localhost:5432/wso2docs\" \\\n  wso2-docs-crawl --product ballerina --limit 20\n\n# Force re-index even unchanged pages\nDATABASE_URL=\"postgresql://wso2mcp:wso2mcp@localhost:5432/wso2docs\" \\\n  wso2-docs-crawl --force\n```\n\nAvailable product IDs: `apim`, `mi`, `bi`, `choreo`, `is`, `ballerina`, `library`\n\n#### 5. Configure your AI client\n\nThe MCP server is launched on demand by your AI client - no background process needed.\n\n**Claude Desktop** - edit `~/Library/Application Support/Claude/claude_desktop_config.json`:\n\n```json\n{\n  \"mcpServers\": {\n    \"wso2-docs\": {\n      \"command\": \"wso2-docs-mcp-server\",\n      \"env\": {\n        \"DATABASE_URL\": \"postgresql://wso2mcp:wso2mcp@localhost:5432/wso2docs\",\n        \"EMBEDDING_PROVIDER\": \"ollama\"\n      }\n    }\n  }\n}\n```\n\n**Claude Code** - run once in your terminal:\n\n```bash\nclaude mcp add wso2-docs \\\n  --transport stdio \\\n  -e DATABASE_URL=\"postgresql://wso2mcp:wso2mcp@localhost:5432/wso2docs\" \\\n  -e EMBEDDING_PROVIDER=\"ollama\" \\\n  -- wso2-docs-mcp-server\n\n# Verify\nclaude mcp list\n```\n\n**Cursor** - create `.cursor/mcp.json` in your project root:\n\n```json\n{\n  \"mcpServers\": {\n    \"wso2-docs\": {\n      \"command\": \"wso2-docs-mcp-server\",\n      \"env\": {\n        \"DATABASE_URL\": \"postgresql://wso2mcp:wso2mcp@localhost:5432/wso2docs\",\n        \"EMBEDDING_PROVIDER\": \"ollama\"\n      }\n    }\n  }\n}\n```\n\n**VS Code** - create `.vscode/mcp.json`:\n\n```json\n{\n  \"servers\": {\n    \"wso2-docs\": {\n      \"type\": \"stdio\",\n      \"command\": \"wso2-docs-mcp-server\",\n      \"env\": {\n        \"DATABASE_URL\": \"postgresql://wso2mcp:wso2mcp@localhost:5432/wso2docs\",\n        \"EMBEDDING_PROVIDER\": \"ollama\"\n      }\n    }\n  }\n}\n```\n\n> **Using `npx` instead of global install?** Replace `\"command\": \"wso2-docs-mcp-server\"` with `\"command\": \"npx\"` and add `\"args\": [\"-y\", \"wso2-docs-mcp-server\"]`.\n\n> **Cloud embedding provider?** Add the key to `env`, e.g. `\"EMBEDDING_PROVIDER\": \"openai\", \"OPENAI_API_KEY\": \"sk-...\"`.\n\n---\n\n### Clone and build\n\n#### 1. Clone and install\n\n```bash\ngit clone https://github.com/iamvirul/wso2-docs-mcp-server.git\ncd wso2-docs-mcp-server\nnpm install\n```\n\n#### 2. Start Ollama (optional but recommended)\n\n[Install Ollama](https://ollama.com) and start it:\n\n```bash\nollama serve\n```\n\n> **No Ollama?** Skip this step. The server detects Ollama is not running and automatically falls back to HuggingFace ONNX inference - the model downloads on first use with no extra setup.\n\n#### 3. Configure environment\n\n```bash\ncp .env.example .env\n# Defaults work out of the box with Ollama.\n# Only edit if using a cloud provider (OpenAI / Gemini / Voyage).\n```\n\n#### 4. Start pgvector\n\n```bash\ndocker compose up -d\n# pgAdmin available at http://localhost:5050 (admin@wso2mcp.local / admin)\n```\n\n#### 5. Run database migration\n\n```bash\nnpm run db:migrate\n```\n\n> **Note:** Run migration again whenever you change `EMBEDDING_DIMENSIONS` (i.e. switch embedding provider). The script detects and handles dimension changes automatically.\n\n#### 6. Index documentation\n\n```bash\n# Index all products\n# On first run the embedding model is downloaded automatically (Ollama or HuggingFace)\nnpm run crawl\n\n# Index a single product (faster, great for testing)\nnpm run crawl -- --product ballerina --limit 20\n\n# Force re-index even unchanged pages\nnpm run crawl -- --force\n```\n\n#### 7. Build and start the MCP server\n\n```bash\nnpm run build\nnpm start\n```\n\nFor development (no build step):\n```bash\nnpm run dev\n```\n\n#### 8. Configure your AI client\n\n> Replace `/ABSOLUTE/PATH/TO/wso2-docs-mcp-server` with your actual clone path.\n\n**Claude Desktop** - edit `~/Library/Application Support/Claude/claude_desktop_config.json`:\n\n```json\n{\n  \"mcpServers\": {\n    \"wso2-docs\": {\n      \"command\": \"node\",\n      \"args\": [\"/ABSOLUTE/PATH/TO/wso2-docs-mcp-server/dist/src/index.js\"],\n      \"env\": {\n        \"DATABASE_URL\": \"postgresql://wso2mcp:wso2mcp@localhost:5432/wso2docs\",\n        \"EMBEDDING_PROVIDER\": \"ollama\"\n      }\n    }\n  }\n}\n```\n\n**Claude Code:**\n\n```bash\nclaude mcp add wso2-docs \\\n  --transport stdio \\\n  -e DATABASE_URL=\"postgresql://wso2mcp:wso2mcp@localhost:5432/wso2docs\" \\\n  -e EMBEDDING_PROVIDER=\"ollama\" \\\n  -- node \"/ABSOLUTE/PATH/TO/wso2-docs-mcp-server/dist/src/index.js\"\n\n# Verify\nclaude mcp list\n```\n\nSee `config-examples/claude_code.sh` for a convenience script.\n\n**Cursor** - create `.cursor/mcp.json` - see `config-examples/cursor_mcp.json`.\n\n**VS Code** - create `.vscode/mcp.json` - see `config-examples/vscode_mcp.json`.\n\n---\n\n## MCP Tools\n\n| Tool | Description |\n|---|---|\n| `search_wso2_docs` | Semantic search across all products. Optional `product` and `limit` filters. |\n| `get_wso2_guide` | Search within a specific product (`apim`, `mi`, `bi`, `choreo`, `is`, `ballerina`, `library`). |\n| `explain_wso2_concept` | Broad concept search across all products, returns 8 top results. |\n| `list_wso2_products` | Returns all supported products with IDs and base URLs. |\n\n### Example response\n\n```json\n[\n  {\n    \"title\": \"Deploying WSO2 API Manager\",\n    \"snippet\": \"WSO2 API Manager can be deployed in various topologies…\",\n    \"source_url\": \"https://apim.docs.wso2.com/en/latest/install-and-setup/...\",\n    \"product\": \"apim\",\n    \"section\": \"Deployment Patterns\",\n    \"score\": 0.8712\n  }\n]\n```\n\n---\n\n## Local Embeddings\n\nThe default `EMBEDDING_PROVIDER=ollama` runs entirely on your machine with no API key. The startup sequence is:\n\n```\nIs Ollama running?\n├── Yes → Is model present?\n│         ├── Yes → Ready (instant)\n│         └── No  → Pull via Ollama (streamed, runs once)\n└── No  → Download ONNX model from HuggingFace Hub (~250 MB, cached after first run)\n           and run inference in-process via @huggingface/transformers\n```\n\nBoth paths use `nomic-embed-text` / `Xenova/nomic-embed-text-v1` by default and produce identical 768-dim vectors, so you can switch between them without re-indexing.\n\n### Hardware acceleration (HuggingFace ONNX fallback)\n\nWhen Ollama is not available, the server auto-detects the best compute backend:\n\n| Machine | Detection | ONNX dtype | Batch size | Throughput |\n|---|---|---|---|---|\n| Apple Silicon (M1/M2/M3/M4) | `process.arch === 'arm64'` | `q8` INT8 | 32 | ~9 ms/chunk |\n| NVIDIA GPU | `nvidia-smi` probe | `fp32` | 64 | GPU-dependent |\n| All others | fallback | `q8` INT8 | 16 | ~10 ms/chunk |\n\n**Why `q8` on Apple Silicon instead of CoreML/Metal?**\nCoreML compiles Metal shaders on first use (~20 min cold-start). For the typical chunk sizes produced by this server (6–20 chunks per page), the CPU↔GPU transfer overhead eliminates any inference gain. INT8 quantized inference on ARM NEON SIMD is consistently **~100× faster than fp32 CPU** with zero cold-start cost.\n\n**Benchmark (Apple M-chip, `Xenova/nomic-embed-text-v1`):**\n```\nfp32 CPU (before): ~1,000 ms/chunk   (68 chunks ≈ 68 s of embedding)\nq8  ARM NEON:          ~9 ms/chunk   (68 chunks ≈  0.6 s of embedding)  ← ~100× speedup\n```\n\n> **Note:** For small crawls (≤ 10 pages) total wall-clock time is dominated by network I/O\n> (HTTPS fetches to docs sites), so the end-to-end improvement is modest. The embedding\n> speedup becomes significant at scale - crawling 500+ pages where embedding previously\n> accounted for hours of runtime. For best crawl performance, run Ollama (`ollama serve`)\n> which parallelises inference natively and has no per-chunk overhead.\n\n---\n\n## Environment Variables\n\n### Core\n\n| Variable | Default | Description |\n|---|---|---|\n| `DATABASE_URL` | - | PostgreSQL connection string (required) |\n| `EMBEDDING_PROVIDER` | `ollama` | `ollama` \\| `openai` \\| `gemini` \\| `voyage` |\n| `EMBEDDING_DIMENSIONS` | `768` | Must match model output dimensions |\n| `CRAWL_CONCURRENCY` | `5` | Concurrent HTTP requests during crawl |\n| `CHUNK_SIZE` | `800` | Approximate tokens per chunk |\n| `CHUNK_OVERLAP` | `100` | Overlap tokens between chunks |\n| `CACHE_TTL_SECONDS` | `3600` | In-memory query cache TTL |\n| `TOP_K_RESULTS` | `10` | Default search result count |\n\n### Ollama (default)\n\n| Variable | Default | Description |\n|---|---|---|\n| `OLLAMA_BASE_URL` | `http://localhost:11434` | Ollama server URL |\n| `OLLAMA_EMBEDDING_MODEL` | `nomic-embed-text` | Model pulled and used via Ollama |\n| `HUGGINGFACE_EMBEDDING_MODEL` | `Xenova/nomic-embed-text-v1` | ONNX fallback when Ollama is not running |\n\n### Cloud providers\n\n| Variable | Default | Description |\n|---|---|---|\n| `OPENAI_API_KEY` | - | Required if `EMBEDDING_PROVIDER=openai` |\n| `OPENAI_EMBEDDING_MODEL` | `text-embedding-3-small` | OpenAI model |\n| `GEMINI_API_KEY` | - | Required if `EMBEDDING_PROVIDER=gemini` |\n| `GEMINI_EMBEDDING_MODEL` | `text-embedding-004` | Gemini model |\n| `VOYAGE_API_KEY` | - | Required if `EMBEDDING_PROVIDER=voyage` |\n| `VOYAGE_EMBEDDING_MODEL` | `voyage-3` | Voyage model |\n\n### Embedding dimension reference\n\n| Provider | Model | Dimensions |\n|---|---|---|\n| Ollama / HuggingFace | `nomic-embed-text` / `Xenova/nomic-embed-text-v1` | **768** (default) |\n| Ollama / HuggingFace | `mxbai-embed-large` / `Xenova/mxbai-embed-large-v1` | 1024 |\n| Ollama / HuggingFace | `all-minilm` / `Xenova/all-MiniLM-L6-v2` | 384 |\n| OpenAI | `text-embedding-3-small` | 1536 |\n| OpenAI | `text-embedding-3-large` | 3072 |\n| Gemini | `text-embedding-004` | 768 |\n| Voyage | `voyage-3` | 1024 |\n| Voyage | `voyage-3-lite` | 512 |\n\n---\n\n## Scheduled Re-indexing\n\n```bash\n# Run a one-off re-index (checks hashes, skips unchanged pages)\nnpm run reindex\n\n# Or from the project directory using node-cron (runs daily at 2 AM)\nDATABASE_URL=... node -e \"\n  const { ReindexJob } = require('./dist/jobs/reindexDocs');\n  const job = new ReindexJob();\n  job.initialize().then(() => job.scheduleDaily());\n\"\n```\n\n---\n\n## Project Structure\n\n```\nsrc/\n  config/          env.ts · constants.ts\n  vectorstore/     pgvector.ts · schema.sql\n  ingestion/       crawler.ts · parser.ts · githubFetcher.ts · markdownParser.ts · chunker.ts · embedder.ts\n  server/          mcpServer.ts · toolRegistry.ts\n  jobs/            reindexDocs.ts\n  index.ts\nscripts/\n  crawl.ts         CLI ingestion pipeline\n  migrate.ts       Dynamic schema migration\nconfig-examples/   claude_desktop.json · claude_code.sh · cursor_mcp.json · vscode_mcp.json\ndocker-compose.yml\n.env.example\n```\n\n---\n\n## Development\n\n```bash\n# Type-check\nnpx tsc --noEmit\n\n# Run crawl with tsx (no build needed)\nnpm run crawl -- --product ballerina --limit 5\n\n# Run server in dev mode\nnpm run dev\n```\n",
  "bytes": 14040,
  "sha": "c3c70509434253be9f6d15e8e0c88881683140e79010b7d56f6c4c6407c91daa",
  "repo_slug": "iamvirul/wso2-docs-mcp-server",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_iamvirul_wso2_docs_mcp_server_ddbdedee/readme"
}