{
  "markdown": "<div align=\"center\">\n  <img src=\"RAGScore.png\" alt=\"RAGScore Logo\" width=\"400\"/>\n  \n  [![PyPI version](https://badge.fury.io/py/ragscore.svg)](https://pypi.org/project/ragscore/)\n  [![PyPI Downloads](https://static.pepy.tech/personalized-badge/ragscore?period=total&units=international_system&left_color=black&right_color=green&left_text=downloads)](https://pepy.tech/projects/ragscore)\n  [![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)\n  [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n  [![Ollama](https://img.shields.io/badge/Ollama-Supported-orange)](https://ollama.ai)\n  [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/HZYAI/RagScore/blob/main/examples/detailed_evaluation_demo.ipynb)\n  [![MCP](https://img.shields.io/badge/MCP-Server-purple)](https://modelcontextprotocol.io)\n  \n<!-- mcp-name: io.github.HZYAI/ragscore -->\n\n  **Generate QA datasets & evaluate RAG systems in 2 commands**\n  \n  🔒 Privacy-First • ⚡ Lightning Fast • 🤖 Any LLM • 🏠 Local or Cloud • 🌍 Multilingual\n  \n  [English](README.md) | [中文](README_CN.md) | [日本語](README_JP.md) | [Deutsch](README_DE.md)\n</div>\n\n---\n\n## ⚡ 2-Line RAG Evaluation\n\n```bash\n# Step 1: Generate QA pairs from your docs\nragscore generate docs/\n\n# Step 2: Evaluate your RAG system\nragscore evaluate http://localhost:8000/query\n```\n\n**That's it.** Get accuracy scores and incorrect QA pairs instantly.\n\n```\n============================================================\n✅ EXCELLENT: 85/100 correct (85.0%)\nAverage Score: 4.20/5.0\n============================================================\n\n❌ 15 Incorrect Pairs:\n\n  1. Q: \"What is RAG?\"\n     Score: 2/5 - Factually incorrect\n\n  2. Q: \"How does retrieval work?\"\n     Score: 3/5 - Incomplete answer\n```\n\n---\n\n## 🚀 Quick Start\n\n### Install\n\n```bash\npip install ragscore              # Core (works with Ollama)\npip install \"ragscore[openai]\"    # + OpenAI support\npip install \"ragscore[notebook]\"  # + Jupyter/Colab support\npip install \"ragscore[all]\"       # + All providers\n```\n\n> **Already installed?** Keep up to date — new versions add features like failure diagnosis and retrieved context capture:\n> ```bash\n> pip install --upgrade ragscore\n> ```\n\n### Option 1: Python API (Notebook-Friendly)\n\nPerfect for **Jupyter, Colab, and rapid iteration**. Get instant visualizations.\n\n```python\nfrom ragscore import quick_test\n\n# 1. Audit your RAG in one line\nresult = quick_test(\n    endpoint=\"http://localhost:8000/query\",  # Your RAG API\n    docs=\"docs/\",                            # Your documents\n    n=10,                                    # Number of test questions\n)\n\n# 1b. Tailored QA — target specific audiences\nresult = quick_test(\n    endpoint=\"http://localhost:8000/query\",\n    docs=\"docs/\",\n    audience=\"developers\",                   # Who asks the questions?\n    purpose=\"api-integration\",               # What's the document for?\n)\n\n# 2. See the report\nresult.plot()\n\n# 3. Inspect failures\nbad_rows = result.df[result.df['score'] < 3]\ndisplay(bad_rows[['question', 'rag_answer', 'reason']])\n```\n\n**Rich Object API:**\n- `result.accuracy` - Accuracy score\n- `result.df` - Pandas DataFrame of all results\n- `result.plot()` - 3-panel visualization (4-panel with `detailed=True`)\n- `result.corrections` - List of items to fix\n\n### Option 2: CLI (Production)\n\n### Generate QA Pairs\n\n```bash\n# Set API key (or use local Ollama - no key needed!)\nexport OPENAI_API_KEY=\"sk-...\"\n\n# Generate from any document\nragscore generate paper.pdf\nragscore generate docs/*.pdf --concurrency 10\n\n# Tailored QA generation — target specific audiences\nragscore generate docs/ --audience developers --purpose faq\nragscore generate docs/ --audience customers --purpose \"pre-sales\"\nragscore generate docs/ --audience \"compliance auditors\" --purpose \"security audit\"\n```\n\n### Evaluate Your RAG\n\n```bash\n# Point to your RAG endpoint\nragscore evaluate http://localhost:8000/query\n\n# Custom options\nragscore evaluate http://api/ask --model gpt-4o --output results.json\n```\n\n---\n\n## 🔬 Detailed Multi-Metric Evaluation\n\nGo beyond a single score. Add `detailed=True` to get **5 diagnostic dimensions** per answer — in the same single LLM call.\n\n```python\nresult = quick_test(\n    endpoint=my_rag,\n    docs=\"docs/\",\n    n=10,\n    detailed=True,  # ⭐ Enable multi-metric evaluation\n)\n\n# Inspect per-question metrics\ndisplay(result.df[[\n    \"question\", \"score\", \"correctness\", \"completeness\",\n    \"relevance\", \"conciseness\", \"faithfulness\"\n]])\n\n# Radar chart + 4-panel visualization\nresult.plot()\n```\n\n```\n==================================================\n✅ PASSED: 9/10 correct (90%)\nAverage Score: 4.3/5.0\nThreshold: 70%\n──────────────────────────────────────────────────\n  Correctness: 4.5/5.0\n  Completeness: 4.2/5.0\n  Relevance: 4.8/5.0\n  Conciseness: 4.1/5.0\n  Faithfulness: 4.6/5.0\n==================================================\n```\n\n| Metric | What it measures | Scale |\n|--------|------------------|-------|\n| **Correctness** | Semantic match to golden answer | 5 = fully correct |\n| **Completeness** | Covers all key points | 5 = fully covered |\n| **Relevance** | Addresses the question asked | 5 = perfectly on-topic |\n| **Conciseness** | Focused, no filler | 5 = concise and precise |\n| **Faithfulness** | No fabricated claims | 5 = fully faithful |\n\n**CLI:**\n```bash\nragscore evaluate http://localhost:8000/query --detailed\n```\n\n### 🔍 Failure Diagnosis (`--diagnose`)\n\nWhen answers fail, `--diagnose` tells you **why** — retriever miss, generator hallucination, incomplete answer, or wrong interpretation:\n\n```bash\nragscore evaluate http://localhost:8000/query --diagnose\n```\n\n```\n🔍 Failure Diagnosis:\n  Retriever Miss: 3 (42.9%)\n  Generator Hallucination: 2 (28.6%)\n  Incomplete Answer: 1 (14.3%)\n  Wrong Interpretation: 1 (14.3%)\n```\n\nUses the `support_span` already generated with each QA pair to give the judge grounding context. Combine with `--detailed` for full diagnostics:\n\n```bash\nragscore evaluate http://localhost:8000/query --diagnose --detailed -o results.json\n```\n\n| Category | Meaning |\n|----------|---------|\n| **Retriever Miss** | RAG didn't retrieve the chunk containing the evidence |\n| **Generator Hallucination** | Retrieved correctly but fabricated information |\n| **Incomplete Answer** | Retrieved correctly but answer is partial |\n| **Wrong Interpretation** | Retrieved correctly but misunderstood the content |\n\n> 📓 [Full demo notebook](examples/detailed_evaluation_demo.ipynb) — build a mini RAG and test it with detailed metrics.\n>\n> 🎯 [Audience & Purpose demo](examples/audience_purpose_demo.ipynb) — generate tailored QA for developers, customers, auditors, and more.\n>\n> 🏠 [Ollama local demo](examples/ollama_local_demo.ipynb) — 100% private RAG evaluation with no API keys.\n\n---\n\n## 🏠 100% Private with Local LLMs\n\n```bash\n# Use Ollama - no API keys, no cloud, 100% private\nollama pull llama3.1\nragscore generate confidential_docs/*.pdf\nragscore evaluate http://localhost:8000/query\n```\n\n**Perfect for:** Healthcare 🏥 • Legal ⚖️ • Finance 🏦 • Research 🔬\n\n### Ollama Model Recommendations\n\nRAGScore generates complex structured QA pairs (question + answer + rationale + support span) in JSON format. This requires models with strong instruction-following and JSON output capabilities.\n\n| Model | Size | Min RAM | QA Quality | Recommended |\n|-------|------|---------|------------|-------------|\n| `llama3.1:70b` | 40GB | 48GB VRAM | Excellent | GPU server (A100, L40) |\n| `qwen2.5:32b` | 18GB | 24GB VRAM | Excellent | GPU server (A10, L20) |\n| `llama3.1:8b` | 4.7GB | 8GB VRAM | Good | **Best local choice** |\n| `qwen2.5:7b` | 4.4GB | 8GB VRAM | Good | Good local alternative |\n| `mistral:7b` | 4.1GB | 8GB VRAM | Good | Good local alternative |\n| `llama3.2:3b` | 2.0GB | 4GB RAM | Fair | CPU-only / testing |\n| `qwen2.5:1.5b` | 1.0GB | 2GB RAM | Poor | Not recommended |\n\n> **Minimum recommended: 8B+ models.** Smaller models (1.5B–3B) produce lower quality support spans and may timeout on longer chunks.\n\n### Ollama Performance Guide\n\n```bash\n# Recommended: 8B model with concurrency 2 for local machines\nollama pull llama3.1:8b\nragscore generate docs/ --provider ollama --model llama3.1:8b\n\n# GPU server (A10/L20): larger model with higher concurrency\nollama pull qwen2.5:32b\nragscore generate docs/ --provider ollama --model qwen2.5:32b --concurrency 5\n```\n\n**Expected performance (28 chunks, 5 QA pairs per chunk):**\n\n| Hardware | Model | Time | Concurrency |\n|----------|-------|------|-------------|\n| MacBook (CPU) | llama3.2:3b | ~45 min | 2 |\n| MacBook (CPU) | llama3.1:8b | ~25 min | 2 |\n| A10 (24GB) | llama3.1:8b | ~3–5 min | 5 |\n| L20/L40 (48GB) | qwen2.5:32b | ~3–5 min | 5 |\n| OpenAI API | gpt-4o-mini | ~2 min | 10 |\n\n> RAGScore auto-reduces concurrency to 2 for local Ollama to avoid GPU/CPU contention.\n\n---\n\n## 🔌 Supported LLMs\n\n| Provider | Setup | Notes |\n|----------|-------|-------|\n| **Ollama** | `ollama serve` | Local, free, private |\n| **OpenAI** | `export OPENAI_API_KEY=\"sk-...\"` | Best quality |\n| **Anthropic** | `export ANTHROPIC_API_KEY=\"...\"` | Long context |\n| **DashScope** | `export DASHSCOPE_API_KEY=\"...\"` | Qwen models |\n| **vLLM** | `export LLM_BASE_URL=\"...\"` | Production-grade |\n| **Any OpenAI-compatible** | `export LLM_BASE_URL=\"...\"` | Groq, Together, etc. |\n\n---\n\n## 📊 Output Formats\n\n### Generated QA Pairs (`output/generated_qas.jsonl`)\n\n```json\n{\n  \"id\": \"abc123\",\n  \"question\": \"What is RAG?\",\n  \"answer\": \"RAG (Retrieval-Augmented Generation) combines...\",\n  \"rationale\": \"This is explicitly stated in the introduction...\",\n  \"support_span\": \"RAG systems retrieve relevant documents...\",\n  \"difficulty\": \"medium\",\n  \"source_path\": \"docs/rag_intro.pdf\"\n}\n```\n\n### Evaluation Results (`--output results.json`)\n\n```json\n{\n  \"summary\": {\n    \"total\": 100,\n    \"correct\": 85,\n    \"incorrect\": 15,\n    \"accuracy\": 0.85,\n    \"avg_score\": 4.2\n  },\n  \"incorrect_pairs\": [\n    {\n      \"question\": \"What is RAG?\",\n      \"golden_answer\": \"RAG combines retrieval with generation...\",\n      \"rag_answer\": \"RAG is a database system.\",\n      \"score\": 2,\n      \"reason\": \"Factually incorrect - RAG is not a database\"\n    }\n  ]\n}\n```\n\n---\n\n## 🧪 Python API\n\n```python\nfrom ragscore import run_pipeline, run_evaluation\n\n# Generate QA pairs\nrun_pipeline(paths=[\"docs/\"], concurrency=10)\n\n# Generate tailored QA pairs for specific audiences\nrun_pipeline(\n    paths=[\"docs/\"],\n    audience=\"support engineers\",\n    purpose=\"fine-tuning a support chatbot\",\n)\n\n# Evaluate RAG\nresults = run_evaluation(\n    endpoint=\"http://localhost:8000/query\",\n    model=\"gpt-4o\",  # LLM for judging\n)\nprint(f\"Accuracy: {results.accuracy:.1%}\")\n```\n\n---\n\n## 🤖 AI Agent Integration\n\nRAGScore is designed for AI agents and automation:\n\n```bash\n# Structured CLI with predictable output\nragscore generate docs/ --concurrency 5\nragscore evaluate http://api/query --output results.json\n\n# Exit codes: 0 = success, 1 = error\n# JSON output for programmatic parsing\n```\n\n**CLI Reference:**\n\n| Command | Description |\n|---------|-------------|\n| `ragscore generate <paths>` | Generate QA pairs from documents |\n| `ragscore generate <paths> --audience <who>` | Tailored QA for specific audience |\n| `ragscore generate <paths> --purpose <why>` | Focus QA on document purpose |\n| `ragscore evaluate <endpoint>` | Evaluate RAG against golden QAs |\n| `ragscore evaluate <endpoint> --detailed` | Multi-metric evaluation |\n| `ragscore evaluate <endpoint> --diagnose` | Failure root-cause classification |\n| `ragscore --help` | Show all commands and options |\n| `ragscore generate --help` | Show generate options |\n| `ragscore evaluate --help` | Show evaluate options |\n\n---\n\n## ⚙️ Configuration\n\nZero config required. Optional environment variables:\n\n```bash\nexport RAGSCORE_CHUNK_SIZE=512          # Chunk size for documents\nexport RAGSCORE_QUESTIONS_PER_CHUNK=5   # QAs per chunk\nexport RAGSCORE_WORK_DIR=/path/to/dir   # Working directory\n```\n\n---\n\n## 🔐 Privacy & Security\n\n| Data | Cloud LLM | Local LLM |\n|------|-----------|-----------|\n| Documents | ✅ Local | ✅ Local |\n| Text chunks | ⚠️ Sent to LLM | ✅ Local |\n| Generated QAs | ✅ Local | ✅ Local |\n| Evaluation results | ✅ Local | ✅ Local |\n\n**Compliance:** GDPR ✅ • HIPAA ✅ (with local LLMs) • SOC 2 ✅\n\n---\n\n## 🧪 Development\n\n```bash\ngit clone https://github.com/HZYAI/RagScore.git\ncd RagScore\npip install -e \".[dev,all]\"\npytest\n```\n\n---\n\n## 📡 Telemetry\n\nRAGScore collects telemetry **only in MCP server mode** (`ragscore serve`). Standard CLI and Python API usage do not send telemetry.\n\nWe collect limited anonymous operational metrics to understand feature usage and improve reliability. No document content, prompts, QA text, model outputs, API keys, endpoint URLs, or file paths are collected.\n\n**Collected in MCP mode:**\n- MCP tool invoked\n- LLM provider and model name\n- `ragscore` version, Python version, OS type\n- Success/failure status\n- Random anonymous installation ID\n\n**Opt out:**\n\n```bash\nexport RAGSCORE_NO_TELEMETRY=1\n```\n\n---\n\n## �� Links\n\n- [GitHub](https://github.com/HZYAI/RagScore) • [PyPI](https://pypi.org/project/ragscore/) • [Issues](https://github.com/HZYAI/RagScore/issues) • [Discussions](https://github.com/HZYAI/RagScore/discussions)\n\n---\n\n<p align=\"center\">\n  <b>⭐ Star us on GitHub if RAGScore helps you!</b><br>\n  Made with ❤️ for the RAG community\n</p>\n",
  "bytes": 13482,
  "sha": "fad6db5db0ae11332aa23f68373a213a3f50edce2743656f9ac14a57837d4830",
  "repo_slug": "hzyai/ragscore",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_hzyai_ragscore_7a3207ba/readme"
}