{
  "markdown": "# MarkGrab\n\n[![PyPI](https://img.shields.io/pypi/v/markgrab)](https://pypi.org/project/markgrab/)\n[![Python](https://img.shields.io/pypi/pyversions/markgrab)](https://pypi.org/project/markgrab/)\n[![License](https://img.shields.io/github/license/QuartzUnit/markgrab)](https://github.com/QuartzUnit/markgrab/blob/main/LICENSE)\n[![Tests](https://img.shields.io/badge/tests-114%20passed-brightgreen)]()\n\n> [한국어 문서](README.ko.md) · [llms.txt](llms.txt)\n\nUniversal web content extraction — any URL to LLM-ready markdown.\n\n```python\nfrom markgrab import extract\n\nresult = await extract(\"https://example.com/article\")\nprint(result.markdown)    # clean markdown\nprint(result.title)       # \"Article Title\"\nprint(result.word_count)  # 1234\nprint(result.language)    # \"en\"\n```\n\n## Features\n\n- **HTML** — BeautifulSoup + content density filtering (removes nav, sidebar, ads)\n- **YouTube** — transcript extraction with timestamps\n- **PDF** — text extraction with page structure\n- **DOCX** — paragraph and heading extraction\n- **Auto-fallback** — tries lightweight httpx first, falls back to Playwright for JS-heavy pages\n- **Async-first** — built on httpx and Playwright async APIs\n\n## Install\n\n```bash\npip install markgrab\n```\n\nOptional extras for specific content types:\n\n```bash\npip install \"markgrab[browser]\"    # Playwright for JS-rendered pages\npip install \"markgrab[youtube]\"    # YouTube transcript extraction\npip install \"markgrab[pdf]\"       # PDF text extraction\npip install \"markgrab[docx]\"      # DOCX text extraction\npip install \"markgrab[all]\"       # everything\n```\n\n## Usage\n\n### Python API\n\n```python\nimport asyncio\nfrom markgrab import extract\n\nasync def main():\n    # HTML (auto-detects content type)\n    result = await extract(\"https://example.com/article\")\n\n    # YouTube transcript\n    result = await extract(\"https://youtube.com/watch?v=dQw4w9WgXcQ\")\n\n    # PDF\n    result = await extract(\"https://arxiv.org/pdf/1706.03762\")\n\n    # Options\n    result = await extract(\n        \"https://example.com\",\n        max_chars=30_000,       # limit output length (default: 50K)\n        use_browser=True,       # force Playwright rendering\n        stealth=True,           # anti-bot stealth scripts (opt-in)\n        timeout=60.0,           # request timeout in seconds\n        proxy=\"http://proxy:8080\",\n    )\n\nasyncio.run(main())\n```\n\n### CLI\n\n```bash\nmarkgrab https://example.com                     # markdown output\nmarkgrab https://example.com -f text             # plain text\nmarkgrab https://example.com -f json             # structured JSON\nmarkgrab https://example.com --browser           # force browser rendering\nmarkgrab https://example.com --max-chars 10000   # limit output\n```\n\n### ExtractResult\n\n```python\nresult.title        # page title\nresult.text         # plain text\nresult.markdown     # LLM-ready markdown\nresult.word_count   # word count\nresult.language     # detected language (\"en\", \"ko\", ...)\nresult.content_type # \"article\", \"video\", \"pdf\", \"docx\"\nresult.source_url   # final URL (after redirects)\nresult.metadata     # extra metadata (video_id, page_count, etc.)\n```\n\n## How it works\n\n```mermaid\nflowchart TD\n    A[\"🔗 URL Input\"] --> B{\"Content\\nType?\"}\n    B -->|\"HTML\"| C[\"HTTP fetch\\n(httpx)\"]\n    C --> D{\"JS\\nrequired?\"}\n    D -->|\"no\"| E[\"HTML Parser\\n→ clean markdown\"]\n    D -->|\"yes\"| F[\"Playwright\\nfallback\"]\n    F --> E\n    B -->|\"YouTube\"| G[\"Transcript API\\n→ timestamped markdown\"]\n    B -->|\"PDF\"| H[\"PDF Parser\\n→ structured markdown\"]\n    B -->|\"DOCX\"| I[\"DOCX Parser\\n→ markdown\"]\n    E --> J[\"✅ LLM-ready\\nMarkdown\"]\n    G --> J\n    H --> J\n    I --> J\n```\n\nFor HTML pages, if the initial httpx fetch yields fewer than 50 words, MarkGrab automatically retries with Playwright to handle JavaScript-rendered content.\n\n## Disclaimer\n\n**This software is provided for legitimate purposes only.** By using MarkGrab, you agree to the following:\n\n- **robots.txt**: MarkGrab does **not** check or enforce `robots.txt`. Users are solely responsible for checking and respecting `robots.txt` directives and the terms of service of any website they access.\n\n- **Rate limiting**: MarkGrab does **not** include built-in rate limiting or request throttling. Users must implement their own rate limiting to avoid overloading target servers. Abusive request patterns may violate applicable laws and website terms of service.\n\n- **YouTube transcripts**: YouTube transcript extraction relies on the third-party `youtube-transcript-api` library, which uses YouTube's internal (unofficial) caption API. This may not comply with YouTube's Terms of Service. Use at your own discretion and risk.\n\n- **Stealth mode**: The optional `stealth=True` feature modifies browser fingerprinting signals to reduce bot detection. This feature is intended for legitimate use cases such as testing, research, and accessing content that is publicly available to regular browser users. Users are responsible for ensuring their use complies with applicable laws and the terms of service of target websites.\n\n- **Legal compliance**: Users are responsible for ensuring that their use of MarkGrab complies with all applicable laws, including but not limited to the Computer Fraud and Abuse Act (CFAA), the Digital Millennium Copyright Act (DMCA), GDPR, and equivalent legislation in their jurisdiction.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND. See the [LICENSE](LICENSE) file for the full MIT license text.\n\n## Acknowledgments\n\nMarkGrab builds on excellent open-source work and well-established techniques:\n\n- **[puppeteer-extra-plugin-stealth](https://github.com/nicoleahmed/puppeteer-extra-plugin-stealth)** — stealth evasion patterns (webdriver removal, plugin mocking, WebGL spoofing) that inspired the opt-in `anti_bot/stealth.py` module\n- **[Mozilla Readability](https://github.com/mozilla/readability)** — content area detection priority (`article > main > body`) and link density filtering concepts used in the density filter\n- **[Boilerpipe](https://github.com/kohlschutter/boilerpipe)** (Kohlschutter et al., 2010) — the academic origin of link density ratio algorithms for boilerplate removal\n- **[Jina Reader](https://github.com/jina-ai/reader)** — validated the market need for URL-to-markdown extraction; MarkGrab aims to be a lightweight, self-hosted alternative\n\nBuilt with [httpx](https://github.com/encode/httpx), [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/), [markdownify](https://github.com/matthewwithanm/python-markdownify), [Playwright](https://github.com/microsoft/playwright-python), [youtube-transcript-api](https://github.com/jdepoix/youtube-transcript-api), [pdfplumber](https://github.com/jsvine/pdfplumber), and [python-docx](https://github.com/python-openxml/python-docx).\n\n## Used in\n\n- [newswatch](https://github.com/QuartzUnit/newswatch) — RSS news monitoring pipeline (feedkit → markgrab → embgrep → diffgrab)\n- [watchdeck](https://github.com/QuartzUnit/watchdeck) — Web page monitoring with visual diffs and safety guards\n\n## License\n\n[MIT](LICENSE)\n\n<!-- mcp-name: io.github.ArkNill/markgrab -->\n\n\n---\n\n<sub>Part of the [QuartzUnit](https://github.com/QuartzUnit) ecosystem — composable Python libraries for data collection, extraction, search, and AI agent safety.</sub>\n",
  "bytes": 7259,
  "sha": "07f3aa21a0ae82788e63d0effdb3e431140e0737408c32c42c45ae862dd5f4e5",
  "repo_slug": "quartzunit/markgrab",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_arknill_markgrab_afed1413/readme"
}