{
  "markdown": "# preprint-fulltext\n\n<p align=\"center\">\n  <b>English</b> |\n  <a href=\"README.zh.md\">简体中文</a> |\n  <a href=\"README.zht.md\">繁體中文</a> |\n  <a href=\"README.ko.md\">한국어</a> |\n  <a href=\"README.de.md\">Deutsch</a> |\n  <a href=\"README.es.md\">Español</a> |\n  <a href=\"README.fr.md\">Français</a> |\n  <a href=\"README.it.md\">Italiano</a> |\n  <a href=\"README.ja.md\">日本語</a>\n</p>\n\n[![PyPI](https://img.shields.io/pypi/v/preprint-fulltext.svg)](https://pypi.org/project/preprint-fulltext/)\n[![Python](https://img.shields.io/pypi/pyversions/preprint-fulltext.svg)](https://pypi.org/project/preprint-fulltext/)\n[![License: BSD-3-Clause](https://img.shields.io/badge/License-BSD--3--Clause-blue.svg)](LICENSE)\n[![CI](https://github.com/genecell/preprint-fulltext/actions/workflows/test.yml/badge.svg)](https://github.com/genecell/preprint-fulltext/actions/workflows/test.yml)\n\nRetrieve the **full text** of bioRxiv / medRxiv / **arXiv** preprints as clean,\nstructured, embedding-ready data — from a CLI, a Python library, or an MCP server.\n\n`preprint-fulltext` turns a DOI (or a search) into structured sections\n(abstract / introduction / methods / results / discussion), a single JSON/Markdown\ndocument, or a chunked JSONL/Parquet corpus ready for embeddings and RAG. openRxiv\ntext-and-data-mining (TDM) compliance is enforced structurally, not left to the user.\n\n> **\"Embedding-ready\" means the output is clean, section-aware, token-bounded chunks —\n> ready to feed to *your* embedding model. Computing embeddings is an optional last step\n> you own; this tool does not bundle an embedding model.**\n\n## Contents\n\n- [Why](#why) · [Who it's for](#who-its-for) · [Full text for AI-driven science](#full-text-for-ai-driven-science)\n- [Features](#features) · [Install](#install) · [Quickstart (CLI)](#quickstart-cli) · [Typical workflows](#typical-workflows)\n- [Python library](#python) · [**MCP server (coding agents)**](#mcp-server) · [Data sources & routing](#data-sources--routing)\n- [Configuration](#configuration) · [Compliance](#compliance) · [Development](#development) · [Contact](#contact) · [License](#license)\n\n---\n\n## Why\n\nPreprint full text is scattered across incompatible channels: Europe PMC serves JATS\nXML for the open-access subset, the openRxiv S3 buckets hold the authoritative\n`.meca` corpus (requester-pays), OpenAlex is a catalog with n-gram-only full-text\nsearch, and the bioRxiv/medRxiv websites render HTML. `preprint-fulltext` unifies\nthem behind one canonical data model and **one shared JATS parser**, so you get the\nsame structured output no matter where a document came from.\n\n## Who it's for\n\n- **ML / NLP researchers** building embedding corpora or RAG systems over the\n  preprint literature.\n- **Bioinformaticians and labs** who need a paper's methods/results as clean text for\n  analysis, extraction, or LLM pipelines.\n- **Coding agents** (via the MCP server / `SKILL.md`) that need to pull a preprint's\n  full text or search the literature mid-task.\n- Anyone who wants **one preprint's sections from a DOI** without hand-parsing JATS or\n  scraping HTML.\n\n## Full text for AI-driven science\n\nLanguage models and agents reason far more reliably over a paper's **methods and results**\nthan over its abstract alone — most scientific claims, protocols, quantities, and caveats\nlive in the body. `preprint-fulltext` gives Claude, Codex, and other agents that body as\nclean, section-labeled, provenance- and license-tagged text, which is the substrate for\n**grounded scientific reasoning and deep research**:\n\n- **Literature deep-research** — read across many papers' full text, not just abstracts.\n- **Methods / protocol extraction** — pull exact procedures, parameters, and datasets.\n- **Claim verification** — check a stated result against the actual Results section.\n- **Reproducibility & meta-analysis** — compare methods and numbers across studies.\n- **RAG over your own corpus** — section-aware, token-bounded chunks with citations.\n\nBecause every `Section`/`Chunk` carries its `kind` (methods / results / …), `source`, and\n`license`, an agent can **cite precisely** (which section of which paper/version) and stay\n**within-license** while it reasons. Full text is retrieval, not memorization: the model\ngrounds its reasoning in the primary source instead of recalling a possibly-stale summary.\n\n## Features\n\n- **`get <id>`** — one preprint's full text as structured JSON or Markdown. bioRxiv/\n  medRxiv route Europe PMC → S3 (opt-in HTML fallback); **arXiv** ids route to arXiv's\n  LaTeXML full text (native HTML → ar5iv). Latest version by default; `--version` selects one.\n- **`search` / `discover`** — keyword, title, abstract, or author search across Europe PMC,\n  OpenAlex, and **arXiv**; topic/category/date discovery.\n- **`ingest`** — resumable, incremental bulk ingestion from the openRxiv S3 buckets\n  into a chunked corpus (JSONL or Parquet) with a sidecar manifest.\n- **MCP server** — the same capabilities as tools for coding agents.\n- **Compliance built in** — an export gate degrades non-redistributable works to\n  link-back stubs; unknown licenses are treated as non-redistributable (fail-safe).\n- **One JATS parser** shared by the Europe PMC and S3 paths; token- and\n  section-aware chunking with deterministic, idempotent chunk ids.\n\n## Install\n\n```bash\npip install preprint-fulltext                       # CLI + Python library + MCP server\npip install \"preprint-fulltext[parquet,openalex]\"   # + Parquet output, pyalex\n```\n\nThe **MCP server is built in** — no extra install and no third-party MCP framework. It's a\nsmall, self-contained JSON-RPC 2.0 stdio server, so `preprint-fulltext-mcp` works out of the\nbox with only the core dependencies.\n\nSet a contact email for the Europe PMC / OpenAlex polite pools (recommended), and an\nOpenAlex API key if you use OpenAlex (required by OpenAlex since 2026-02-13):\n\n```bash\nexport CONTACT_EMAIL=\"you@example.org\"\nexport OPENALEX_API_KEY=\"...\"            # only needed for OpenAlex discover/search\n```\n\n## Quickstart (CLI)\n\n```bash\n# Structured full text for one preprint (Europe PMC → S3 router)\npreprint-fulltext get 10.1101/2024.01.15.575000 --markdown\n\n# Accepts a DOI, a doi.org URL, or a bioRxiv/medRxiv content URL\npreprint-fulltext get https://www.biorxiv.org/content/10.64898/2026.06.13.731750v1.full --html --markdown\n\n# Versions: the DOI resolves to the latest version by default; --version selects one\npreprint-fulltext get 10.64898/2026.01.29.702557 --version 1 --source html --markdown\n\n# arXiv: id, arxiv.org URL, or 10.48550/arXiv.* DOI — routed to arXiv LaTeXML full text\npreprint-fulltext get arXiv:1706.03762 --markdown\npreprint-fulltext get https://arxiv.org/abs/2401.10515 --markdown\n\n# Search: keyword, title, or author (add --source arxiv to search arXiv)\npreprint-fulltext search \"cortical interneurons\" -n 20\npreprint-fulltext search \"Fezf2\" --field title\npreprint-fulltext search \"Min Dai\" --field author\npreprint-fulltext search \"diffusion model\" --field title --source arxiv\n\n# Discover by topic + date window (OpenAlex)\npreprint-fulltext discover --query \"spatial transcriptomics\" --since 2025-01 -n 100\n\n# Bulk corpus from S3 (requester-pays; needs AWS credentials)\npreprint-fulltext ingest corpus.jsonl --source s3 --server biorxiv --since 2025-06\n\n# A free, no-AWS corpus of the open-access (CC) subset via Europe PMC\npreprint-fulltext ingest corpus.jsonl --source europepmc --query \"long covid\"\n```\n\n`get` emits a `FullText` document (JSON) or Markdown (`--markdown`). `search` /\n`discover` stream one `SearchHit` per line (JSONL). `ingest` writes one `Chunk` per\nline plus a `<out>_manifest.jsonl` audit/resume sidecar.\n\n## Typical workflows\n\n**1. Read one paper's methods/results as text.**\n\n```bash\npreprint-fulltext get 10.64898/2026.01.29.702557 --markdown > paper.md\n# -> # Title / ## Abstract / ## Introduction / ## Methods / ## Results / ## Discussion\n```\n\n**2. Build an embedding-ready corpus on a topic (free, no AWS).**\n\n```bash\n# CC/open-access subset via Europe PMC — one Chunk per JSONL line\npreprint-fulltext ingest cortex.jsonl --source europepmc --query \"cortical interneurons\" -n 500\n# cortex.jsonl          -> {doi, version, chunk_id, section_kind, text, token_count, license, ...}\n# cortex_manifest.jsonl -> one row per preprint (doi, version, license, n_chunks, status)\n```\n\n**3. Build the complete corpus for a month from S3 (requester-pays).**\n\n```bash\nexport AWS_PROFILE=...           # needs AWS credentials; ~$0.09/GB\npreprint-fulltext ingest 2025-06.jsonl --source s3 --server both --since 2025-06 --format parquet\n# resumable: rerun after an interruption and it skips finished preprints (no duplicates)\n```\n\n**4. Find papers by author or title, then fetch.**\n\n```bash\npreprint-fulltext search \"Min Dai\" --field author -n 20 > hits.jsonl\npreprint-fulltext get \"$(head -1 hits.jsonl | python -c 'import sys,json;print(json.load(sys.stdin)[\"doi\"])')\" --markdown\n```\n\n**5. Give a coding agent literature access** — run `preprint-fulltext-mcp` and point your\nagent at it (see [`skills/preprint-fulltext/SKILL.md`](skills/preprint-fulltext/SKILL.md)).\n\n## Python\n\n```python\nfrom preprint_fulltext.pipeline.router import Router\n\nresult = Router().get_fulltext(\"10.1101/2024.01.15.575000\")\nif result.fulltext:\n    for section in result.fulltext.sections:\n        print(section.kind, section.title)\n\nfrom preprint_fulltext.core.chunk import chunk_fulltext\nchunks = chunk_fulltext(result.fulltext)   # embedding-ready Chunk records\n```\n\n## MCP server\n\nGive a coding agent live preprint access. The server exposes four tools —\n`search_preprints`, `get_fulltext`, `get_metadata`, `resolve` — over stdio. (Bulk `ingest`\nis intentionally **not** a tool: it is long-running and incurs requester-pays cost.)\n\n`mcp-name: io.github.genecell/preprint-fulltext`\n\nIt's a **local stdio** server, so it works in Claude Code / Cursor / VS Code / Windsurf /\nZed / Codex / Cline — but not the claude.ai web app (there, use the\n[Skill](skills/preprint-fulltext/SKILL.md) instead).\n\n### Recommended: run via `uvx` (no install)\n\n[uv](https://docs.astral.sh/uv/) runs the published package on demand — nothing to\n`pip install` or keep on a PATH. Install uv once:\n\n```bash\ncurl -LsSf https://astral.sh/uv/install.sh | sh    # macOS / Linux\n# or:  pipx install uv  |  pip install --user uv  |  brew install uv  |  winget install astral-sh.uv\n```\n\nThe launch command is `uvx --from preprint-fulltext preprint-fulltext-mcp` (the `--from` is\nneeded because the run command differs from the package name). First launch downloads the\npackage (~30 s); later launches are cached.\n\n<details>\n<summary><b>Claude Code</b> — key <code>mcpServers</code></summary>\n\n```bash\nclaude mcp add preprint-fulltext --scope user -- uvx --from preprint-fulltext preprint-fulltext-mcp\n# uvx not on PATH? use its absolute path:\nclaude mcp add preprint-fulltext --scope user -- \"$(which uvx)\" --from preprint-fulltext preprint-fulltext-mcp\nclaude mcp get preprint-fulltext        # verify → Status: ✔ Connected\n```\n\nOr edit `~/.claude.json` (user) / project `.mcp.json`:\n\n```json\n{ \"mcpServers\": { \"preprint-fulltext\": {\n  \"command\": \"uvx\",\n  \"args\": [\"--from\", \"preprint-fulltext\", \"preprint-fulltext-mcp\"],\n  \"env\": { \"CONTACT_EMAIL\": \"you@example.org\" }\n} } }\n```\n</details>\n\n<details>\n<summary><b>Cursor / Windsurf / Cline / Continue</b> — key <code>mcpServers</code> (same shape)</summary>\n\nCursor: `~/.cursor/mcp.json` (global) or `.cursor/mcp.json` (project). Windsurf:\n`~/.codeium/windsurf/mcp_config.json`. Cline: *MCP Servers → Configure*. Continue:\n`~/.continue/config`.\n\n```json\n{ \"mcpServers\": { \"preprint-fulltext\": {\n  \"command\": \"uvx\",\n  \"args\": [\"--from\", \"preprint-fulltext\", \"preprint-fulltext-mcp\"],\n  \"env\": { \"CONTACT_EMAIL\": \"you@example.org\" }\n} } }\n```\n</details>\n\n<details>\n<summary><b>VS Code</b> (GitHub Copilot, Agent mode) — key <code>servers</code> + <code>type</code></summary>\n\n`.vscode/mcp.json` (workspace) or user `settings.json` under `\"mcp\"`:\n\n```json\n{ \"servers\": { \"preprint-fulltext\": {\n  \"type\": \"stdio\",\n  \"command\": \"uvx\",\n  \"args\": [\"--from\", \"preprint-fulltext\", \"preprint-fulltext-mcp\"]\n} } }\n```\n\nOr one-shot: `code --add-mcp '{\"name\":\"preprint-fulltext\",\"command\":\"uvx\",\"args\":[\"--from\",\"preprint-fulltext\",\"preprint-fulltext-mcp\"]}'`\n</details>\n\n<details>\n<summary><b>Zed</b> — key <code>context_servers</code> (different shape)</summary>\n\n`~/.config/zed/settings.json`:\n\n```json\n{ \"context_servers\": { \"preprint-fulltext\": {\n  \"source\": \"custom\",\n  \"command\": \"uvx\",\n  \"args\": [\"--from\", \"preprint-fulltext\", \"preprint-fulltext-mcp\"],\n  \"env\": {}\n} } }\n```\n</details>\n\n<details>\n<summary><b>Codex</b> (OpenAI Codex CLI) — TOML, not JSON</summary>\n\n`~/.codex/config.toml`:\n\n```toml\n[mcp_servers.preprint-fulltext]\ncommand = \"uvx\"\nargs = [\"--from\", \"preprint-fulltext\", \"preprint-fulltext-mcp\"]\n# env = { CONTACT_EMAIL = \"you@example.org\" }\n```\n\nOr: `codex mcp add preprint-fulltext -- uvx --from preprint-fulltext preprint-fulltext-mcp`\n</details>\n\n### Alternative: install with pip\n\nIf you already `pip install preprint-fulltext`, the server is on your PATH as\n`preprint-fulltext-mcp` — use `\"command\": \"preprint-fulltext-mcp\"` (no `args`) in any config\nabove.\n\n> **Env vars:** set `CONTACT_EMAIL` (Europe PMC / OpenAlex polite pools) and\n> `OPENALEX_API_KEY` (only for OpenAlex search/discover) via the config's `env` block, or in\n> your shell before launching the client. See [`SKILL.md`](skills/preprint-fulltext/SKILL.md)\n> for the full agent-facing tool reference.\n\n## Data sources & routing\n\n| Verb        | Default source        | Notes                                             |\n|-------------|-----------------------|---------------------------------------------------|\n| `get`       | auto (Europe PMC → S3, or arXiv) | bioRxiv/medRxiv: EPMC (CC/OA subset) → S3 (complete, needs AWS creds), `--html` opt-in fallback. **arXiv ids** → arXiv LaTeXML full text (native HTML → ar5iv). |\n| `search`    | Europe PMC            | Real relevance ranking; `--source openalex\\|arxiv`. |\n| `discover`  | OpenAlex              | 250M+ works, OA locations, topic/date; `--source arxiv`. |\n| `ingest`    | S3 (or Europe PMC)    | S3 = complete corpus; Europe PMC = free CC subset. arXiv bulk is out of scope (use arXiv's own S3 LaTeX bucket). |\n\n## Configuration\n\nVia environment variables (prefixed `PREPRINT_FULLTEXT_` or the bare names below),\na `.env` file, or a `preprint-fulltext.toml`:\n\n| Setting | Default | Purpose |\n|---|---|---|\n| `CONTACT_EMAIL` | – | Polite-pool identity for Europe PMC / OpenAlex |\n| `OPENALEX_API_KEY` | – | Required by OpenAlex since 2026-02-13 |\n| `AWS_REGION` | `us-east-1` | Region for the requester-pays openRxiv buckets |\n| `PREPRINT_FULLTEXT_CACHE_DIR` | `~/.cache/preprint-fulltext` | Content-addressed cache |\n| `PREPRINT_FULLTEXT_CHUNK_TOKENS` | `512` | Max tokens per chunk |\n| `PREPRINT_FULLTEXT_CHUNK_OVERLAP` | `64` | Token overlap within a section |\n\n## Compliance\n\nCorpora are for the operator's own text/data mining under the openRxiv TDM terms.\n`preprint-fulltext` does **not** re-host or redistribute preprint full text. Every\n`FullText`/`Chunk` carries its license; the export gate has two modes:\n\n- **analysis** (default): pass-through for your own mining.\n- **redistribution** (`--redistribution`): works whose license permits redistribution\n  pass unchanged; all others are degraded to a **link-back stub** (metadata + URL,\n  no body text). Unknown/ambiguous licenses are treated as non-redistributable.\n\n## Development\n\n```bash\npip install -e \".[dev]\"\npytest              # offline suite (HTTP mocked with respx, S3 with moto)\nruff check preprint_fulltext/\n```\n\nLive tests are opt-in (they hit the real public APIs — Europe PMC, arXiv, and the\nbioRxiv/medRxiv JSON API):\n\n```bash\nPREPRINT_FULLTEXT_LIVE=1 CONTACT_EMAIL=you@example.org pytest -m live   # EPMC / arXiv / medRxiv / versions\nPREPRINT_FULLTEXT_LIVE_S3=1 pytest -m live_s3    # requester-pays S3 (small; needs AWS creds)\n```\n\nThe same live smoke runs in CI on demand (Actions → **live-smoke**) and weekly, to catch\nupstream API drift; the default `test` workflow stays fully offline.\n\nAgent docs (`AGENTS.md`, `llms.txt`, `.cursor/rules/…`, `.github/copilot-instructions.md`)\nare generated from `skills/preprint-fulltext/SKILL.md`:\n\n```bash\npython scripts/build_agent_docs.py\n```\n\n## Contact\n\n**Min Dai** — <dai@broadinstitute.org> ([Gord Fishell Lab](https://fishelllab.hms.harvard.edu),\nHarvard Medical School / Broad Institute). Issues and pull requests welcome at\n<https://github.com/genecell/preprint-fulltext>.\n\n## License\n\nBSD-3-Clause (see [`LICENSE`](LICENSE)). This covers the **software** only —\nretrieved preprint content remains under its author-selected license.\n",
  "bytes": 16722,
  "sha": "5eb7eea1b10892a00775b65e099b951701ddd9810d95c8522e40ded7bc9b39ad",
  "repo_slug": "genecell/preprint-fulltext",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_genecell_preprint_fulltext_4d915e05/readme"
}