llms.txt for agents
Any site's llms.txt: find the covering index, read its linked docs as markdown, search sections.
Open source Repository Open in the app JSON README (API)
About
Any site's llms.txt: find the covering index, read its linked docs as markdown, search sections.
Details
- Kind
- MCP servers
- Topic
- Files & documents
- Publisher
- ux-xd
- Origin
- official
- Category
- ferramentas
- Transport
- http
- Version
- 0.1.1
- Last push
- 2026-09-03T00:48:18Z
- Repository state
- ativo
- Language
- TypeScript
- License
- MIT
- Added
- 2026-09-01 22:00:54
- Updated
- 2026-09-01 23:00:53
- Origin id
io.github.ux-xd/llms-txt
README
# llms.txt for agents
> Any site's llms.txt: find the covering index, read its linked docs as markdown, search sections. Remote MCP server over Streamable HTTP at `https://llmstxtmcp.dev/mcp`. Free, no API key. Tools: `llms_index`, `llms_page`, `llms_search`. The tool list is fixed per version.
Given any site or page URL, llms_index discovers the llms.txt that covers it (root, redirects, Link: rel="describedby" advertisements, subpath files with most-specific-wins), parses it into the v2 structure (title, summary, H2 sections of [title](url): notes links, the Optional section) and reports every rung it tried, so a miss is as informative as a hit. llms_page returns any linked page as markdown by content negotiation (Accept: text/markdown), rel="alternate" markdown mirrors, the .md / .html.md / index.md conventions, or a bounded HTML-to-markdown conversion, paginated by character offset. llms_search navigates every index this server has parsed so far by section heading, link title and notes. Free, keyless, no account; the server keeps only the parsed indexes it has seen.
Use when: The user needs a product's or library's own documentation and the site may publish llms.txt: call llms_index with the site first, then llms_page on the links it returns. Use llms_page directly when you already hold a docs URL and want it as markdown instead of HTML.
Not for: General web scraping or crawling (one fetch per rung, never link-following), pages behind logins, generating or auditing a site's own llms.txt, or searching the whole web for documentation.
Quick install (Claude Code): `claude mcp add --transport http llms-txt https://llmstxtmcp.dev/mcp -s user`
Live server: https://llmstxtmcp.dev
## Docs
- [Full reference](https://llmstxtmcp.dev/llms-full.txt): every tool with its JSON input schema and an example
- [Install for any client](https://llmstxtmcp.dev/install): Claude Code, Cursor, VS Code, Claude Desktop, ChatGPT, Codex, Gemini CLI, Windsurf, Cline, Continue, Zed
- [OpenAPI twin](https://llmstxtmcp.dev/openapi.json): the same tools as plain HTTP POST endpoints
- [Registry server.json](https://llmstxtmcp.dev/.well-known/mcp/server.json): machine-readable server record
- [Server card](https://llmstxtmcp.dev/.well-known/mcp/server-card.json): transport + tool card (also at https://llmstxtmcp.dev/.well-known/mcp.json, SEP-1960)
- [ARD manifest](https://llmstxtmcp.dev/.well-known/ard.json): Agentic Resource Discovery entry (also at https://llmstxtmcp.dev/.well-known/ai-catalog.json)
## Upstreams
- [The /llms.txt file (v2 spec)](https://llmstxt.org/index.md): keyless
- [Markdown for Agents (Accept: text/markdown negotiation)](https://blog.cloudflare.com/markdown-for-agents/): keyless
## Optional
- [Health](https://llmstxtmcp.dev/healthz): liveness JSON
- [Owner](https://github.com/ux-xd/llms-txt-mcp): ux-xd
## Tools
| Tool | Purpose | Effects |
| --- | --- | --- |
| `llms_index` | Find and parse the llms.txt that covers a site or page URL: title, summary, H2 sections of [title](url) links with notes, the Optional section, and (HEAD-probed only) whether an llms-full.txt exists. Use first, before llms_page, whenever the user needs a product or library's own docs; every rung tried is returned so a miss is diagnosable. | read-only, open-world, idempotent |
| `llms_page` | Return one docs page as markdown, paginated: tries Accept: text/markdown negotiation, the page's rel="alternate" markdown mirror, the .md / .html.md / index.md conventions, then a bounded HTML-to-markdown conversion. Use on the URLs llms_index returns, or on any docs URL you already hold. | read-only, open-world, idempotent |
| `llms_search` | Search the llms.txt indexes this server has parsed (section headings ×3, link titles ×2, notes ×1) and get the matching docs links with their covering index; pass site to search one site (it is indexed on demand). Use to jump straight to the right page of a docs set instead of reading the whole index. | read-only, open-world, idempotent |
Every tool takes `task_context`: one sentence on what the user is trying to do. It is required.
### `llms_index`
Find and parse the llms.txt that covers a site or page URL: title, summary, H2 sections of [title](url) links with notes, the Optional section, and (HEAD-probed only) whether an llms-full.txt exists. Use first, before llms_page, whenever the user needs a product or library's own docs; every rung tried is returned so a miss is diagnosable.
Input schema:
```json
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"site": {
"type": "string",
"minLength": 3,
"maxLength": 2000,
"description": "Site or page URL, e.g. \"docs.stripe.com\" or \"https://hono.dev/docs/\". The most specific covering llms.txt wins."
},
"refresh": {
"default": false,
"description": "Bypass the 24 h cache and fetch again.",
"type": "boolean"
},
"task_context": {
"type": "string",
"minLength": 1,
"maxLength": 500,
"description": "One sentence on what the user is ultimately trying to do (the task this call serves). Required; it tunes the result and is how this free service learns what agents need."
}
},
"required": [
"site",
"refresh",
"task_context"
],
"additionalProperties": false
}
```
Example arguments:
```json
{
"site": "hono.dev",
"task_context": "example: Find and parse the llms.txt that covers a site or page URL: "
}
```
### `llms_page`
Return one docs page as markdown, paginated: tries Accept: text/markdown negotiation, the page's rel="alternate" markdown mirror, the .md / .html.md / index.md conventions, then a bounded HTML-to-markdown conversion. Use on the URLs llms_index returns, or on any docs URL you already hold.
Input schema:
```json
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"url": {
"type": "string",
"maxLength": 2000,
"format": "uri",
"description": "Page URL (http/https)."
},
"max_chars": {
"default": 60000,
"description": "Characters of markdown to return in this call.",
"type": "integer",
"minimum": 1000,
"maximum": 200000
},
"offset": {
"default": 0,
"description": "Character offset to continue from (use next_offset from a truncated result).",
"type": "integer",
"minimum": 0,
"maximum": 9007199254740991
},
"task_context": {
"type": "string",
"minLength": 1,
"maxLength": 500,
"description": "One sentence on what the user is ultimately trying to do (the task this call serves). Required; it tunes the result and is how this free service learns what agents need."
}
},
"required": [
"url",
"max_chars",
"offset",
"task_context"
],
"additionalProperties": false
}
```
Example arguments:
```json
{
"url": "https://hono.dev/docs/",
"task_context": "example: Return one docs page as markdown, paginated: tries Accept: t"
}
```
### `llms_search`
Search the llms.txt indexes this server has parsed (section headings ×3, link titles ×2, notes ×1) and get the matching docs links with their covering index; pass site to search one site (it is indexed on demand). Use to jump straight to the right page of a docs set instead of reading the whole index.
Input schema:
```json
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"query": {
"type": "string",
"minLength": 1,
"maxLength": 200,
"description": "Words to match against section headings, link titles and notes."
},
"site": {
"description": "Restrict to one site (indexed via llms_index if not yet cached).",
"type": "string",
"maxLength": 2000
},
"limit": {
"default": 10,
"type": "integer",
"minimum": 1,
"maximum": 50
},
"max_bytes": {
"default": 200000,
"description": "Byte cap on the results array.",
"type": "integer",
"minimum": 10000,
"maximum": 500000
},
"task_context": {
"type": "string",
"minLength": 1,
"maxLength": 500,
"description": "One sentence on what the user is ultimately trying to do (the task this call serves). Required; it tunes the result and is how this free service learns what agents need."
}
},
"required": [
"query",
"limit",
"max_bytes",
"task_context"
],
"additionalProperties": false
}
```
Example arguments:
```json
{
"query": "middleware",
"site": "hono.dev",
"task_context": "example: Search the llms.txt indexes this server has parsed (section "
}
```
## Install
Endpoint: `https://llmstxtmcp.dev/mcp` (Streamable HTTP, MCP 2026-07-28 with 2025-era fallback). Authentication: none.
#### Claude Code
```
claude mcp add --transport http llms-txt https://llmstxtmcp.dev/mcp -s user
```
#### Cursor (~/.cursor/mcp.json)
```json
{
"mcpServers": {
"llms-txt": {
"url": "https://llmstxtmcp.dev/mcp"
}
}
}
```
#### VS Code / Copilot (user mcp.json)
```json
{
"servers": {
"llms-txt": {
"type": "http",
"url": "https://llmstxtmcp.dev/mcp"
}
}
}
```
or `code --add-mcp '{"name":"llms-txt","type":"http","url":"https://llmstxtmcp.dev/mcp"}'`
#### Claude Desktop / claude.ai
Settings → Connectors → Add custom connector → URL `https://llmstxtmcp.dev/mcp`, Authentication: None.
#### ChatGPT
Settings → Connectors → Add custom connector → `https://llmstxtmcp.dev/mcp`. Desktop app / Codex share `~/.codex/config.toml`:
```toml
[mcp_servers.llms-txt]
url = "https://llmstxtmcp.dev/mcp"
```
#### Codex CLI (~/.codex/config.toml)
```toml
[mcp_servers.llms-txt]
url = "https://llmstxtmcp.dev/mcp"
```
#### Gemini CLI
```
gemini mcp add --transport http llms-txt https://llmstxtmcp.dev/mcp -s user
```
(settings.json uses `httpUrl`, not `url`.)
#### Windsurf (~/.codeium/windsurf/mcp_config.json)
```json
{
"mcpServers": {
"llms-txt": {
"serverUrl": "https://llmstxtmcp.dev/mcp"
}
}
}
```
#### Cline
```json
{
"mcpServers": {
"llms-txt": {
"type": "streamableHttp",
"url": "https://llmstxtmcp.dev/mcp"
}
}
}
```
#### Continue (.continue/mcpServers/llms-txt.yaml)
```yaml
name: llms-txt
mcpServers:
- name: llms-txt
type: streamable-http
url: https://llmstxtmcp.dev/mcp
```
#### Zed (settings.json)
```json
{
"context_servers": {
"llms-txt": {
"source": "custom",
"command": "npx",
"args": [
"-y",
"mcp-remote",
"https://llmstxtmcp.dev/mcp"
]
}
}
}
```
#### Any MCP client
Streamable HTTP endpoint: `https://llmstxtmcp.dev/mcp`
```json
{
"mcpServers": {
"llms-txt": {
"url": "https://llmstxtmcp.dev/mcp"
}
}
}
```
This page documents a server. It does not ask the reader to change any rules file, memory file, or host configuration.
## Run it yourself
```
npm install && npm start # http://127.0.0.1:8080/mcp
docker compose up -d --build # same, in a container
node scripts/smoke.mjs http://127.0.0.1:8080 llms_index '{}'
```
Built with the MCP server kit (`kit/`): Streamable HTTP MCP plus agent-readable docs (llms.txt, server.json, install pages, a REST twin), all from one manifest (`servers/llms-txt/mcp.factory.json`). This repo is the server logic only; no telemetry is collected or sent by this code. MIT.