PageMap
Structured web intelligence for AI agents. 5x fewer tokens than Playwright/Firecrawl, 84.7% accuracy
Open source Open in the app JSON README (API)
About
Structured web intelligence for AI agents. 5x fewer tokens than Playwright/Firecrawl, 84.7% accuracy
Details
- Kind
- MCP servers
- Topic
- Web search, scraping & browser
- Publisher
- retio-ai
- Origin
- official
- Category
- ferramentas
- Transport
- local
- Version
- 0.7.3
- Stars
- 36
- Forks
- 5
- Open pull requests
- 5
- Last push
- 2026-05-21T08:23:20Z
- Repository state
- ativo
- Language
- Python
- License
- NOASSERTION
- Added
- 2026-08-29 03:02:12
- Updated
- 2026-08-29 03:02:12
- Origin id
io.github.Retio-ai/pagemap
README
<!-- mcp-name: io.github.Retio-ai/pagemap -->
# PageMap
PageMap converts raw HTML (100K+ tokens) into structured, AI-readable page maps (2-5K tokens) — a **97% token reduction**. It works as an MCP server, Python SDK, and CLI, supporting 16 page types and 30+ e-commerce sites. Agents can read, click, type, and navigate any web page.
> *"Give your agent eyes and hands on the web."*
[](https://github.com/Retio-ai/Retio-pagemap/actions/workflows/ci.yml)
[](https://pypi.org/project/retio-pagemap/)
[](https://pypi.org/project/retio-pagemap/)
[](https://www.gnu.org/licenses/agpl-3.0)
[](https://hub.docker.com/r/retio1001/pagemap)
[](https://github.com/punkpeye/awesome-mcp-servers)
---
<!-- ============================================================ -->
<!-- HUMAN GUIDE -->
<!-- ============================================================ -->
## Why PageMap?
Playwright MCP dumps 50-540KB accessibility snapshots per page, overflowing context windows after 2-3 navigations. Firecrawl and Jina convert HTML to markdown — read-only, no interaction.
PageMap gives your agent a **compressed, actionable** view of any web page:
| | PageMap | Playwright MCP | Firecrawl | Jina Reader |
|--|:------:|:---------:|:-----------:|:--------:|
| **Tokens / page** | **2-5K** | 6-50K | 10-50K | 10-50K |
| **Interaction** | **click / type / select / hover** | Raw tree parsing | Read-only | Read-only |
| **Multi-page sessions** | **Unlimited** | Breaks at 2-3 pages | N/A | N/A |
| **Task success (94 tasks)** | **84.7%** | 61.5% | 64.5% | 57.8% |
| **Avg tokens / task** | **2,710** | 13,737 | 13,888 | 11,424 |
| **Cost / 94 tasks** | **$1.06** | $4.09 | $3.98 | $2.26 |
> Benchmarked across 11 e-commerce sites, 94 static tasks, 7 conditions. 8,100+ tests passing.
---
## Quick Start
Chromium is auto-installed on first use — no manual `playwright install` needed.
### Install
```bash
pip install retio-pagemap
```
### MCP Client Config
Add to Claude Code, Cursor, Windsurf, or Claude Desktop:
```json
{
"mcpServers": {
"pagemap": {
"command": "uvx",
"args": ["retio-pagemap"]
}
}
}
```
> **Claude Desktop (macOS)**: Use the absolute path to `uvx` — run `which uvx` (e.g. `/opt/homebrew/bin/uvx`).
> **VS Code (Copilot)**: Use `"servers"` instead of `"mcpServers"` in `.vscode/mcp.json`.
### Docker
```bash
docker run -p 8000:8000 retio1001/pagemap --transport http
```
---
## Features
### 13 MCP Tools — Read + Interact
Not just reading — your agent can click buttons, fill forms, select options, manage tabs, and navigate across pages. 13 tools cover the full browsing workflow:
`get_page_map` · `execute_action` · `fill_form` · `scroll_page` · `wait_for` · `take_screenshot` · `get_page_state` · `navigate_back` · `batch_get_page_map` · `open_tab` · `switch_tab` · `list_tabs` · `close_tab`
### 16 Page Types, Auto-Detected
PageMap automatically classifies pages and applies optimized extraction for each type:
`product_detail` · `listing` · `search_results` · `article` · `news` · `video` · `login` · `form` · `checkout` · `dashboard` · `help_faq` · `settings` · `error` · `documentation` · `landing` · `blocked`
### E-Commerce Deep Coverage
Built-in support for **30+ major e-commerce sites** across 4 tiers:
- **Global mega-platforms** — Amazon, eBay, AliExpress, SHEIN, Walmart, Rakuten
- **Global fashion** — Zara, H&M, Nike, Uniqlo, ASOS, Zalando, SSENSE, Farfetch, COS
- **Korea** — Coupang, Naver Shopping, Musinsa, 29CM, W Concept, SSG, 11st
- **Japan/China** — ZOZO, Tmall, JD.com, Taobao
Structured extraction of prices, options (size/color), ratings, availability — with automatic cookie consent handling and login barrier detection.
### Smart Recovery
PageMap detects problems and tells your agent what to do:
- **Barrier detection** — Login required? Bot blocked? Out of stock? Age verification? Popup overlay? PageMap adds a `barrier` field with the diagnosis and suggested next steps
- **Cookie consent auto-dismiss** — 7 CMP providers auto-detected (Cookiebot, OneTrust, TrustArc, Didomi, Quantcast, Usercentrics, generic fallback). 5-tier dismiss cascade: CMP JS API → Reject → Accept → Dismiss → Close symbol. GDPR reject-first default policy
- **Popup overlay detection** — AX tree `role="dialog"` + HTML regex 2-phase detection. Promotional popups (newsletter, exit-intent) auto-dismissed
- **Bot detection awareness** — Detects Cloudflare, Turnstile, reCAPTCHA, hCaptcha, and Akamai. Reports the provider and suggests wait/retry strategies
- **Stale ref recovery** — When DOM changes invalidate refs, PageMap returns clear guidance to re-fetch
### Content Intelligence
- **8 JSON-LD schemas** — Product, NewsArticle, VideoObject, FAQPage, Event, LocalBusiness, BreadcrumbList, and ItemList
- **Metadata extraction** — Prices, ratings, reviews, descriptions, images from structured data and DOM fallbacks
- **2-layer caching** — Cache hit (~10ms), content refresh (~500ms), full rebuild (~1.5s). Diff-based updates for unchanged sections
- **Delta evidence packet output** - Optional `to_delta_packet()` serializer emits digest-bound evidence units, claim candidates, provenance, and authority flags for downstream memory/review systems without changing the default MCP output
### 10 Languages
Locale auto-detected from URL. Token budgets adjusted for CJK scripts.
| Language | Locale | Language | Locale |
|----------|:------:|----------|:------:|
| English | `en` | Chinese | `zh` |
| Korean | `ko` | Spanish | `es` |
| Japanese | `ja` | Italian | `it` |
| French | `fr` | Portuguese | `pt` |
| German | `de` | Dutch | `nl` |
---
## Deployment
### Local (STDIO)
Default mode. Runs as a local MCP server — no server setup needed.
```bash
retio-pagemap
```
### Docker
```bash
docker run -p 8000:8000 retio1001/pagemap --transport http
```
Multi-architecture images (amd64/arm64) available on [Docker Hub](https://hub.docker.com/r/retio1001/pagemap) and GitHub Container Registry.
---
## Python API
```python
import asyncio
from pagemap.browser_session import BrowserSession
from pagemap.delta_serializer import to_delta_packet
from pagemap.page_map_builder import build_page_map_live
from pagemap.serializer import to_agent_prompt, to_json
async def main():
async with BrowserSession() as session:
page_map = await build_page_map_live(session, "https://example.com/product/123")
print(to_agent_prompt(page_map)) # Agent-optimized text format
print(to_json(page_map)) # Structured JSON
print(to_delta_packet(page_map)) # Digest-bound evidence packet
print(page_map.page_type) # "product_detail"
print(page_map.interactables) # [Interactable(ref=1, role="button", ...)]
print(page_map.metadata) # {"name": "...", "price": "..."}
asyncio.run(main())
```
For offline processing (no browser):
```python
from pagemap.page_map_builder import build_page_map_offline
page_map = build_page_map_offline(open("page.html").read(), url="https://example.com/product/123")
```
---
## Security
PageMap treats all web content as untrusted input:
- **SSRF defense** — Multi-layer protection against server-side request forgery
- **Prompt injection defense** — Content boundaries, role-prefix stripping, suspicious content flagging
- **robots.txt compliance** — RFC 9309 compliant. `--ignore-robots` opt-out flag
- **Resource guards** — DOM node limit, HTML size limit, response size limit
- **Session isolation** — Each session has independent cookies and storage, automatically cleaned up
**Local development**: Private IPs are blocked by default. Use `--allow-local` or `PAGEMAP_ALLOW_LOCAL=1`.
### Disclaimer
Users are responsible for complying with the terms of service of target websites and all applicable laws when using PageMap.
---
## Troubleshooting
**"spawn uvx ENOENT" (Claude Desktop on macOS)** — Claude Desktop does not inherit your shell PATH. Run `which uvx` and use the absolute path in your config.
**First page takes a long time** — Chromium cold start takes ~10-30s on first navigation. Subsequent pages load in 1-3 seconds.
**Localhost blocked** — Use `--allow-local` flag or set `PAGEMAP_ALLOW_LOCAL=1`.
**Chromium not found** — Run `pip install retio-pagemap && playwright install chromium` to install manually.
---
## Requirements
- Python 3.11+
- Chromium (auto-installed on first use)
## Community
Have a question or idea? Join the conversation in [GitHub Discussions](https://github.com/Retio-ai/Retio-pagemap/discussions).
## Development
[](https://codespaces.new/Retio-ai/Retio-pagemap?quickstart=1)
```bash
git clone https://github.com/Retio-ai/Retio-pagemap.git
cd Retio-pagemap
uv sync --group dev
playwright install chromium
uv run pytest --tb=short -q
```
## Pricing
**Local (STDIO)** — Free forever. Self-hosted, open source under AGPL-3.0.
**Cloud API** — Hosted multi-tenant server with auth, rate limiting, and credit-based billing. Contact **retio1001@retio.ai** for access.
## License
AGPL-3.0-only — see [LICENSE](LICENSE) for the full text.
For commercial licensing options, contact **retio1001@retio.ai**.
---
<!-- ============================================================ -->
<!-- AGENT REFERENCE -->
<!-- ============================================================ -->
## For Agents
*This section is written for AI agents using PageMap as an MCP tool.*
### Tools
| Tool | When to use |
|------|-------------|
| `get_page_map` | **Start here.** Navigate to a URL and get a full structured map with numbered refs. |
| `execute_action` | Click, type, select, or hover using a ref number from the last `get_page_map`. |
| `fill_form` | Fill multiple form fields in one call. More efficient than sequential `execute_action` calls. |
| `get_page_state` | Check current URL and title without a full rebuild. Use after actions that may navigate. |
| `scroll_page` | Scroll to reveal lazy-loaded content before calling `get_page_map` again. |
| `wait_for` | Wait for dynamic content to appear (e.g. after a search or form submit). |
| `take_screenshot` | Capture the visual state when the PageMap alone is ambiguous. |
| `navigate_back` | Go back one step in browser history. |
| `open_tab` | Open a new browser tab and navigate to a URL. |
| `switch_tab` | Switch to a different open tab by index. |
| `list_tabs` | List all open tabs with their URLs and titles. |
| `close_tab` | Close a tab by index. |
| `batch_get_page_map` | Fetch multiple URLs in parallel. Use for comparison tasks. |
### Output Format
```yaml
URL: https://example.com/product/123
Title: Product Name
Type: product_detail # auto-detected page type
## Actions
[1] button: Add to cart (click)
[2] select: Size (select) — options: S, M, L, XL
[3] link: See all reviews (click)
...
## Info
Price: $49.99
Rating: 4.5 / 5 (128 reviews)
Description: ...
## Images
[1] https://cdn.example.com/product.jpg
## Meta
Tokens: ~1,800 | Interactables: 24 | Generation: 380ms
```
- **`## Actions`** — Every interactive element on the page with a stable `ref` number.
- **`## Info`** — Key page content extracted from HTML: prices, titles, ratings, descriptions.
- **`## Images`** — Product/content image URLs.
- **`## Meta`** — Token count, interactable count, generation time.
### Barrier Detection
When PageMap encounters a page-level obstacle, it includes a `barrier` field in the response:
```yaml
State:
barrier: login_required
barrier_hint: "Login form detected with email + password fields. Use fill_form to authenticate."
```
Possible barriers: `cookie_consent`, `login_required`, `bot_blocked`, `out_of_stock`, `empty_results`, `error_page`, `age_verification`, `region_restricted`, `popup_overlay`.
**When you see a barrier:** follow the `barrier_hint` guidance. For `bot_blocked`, wait and retry. For `login_required`, use `fill_form` with credentials.
### Ref Lifecycle
Refs are assigned by `get_page_map` and remain valid until the page state changes.
**Refs are invalidated when:**
- The page navigates to a new URL
- A DOM mutation occurs (modal opens, SPA navigation, accordion toggles)
- `execute_action` causes a page-level change
**When you get a stale ref error:** call `get_page_map` again to get fresh refs before retrying.
### Token Budget Behavior
When a page exceeds the token budget, content is pruned in this order:
1. Navigation menus, footers, sidebars removed first
2. Secondary body content trimmed
3. `## Actions` and `## Info` are always preserved
If key content seems missing, try `scroll_page` to load lazy content, then `get_page_map` again.
### Recommended Workflow
```
1. get_page_map(url) → read Actions + Info, pick refs
2. execute_action(ref, ...) → interact
3. get_page_state() → confirm navigation occurred
4. get_page_map(new_url) → get fresh refs for next step
```
For pages with dynamic content (search results, filters):
```
1. get_page_map(url)
2. execute_action(ref, "click") → trigger search/filter
3. wait_for(text="results") → wait for content
4. get_page_map(url) → get updated map
```
### Known Limitations
- **Login-gated pages** — PageMap does not manage sessions or cookies. Authentication must be handled externally.
- **Heavy bot detection** (Cloudflare, Akamai) — May block automated access. PageMap detects the provider and suggests strategies, but cannot bypass active bot mitigation.
- **Private network access** — Blocked by default. Requires `--allow-local` flag.
- **iframes** — Cross-origin iframes are not accessible due to browser security policies.
---
*PageMap — Structured Web Intelligence for the Agent Era.*