{
  "markdown": "# webEmbedding\n\n<!-- mcp-name: io.github.jongko54/web-embedding -->\n\n`webEmbedding` is a source-first website cloning engine for AI coding agents: it captures live pages with Playwright, replays network evidence from HAR artifacts, rebuilds only when direct reuse is blocked, and self-verifies the result.\n\nIt ships as a Skill + MCP server. Instead of asking a model to \"clone this site\" from a screenshot, it inspects the URL, chooses a reuse or rebuild route, captures DOM/runtime HTML/styles/assets/network traces, generates bounded frontend reconstruction artifacts, and checks the output with visual, DOM, computed-style, interaction, and responsive-breakpoint verification.\n\n![webEmbedding Skill and MCP workflow](./docs/assets/webembedding-skill-mcp-card.png)\n\nGitHub listing, social preview, and launch-copy recommendations are in [`docs/github-listing.md`](./docs/github-listing.md).\n\n## Current Status\n\nThe current pipeline is strongest for static and semi-static web pages:\n\n- company, brand, marketing, and documentation pages\n- public landing pages\n- iframe-blocked pages that need capture-based reconstruction\n- responsive page snapshots across desktop, tablet, and mobile\n\nIt is not a full backend or app-logic clone engine. Login-only screens, app-first or native-app-required services, captcha-heavy sites, maps, games, canvas/WebGL-heavy pages, real-time feeds, payments, booking flows, and private server behavior still need separate handling.\n\nOperationally, the repo is now a production-candidate clone engine for URL-based capture and bounded reconstruction: jobs can be queued, network evidence can be replay-audited from HAR artifacts, authenticated dashboard runs can be driven from user-owned browser state, and local gates verify the route corpus, score checks, package contents, and CI wiring. The remaining hard boundary is server-side product behavior, not front-end evidence capture and reconstruction.\n\n## Measured Checkpoints\n\nRecent local benchmark runs from this repo:\n\n| URL | Path | Score |\n| --- | --- | --- |\n| `https://developer.mozilla.org/en-US/` | iframe-blocked bounded rebuild | root `94`, visual `95`, mobile `94`, tablet `94`, breakpoint average `94` |\n| `https://www.mozilla.org/` | bounded rebuild | root `94`, visual `100` |\n| `https://www.python.org` | harder bounded rebuild sample | root `90`, visual `100` |\n| `https://www.example.com` | exact reuse | ready `yes` |\n\nThese are generated by the local self-verify pipeline, not manually assigned ratings.\nThe reproducible commands and score thresholds are tracked in `docs/benchmark-evidence.json`.\nProduction readiness gates are tracked in `docs/production-pipeline-gates.json`.\n\n## Core Features\n\n- Source-first routing:\n  - direct iframe or embed reuse when it is safe and frameable\n  - original preview, export, remix, or source routes when available\n  - bounded rebuild only when exact reuse is unavailable\n- Live browser capture:\n  - DOM snapshot\n  - runtime HTML\n  - full-page screenshot\n  - computed style summaries\n  - CSS analysis\n  - asset inventory\n  - HAR-like network metadata\n  - interaction states and replay traces\n  - storage state export for session-aware flows\n- Blocked-site rebuild:\n  - handles `X-Frame-Options` and CSP-blocked pages by rebuilding from captured evidence\n  - generates reusable frontend reconstruction artifacts from captured page structure\n  - preserves custom tags, shadow-root host structure, and semantic document structure where captured\n- Evidence limitation reporting:\n  - separates directly captured artifacts from inferred or missing evidence in reproduction results and prompts\n  - marks app-gated, auth-gated, and native-app-led surfaces as bounded evidence, with recommendations for user screenshots or authenticated session capture\n- Operational failure classification:\n  - reports typed pipeline action codes such as `network-replay-limited`, `auth-session-missing`, `public-app-gate`, and `canvas-visual-fallback`\n  - exposes HAR/network `replay_readiness` before treating captured network evidence as replay-grade\n- Production pipeline helpers:\n  - filesystem-backed async clone job queue with durable JSON records, worker locks, retry scheduling, cancellation, and manifest annotation\n  - deterministic HAR replay engine for standard HAR, near-HAR, and captured `network/manifest.json` artifacts\n  - authenticated dashboard live corpus runner that accepts user-provided `storage_state_path` or `user_data_dir` outside the repo\n- Self-verification:\n  - screenshot similarity\n  - DOM snapshot similarity\n  - computed-style similarity\n  - hover/focus/click interaction state parity\n  - interaction trace parity\n  - desktop/mobile/tablet breakpoint reports\n- Responsive benchmark support:\n  - primary desktop viewport: `1440x1200`\n  - tablet profile: `768x1024`\n  - mobile profile: `390x844`\n- Repair loop:\n  - bounded self-repair can run when the first scaffold misses the readiness threshold\n\n## Install\n\n### Requirements\n\n- Node.js 18 or newer\n- Python 3.9 or newer\n- Chrome or Chromium available locally for Playwright runtime capture\n\nThe package uses `playwright-core`; it does not download a browser by itself.\n\nInstalling this project adds the `source-first-clone` plugin bundle, the `exact-clone-intake` skill, and the MCP server that exposes the URL inspection, capture, rebuild, and verification tools.\n\n### Install From npm\n\n```bash\nnpm install -g web-embedding\nweb-embedding install\nweb-embedding doctor\n```\n\nClone a public URL after installing:\n\n```bash\nweb-embedding clone \\\n  --url https://developer.mozilla.org/en-US/ \\\n  --output-dir ./.tmp/mdn-clone \\\n  --wait-seconds 2 \\\n  --timeout-seconds 35 \\\n  --breakpoints mobile tablet\n```\n\nIf you already have an older local plugin installed, overwrite it with:\n\n```bash\nweb-embedding install --force\nweb-embedding doctor\n```\n\nYou can also run the installer without a global install:\n\n```bash\nnpx web-embedding install\n```\n\n### Use As An MCP Server\n\nFor MCP clients that can launch npm stdio servers:\n\n```json\n{\n  \"mcpServers\": {\n    \"source-first-clone\": {\n      \"command\": \"npx\",\n      \"args\": [\"-y\", \"web-embedding@latest\", \"mcp\"]\n    }\n  }\n}\n```\n\nFor local smoke testing:\n\n```bash\nnpx web-embedding@latest mcp\n```\n\nThe MCP Registry identity is `io.github.jongko54/web-embedding`; `server.json` and `package.json#mcpName` are kept in sync for registry ownership verification.\n\n### Hosted Apps SDK Intake Endpoint\n\nThe public remote MCP intake endpoint for Apps SDK Developer Mode is:\n\n```text\nhttps://webembedding-jongkos-mcp.vercel.app/mcp\n```\n\nIt exposes low-risk source-first routing tools such as URL inspection, embed candidate discovery, clone-mode classification, and embed snippet generation. Full browser capture, HAR replay, queues, bounded rebuilds, and one-pass clone execution remain local-first through the stdio MCP package.\n\nApps SDK review pages are hosted alongside the endpoint:\n`https://webembedding-jongkos-mcp.vercel.app/privacy.html`,\n`https://webembedding-jongkos-mcp.vercel.app/terms.html`, and\n`https://webembedding-jongkos-mcp.vercel.app/submission.html`.\n\n### Sandboxing And Approvals\n\nwebEmbedding has two different execution boundaries:\n\n- Hosted Apps SDK intake: read-only URL routing and classification only. It accepts absolute `http` and `https` URLs, does not run Playwright, does not read local files, does not use browser profiles or storage state, and does not persist capture artifacts.\n- Local stdio MCP and CLI: full capture, HAR replay, queues, rebuild scaffolds, and self-verify run on the user's machine under the user's local agent and filesystem permissions. Output is written only to caller-provided paths such as `output_dir` or `queue_root`.\n- Authenticated capture: session-aware runs require the caller to intentionally provide a `storage_state_path` or `user_data_dir`. webEmbedding does not collect credentials, perform login bypasses, or treat a public login shell as private app evidence.\n- Access-controlled surfaces: paywalls, captcha flows, private dashboards, payment/checkout/account/admin flows, and native-app-led screens should be blocked, marked `needs_session`, or sent to manual review unless the user has explicit authorization and supplies the needed evidence.\n\nLocal URL entrypoints reject non-HTTP schemes such as `file://` so an agent cannot use clone/capture tools as a local file reader. Telemetry is disabled by default and, when enabled, excludes target URLs, local paths, captured HTML, screenshots, storage state, environment variables, API keys, and command output.\n\n### Agent Marketplaces\n\nThis repository includes marketplace metadata for the two local agent surfaces:\n\n- Codex: `.agents/plugins/marketplace.json` points to `./bundle/source-first-clone`.\n- Claude Code: `.claude-plugin/marketplace.json` points to the same bundle and the bundle includes `.claude-plugin/plugin.json`.\n\nClaude Code users can add the marketplace from GitHub with:\n\n```text\n/plugin marketplace add jongko54/webEmbedding\n/plugin install source-first-clone@webembedding\n```\n\nAI auto-selection expectations and golden prompts live in `docs/ai-distribution.md` and `evals/ai-selection/webembedding-golden-prompts.json`.\n\n### Install From Release\n\n```bash\ncurl -fsSL https://github.com/jongko54/webEmbedding/releases/latest/download/install.sh | bash\n```\n\n### Install From This Checkout\n\n```bash\ngit clone https://github.com/jongko54/webEmbedding.git\ncd webEmbedding\nnpm install\nnode ./bin/web-embedding.mjs install\nnode ./bin/web-embedding.mjs doctor\n```\n\n### Install Into A Temporary Home\n\nUseful for testing without touching your real agent home:\n\n```bash\npython3 python/web_embedding/installer.py install --target-home ./.tmp/home\npython3 python/web_embedding/installer.py doctor --target-home ./.tmp/home\npython3 python/web_embedding/installer.py uninstall --target-home ./.tmp/home\n```\n\n## Opt-in Telemetry\n\nTelemetry is disabled by default. On an interactive first install, `web-embedding install` asks once and defaults to `No`. Non-interactive installs such as CI and `curl | bash` do not prompt. If you opt in, `web-embedding` sends a small anonymous command-completion event to a JSON POST endpoint you control. It does not send target URLs, local paths, captured HTML, screenshots, storage state, environment variables, API keys, or command output.\n\nEnable it during install:\n\n```bash\nweb-embedding install --telemetry --telemetry-endpoint https://your-collector.example/events\n```\n\nOr manage it later:\n\n```bash\nweb-embedding telemetry enable --endpoint https://your-collector.example/events\nweb-embedding telemetry status\nweb-embedding telemetry disable\nweb-embedding telemetry reset-id\n```\n\nEach event contains an anonymous install id, package version, command name, success/failure status, OS/runtime basics, and coarse option flags such as `breakpoint_count` or `install_source`.\n\nEnvironment controls:\n\n```bash\nWEB_EMBEDDING_TELEMETRY=1\nWEB_EMBEDDING_NO_TELEMETRY=1\nWEB_EMBEDDING_TELEMETRY_PROMPT=0\nWEB_EMBEDDING_TELEMETRY_ENDPOINT=https://your-collector.example/events\nWEB_EMBEDDING_TELEMETRY_LOG=./telemetry.jsonl\n```\n\nRun a local/self-hosted JSONL collector:\n\n```bash\nnpm run telemetry:collector -- --host 127.0.0.1 --port 8765 --out ./telemetry.jsonl\nWEB_EMBEDDING_TELEMETRY=1 \\\nWEB_EMBEDDING_TELEMETRY_ENDPOINT=http://127.0.0.1:8765/events \\\nweb-embedding doctor\n```\n\nSummarize collected usage:\n\n```bash\nnpm run telemetry:summarize -- ./telemetry.jsonl\n```\n\nThe summary includes install and clone executions, total command executions, unique anonymous install IDs, command counts, and version counts. See [docs/telemetry.md](docs/telemetry.md) for collector and analyzer details.\n\n## Quick Start\n\nInspect a URL and get route hints:\n\n```bash\nnode ./bin/web-embedding.mjs inspect \\\n  --url https://developer.mozilla.org/en-US/\n```\n\nRun a safe preflight audit before capture or clone:\n\n```bash\nnode ./bin/web-embedding.mjs audit \\\n  --url https://developer.mozilla.org/en-US/\n```\n\nThe audit reports whether the reference is ready for exact/embed reuse, needs local capture, needs an authenticated session, requires manual review, or should be blocked before any browser capture or filesystem output runs.\n\nRun the full clone workflow:\n\n```bash\nnode ./bin/web-embedding.mjs clone \\\n  --url https://developer.mozilla.org/en-US/ \\\n  --output-dir ./.tmp/mdn-clone \\\n  --wait-seconds 2 \\\n  --timeout-seconds 35 \\\n  --breakpoints mobile tablet\n```\n\nRun a lightweight quality benchmark:\n\n```bash\npython3 scripts/check_clone_quality_bench.py \\\n  https://developer.mozilla.org/en-US/ \\\n  --output-root ./.tmp/clone-quality-bench \\\n  --wait-seconds 1 \\\n  --timeout-seconds 35 \\\n  --breakpoints mobile tablet\n```\n\nThe benchmark prints compact rows for root, visual, and breakpoint scores. The full artifacts are written under the output directory.\n\n## CLI Commands\n\n```bash\nnode ./bin/web-embedding.mjs capabilities\nnode ./bin/web-embedding.mjs install\nnode ./bin/web-embedding.mjs doctor\nnode ./bin/web-embedding.mjs uninstall\nnode ./bin/web-embedding.mjs paths\nnode ./bin/web-embedding.mjs telemetry status\n```\n\n```bash\nnode ./bin/web-embedding.mjs inspect --url https://www.mozilla.org/\n```\n\n```bash\nnode ./bin/web-embedding.mjs audit --url https://www.mozilla.org/\n```\n\n```bash\nnode ./bin/web-embedding.mjs capture \\\n  --url https://www.mozilla.org/ \\\n  --output-dir ./.tmp/capture-mozilla \\\n  --breakpoints mobile tablet\n```\n\n```bash\nnode ./bin/web-embedding.mjs reproduce \\\n  --url https://www.mozilla.org/ \\\n  --output-dir ./.tmp/reproduce-mozilla \\\n  --breakpoints mobile tablet\n```\n\n```bash\nnode ./bin/web-embedding.mjs clone \\\n  --url https://www.mozilla.org/ \\\n  --output-dir ./.tmp/clone-mozilla \\\n  --breakpoints mobile tablet\n```\n\n```bash\nnode ./bin/web-embedding.mjs verify \\\n  --reference-bundle ./.tmp/reference/capture.json \\\n  --candidate-bundle ./.tmp/candidate/capture.json\n```\n\n## Output Artifacts\n\nA clone run can produce:\n\n- `capture.json`\n- `pipeline-run-manifest.json`\n- `dom/snapshot.json`\n- `dom/runtime.html`\n- `styles/computed-summary.json`\n- `styles/css-analysis.json`\n- `network/manifest.json`\n- `network/har.json`\n- `network/har-like.json`\n- `network/replay-report.json`\n- `assets/inventory.json`\n- `interactions/states.json`\n- `interactions/trace.json`\n- `screenshots/runtime.png`\n- `session/storage-state.json`\n- `reproduction/plan.json`\n- `reproduction/evidence-limitations.json`\n- `reproduction/rebuild-prompt.txt`\n- `reproduction/rebuild/starter.html`\n- `reproduction/rebuild/starter.css`\n- `reproduction/rebuild/starter.tsx`\n- `reproduction/rebuild/next-app/`\n- `reproduction/self-verify/summary.json`\n- `reproduction/self-verify/renderers/*/verification.json`\n- `reproduction/self-verify/renderers/*/visual-qa.json`\n- `reproduction/self-verify/renderers/*/breakpoints/*-verification.json`\n\n## Quality Benchmark\n\nRun the default small benchmark:\n\n```bash\nnpm run check:clone-bench:local\n```\n\nRun the universal route regression corpus and expectations gate:\n\n```bash\nnpm run check:benchmark-routes:local\n```\n\nRun a lightweight clone score gate:\n\n```bash\nnpm run check:clone-score-gate:local\n```\n\nValidate the committed benchmark evidence manifest:\n\n```bash\nnpm run check:benchmark-evidence:local\n```\n\nValidate production pipeline gates:\n\n```bash\nnpm run check:production-readiness:local\n```\n\nRun the operational smokes individually:\n\n```bash\nnpm run check:job-queue:local\nnpm run check:har-replay:local\nnpm run check:authenticated-corpus:local\n```\n\nClassify failure/action codes from a route report:\n\n```bash\nnpm run classify:pipeline-failures -- --report ./.tmp/universal-route-benchmark/universal-route-report.json\n```\n\nFind low-scoring persisted benchmark artifacts:\n\n```bash\nnpm run summarize:benchmark-scores -- --root ./.tmp --min-score 60 --max-score 70\n```\n\nRun specific URLs:\n\n```bash\npython3 scripts/check_clone_quality_bench.py \\\n  https://www.example.com \\\n  https://www.mozilla.org/ \\\n  --no-breakpoints\n```\n\nRun a responsive benchmark:\n\n```bash\npython3 scripts/check_clone_quality_bench.py \\\n  https://developer.mozilla.org/en-US/ \\\n  --breakpoints mobile tablet\n```\n\n## Development Checks\n\n```bash\npython3 -m py_compile \\\n  bundle/source-first-clone/mcp/source_first_clone/*.py \\\n  scripts/check_integration_smoke.py \\\n  scripts/check_clone_quality_bench.py\n```\n\n```bash\nnpm run check:integration:local\n```\n\n```bash\ngit diff --check\n```\n\n## Repo Layout\n\n- `bundle/source-first-clone`\n  Installed plugin bundle, MCP server, and exact-clone intake skill.\n- `bundle/source-first-clone/mcp/source_first_clone`\n  Capture, planning, rebuild, repair, and verification engine.\n- `bin/web-embedding.mjs`\n  Node CLI wrapper.\n- `python/web_embedding/installer.py`\n  Shared installer and command dispatcher.\n- `scripts/check_clone_quality_bench.py`\n  URL clone quality benchmark helper.\n- `scripts/benchmark_routes.py`\n  Universal route/capture-depth regression benchmark helper.\n- `scripts/check_benchmark_report.py`\n  Benchmark expectation validator for exact, minimum, and contains-style checks.\n- `scripts/check_benchmark_evidence.py`\n  Benchmark evidence manifest validator.\n- `scripts/check_job_queue_smoke.py`\n  Filesystem async clone job queue smoke test.\n- `scripts/check_har_replay_smoke.py`\n  Deterministic HAR replay engine smoke test.\n- `scripts/benchmark_authenticated_corpus.py`\n  User-provided authenticated dashboard corpus runner.\n- `scripts/summarize_benchmark_scores.py`\n  Utility for finding low or high scoring persisted benchmark artifacts under an output root.\n- `scripts/classify_pipeline_failures.py`\n  Operational failure/action taxonomy summarizer for reports and capture artifacts.\n- `scripts/check_production_readiness.py`\n  Production readiness gate validator for corpus, failure taxonomy, CI wiring, and policy docs.\n- `scripts/check_integration_smoke.py`\n  Release, install, and URL-only clone smoke test.\n- `scripts/release_bundle.py`\n  Release artifact builder.\n- `docs/`\n  Architecture notes and universal benchmark documentation.\n\n## Positioning\n\nThe strongest claim for this project is:\n\n> A source-first website cloning engine that combines Playwright capture, HAR replay, MCP tools, and self-verification to rebuild iframe-blocked public pages with reproducible visual, DOM, style, interaction, and responsive scores.\n\nAvoid treating the output as a legal or ownership bypass. The engine can reconstruct public page structure, but permission, licensing, and acceptable use still matter.\n\n## License\n\nMIT\n",
  "bytes": 18467,
  "sha": "718d74767a2ac4bb803083c59795614c9abc925403746137257abb3d78040f73",
  "repo_slug": "jongko54/webembedding",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_jongko54_web_embedding_ad6cd7e8/readme"
}