\n\n# Purl\n\n**Save anything. Ask questions. Get answers.**\n\n**Live preview:** [https://purl.nublson.com](https://purl.nublson.com)\n\nPurl is an AI-powered read-it-later app and personal knowledge base. You paste URLs (or upload files): web pages, PDFs, YouTube videos, and audio. Purl ingests the content, stores chunked text with vector embeddings, and answers questions by searching what you saved — optionally scoped with `@` mentions to specific items.\n\n**Plans:** **Free**, **Pro**, and **BYOK** are enforced server-side (see [`docs/commercial-model.md`](docs/commercial-model.md)). New signups get a **7-day Pro trial** (no card required). **Stripe Checkout** handles the one-time Pro payment; **webhooks** sync status to Postgres.\n\nThe product goal: one place to stash material you care about, then query it later with citations instead of digging through bookmarks.\n\n## Plans (summary)\n\n| Feature | Free | Pro ($39 one-time) | BYOK (free) |\n| ----------------------------- | ---- | ------------------ | --------------- |\n| Save links (100 lifetime cap) | Yes | Unlimited | Unlimited |\n| Full-text search | Yes | Yes | Yes |\n| AI extraction & embeddings | No | Yes (150/mo) | Yes (unlimited) |\n| Semantic search | No | Yes | Yes |\n| PDF/audio **upload** | No | Yes | Yes |\n| AI chat | No | 300 msg/mo | Unlimited |\n\nExact limits are in [`docs/commercial-model.md`](docs/commercial-model.md).\n\n## Implemented today\n\n- **Marketing site** — Landing page with hero, features, supported content types, pricing section, and FAQ.\n- **Authentication** — Email/password (and related flows) via [Better Auth](https://www.better-auth.com/); optional email verification through [Resend](https://resend.com/).\n- **Save & organize**\n - Add items by URL with automatic content-type detection (web, PDF, YouTube, audio).\n - **File upload** for PDF and audio (size limits enforced server-side).\n - Links grouped by relative time (e.g. Today, This Week, Last Month).\n - Preview metadata (title, description, favicon, thumbnail where available).\n- **Ingestion pipeline** — Fetches or extracts text (including transcripts for YouTube/audio), chunks it, embeds via **Vercel AI Gateway** (`openai/text-embedding-3-small`), stores in Postgres with **pgvector**; tracks per-link ingest status (pending, processing, completed, failed, skipped for edge cases like heavy SPAs).\n- **AI providers** — **Vercel AI Gateway** for streaming chat (Claude) and **embeddings** (`openai/text-embedding-3-small`). **OpenAI** directly for **Whisper** transcription only (`OPENAI_API_KEY`). Keys live in server environment variables only.\n- **AI Gateway observability** — Chat, ingest embeddings, and the chat tool’s semantic search send `providerOptions.gateway` with the signed-in **`user`** id and **`tags`** so the [Vercel AI](https://vercel.com/docs/ai-gateway) dashboard can filter spend and usage by person and surface (`feature:chat`, `env:…` from `VERCEL_ENV` / `NODE_ENV`; `feature:ingest` on save pipelines; `feature:semantic-search` when the model runs vector search over saved chunks).\n- **Hardened outbound fetch** — Server-side `safeFetch` with optional proxy/DNS controls (see `AGENTS.md`). For reliable **YouTube transcripts on Vercel**, configure [`SAFE_OUTBOUND_HTTP_PROXY`](docs/production-outbound-proxy.md) in production.\n- **Realtime list sync** — Supabase Realtime so saves and updates propagate across tabs/devices quickly.\n- **AI chat**\n - Streaming replies (**Anthropic Claude**) with tool use: list saved items (filters by date/type) and search over stored chunks.\n - **`@` mentions** to focus the model on specific saved links; mentions persist on messages.\n - Multiple chats, titles, and message history stored in the database.\n- **Link actions** — Open original, copy URL, edit metadata, re-ingest, delete, add to chat context from the list.\n- **Operational extras** — Optional Upstash-backed API rate limiting, optional Sentry, Vitest coverage for critical paths.\n- **PWA (installable app)** — [Web App Manifest](public/manifest.json) plus a [Serwist](https://serwist.pages.dev/) service worker ([`src/app/sw.ts`](src/app/sw.ts)) that builds to **`public/sw.js`** (generated on `pnpm build`, gitignored). Enables **Install** in Chrome/Edge and similar where the platform supports it, with runtime caching via Serwist's Next.js defaults and a static offline shell at [`/~offline`](src/app/~offline/page.tsx). **Serwist is disabled in `pnpm dev`** to avoid service-worker cache surprises during development — use **`pnpm build && pnpm start`** (or your production URL) to exercise installability and the SW.\n\n## Ingestion flow\n\nSaving a link is **synchronous** through metadata resolution and the database row; **heavy work runs afterward** so the API can return quickly.\n\n1. **Input** — `POST /api/links` with a URL, or `POST /api/upload` with a PDF/audio file (files go to Supabase Storage; the `Link` stores the public URL).\n2. **Classify & decorate** — Server-side [`detectContentType`](src/lib/server-detect-content-type.ts) (SSRF-safe `HEAD` / sniff) plus [`scrapeLinkMetadata`](src/lib/links.ts) (Open Graph HTML, PDF `Content-Disposition` / size, YouTube oEmbed). Duplicates of the same URL **refresh** metadata and reset ingestion.\n3. **Persist** — A `Link` row is created (default **`PENDING`**) with title, favicon, thumbnail, domain, and `contentType` (`WEB`, `PDF`, `YOUTUBE`, or `AUDIO`).\n4. **Schedule** — [`prepareIngestForLink`](src/lib/links.ts) enforces plan limits, then uses Next.js [`after()`](https://nextjs.org/docs/app/api-reference/functions/after) to run the right handler: [`ingestWeb`](src/lib/ingest-web.ts), [`ingestPdf`](src/lib/ingest-pdf.ts), [`ingestYoutube`](src/lib/ingest-youtube.ts), or [`ingestAudio`](src/lib/ingest-audio.ts). Free accounts skip extraction (metadata-only; ingest **`SKIPPED`**).\n5. **Pipeline** (each handler) — Set **`PROCESSING`** → fetch or extract plain text → split into chunks (with a synthetic **metadata** chunk first) → **Vercel AI Gateway** embeddings (`openai/text-embedding-3-small`) → replace `LinkContent` rows and attach **pgvector** values → **`COMPLETED`**. Failures set `ingestFailureReason` (`SCRAPE_FAILED`, `LINK_NOT_FOUND`, `OTHER`, etc.) alongside **`FAILED`**. **Re-ingest** reuses the same pipeline without re-scraping listing metadata.\n\n**Web pages (`WEB`).** Article-style HTML is fetched with [`safeFetch`](src/lib/safe-outbound-fetch.ts), parsed in **jsdom**, and the main content is extracted with Mozilla's [**Readability**](https://github.com/mozilla/readability) ([`scrapeWebContent`](src/lib/web-scraper.ts)). That matches how Firefox's reader mode chooses \"the article,\" but it is **not universal**: many **SPAs** and other **client-rendered** sites return a thin HTML shell to crawlers, so Readability finds little or nothing and ingest may **`FAIL`**. A small set of hosts that need a full browser are rejected early (`UnsupportedSpaError` → ingest **`SKIPPED`**).\n\nRealtime subscribers get updates when ingestion finishes via [`notifyLinksAfterIngest`](src/lib/notify-links-after-ingest.ts) (which calls [`broadcastLinksChanged`](src/lib/realtime-broadcast.ts)).\n\n```mermaid\nflowchart TB\n subgraph save[\"Save path (responds to client)\"]\n A([\"URL or file upload\"]) --> B[\"/api/links or /api/upload\"]\n B --> C[\"detectContentType + scrapeLinkMetadata (safeFetch)\"]\n C --> D[\"Insert Link — ingestStatus PENDING\"]\n D --> E[\"after() → prepareIngestForLink by contentType\"]\n end\n\n subgraph work[\"Background ingest\"]\n E --> F[\"ingestStatus PROCESSING\"]\n F --> G{\"Extract text\"}\n G --> W[\"WEB — jsdom + Mozilla Readability\"]\n G --> P[\"PDF — page text\"]\n G --> Y[\"YOUTUBE — transcript\"]\n G --> A2[\"AUDIO — transcription\"]\n W --> H[\"Chunk + metadata header\"]\n P --> H\n Y --> H\n A2 --> H\n H --> I[\"Gateway embeddings\"]\n I --> J[\"Write LinkContent + pgvector\"]\n J --> K{\"Outcome\"}\n K --> K1[\"COMPLETED\"]\n K --> K2[\"FAILED (+ ingestFailureReason)\"]\n K --> K3[\"SKIPPED (known SPA hosts)\"]\n end\n\n K1 --> R[\"Realtime: list refresh\"]\n K2 --> R\n K3 --> R\n```\n\n## Not implemented yet\n\nThese are called out explicitly because the repo is going public:\n\n- **Settings breadth** — Settings include account deletion; broader account preferences (profile edits, password change, notification settings, etc.) are not implemented yet.\n\n**Marketing vs. product:** The landing page copy mentions ideas such as **collections** and a **weekly digest**. Those are **not** built in the current schema or app — treat them as roadmap, not shipped features.\n\n## Tech stack\n\n- **Web:** Next.js (App Router), React, TypeScript\n- **UI:** Tailwind CSS, shadcn/ui\n- **Auth:** Better Auth\n- **Database:** PostgreSQL + Prisma (with vector column for embeddings)\n- **AI:** **Vercel AI Gateway** (Claude chat + OpenAI embeddings through gateway) and **OpenAI** (Whisper transcription direct) via the Vercel AI SDK — server environment variables\n- **Email (optional in dev):** Resend for verification emails\n- **Realtime:** Supabase client (anon + service role on server)\n- **PWA:** [Serwist](https://serwist.pages.dev/) (`@serwist/next`), web manifest + precache / offline fallback\n\n## CI / GitHub Actions\n\nAutomation lives under [`.github/workflows/`](.github/workflows/). Every PR and manual release is gated by these pipelines.\n\n### PR checks — [`pr-checks.yml`](.github/workflows/pr-checks.yml)\n\nRuns on **`pull_request`** to **`develop`** and **`main`**: **Setup & validation** → **Prisma** (generate client + type fixes) → **Lint** and **type check** (in parallel) → **Tests** and **production build** (in parallel, after lint and type check pass). Concurrency is per-PR so new pushes cancel stale runs.\n\n
\n \n
\n\n### Release — [`release.yml`](.github/workflows/release.yml)\n\nRuns on **`workflow_dispatch`** (manual): **Merge `develop` into `main`**, then **build validation** so production is only promoted after a green build.\n\n
\n \n
\n\n## Security\n\nPurl is built around **untrusted input** (arbitrary URLs and uploaded files). A few layers matter in production:\n\n- **SSRF-aware outbound fetches** — User-supplied URLs are not passed to raw `fetch`. Ingest, OG/thumbnail probes, PDF fetch, content-type sniffing, and similar paths go through [`safeFetch`](src/lib/safe-outbound-fetch.ts): HTTP(S) only, blocked private/link-local/reserved targets, redirect handling with per-hop host checks, DNS resolution pinned before connect (mitigates classic DNS rebinding against the pre-check), optional response size caps (e.g. PDF proxy). Optional **egress proxy** and custom DNS servers are documented in [`AGENTS.md`](AGENTS.md).\n- **Authentication & route gating** — [Better Auth](https://www.better-auth.com/) sessions; Next.js [`proxy`](src/proxy.ts) redirects unauthenticated users away from private routes and can require **email verification** before app access.\n- **API authorization** — Sensitive routes (`/api/chat`, `/api/links`, `/api/upload`, chats, etc.) resolve the session server-side and scope work to the signed-in user (e.g. chat mention IDs are validated against ownership).\n- **Server-managed AI keys** — Chat and embeddings route through **Vercel AI Gateway** (`AI_GATEWAY_API_KEY` or `VERCEL_OIDC_TOKEN` after `vercel env pull`). **Whisper** transcription still calls **OpenAI** directly via `OPENAI_API_KEY`. These keys are never exposed to the client.\n- **Rate limiting** — When `UPSTASH_REDIS_REST_URL` and `UPSTASH_REDIS_REST_TOKEN` are set, the proxy applies per-IP limits to **`/api/auth/*`**, **`POST /api/chat`**, **`POST /api/links`**, and **`POST /api/upload`** (see [`proxy-rate-limit.ts`](src/lib/proxy-rate-limit.ts)). Without Upstash, limits are disabled — fine locally, not ideal for production.\n- **Secrets & client exposure** — `SUPABASE_SERVICE_ROLE_KEY` and similar values are server-only. The browser uses the Supabase **anon** key for Realtime only; `.env` stays gitignored.\n- **Upload bounds** — Audio uploads enforce a maximum size server-side; PDF proxy streaming is capped (see `safe-outbound-fetch` / upload limits in code).\n\n**Reporting a vulnerability:** use [GitHub Security Advisories](https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing-information-about-vulnerabilities/privately-reporting-a-security-vulnerability) for this repository so details stay private until patched.\n\n## Stripe (one-time payment)\n\nBilling uses **Stripe Checkout** (`POST /api/billing/checkout`) for the one-time Pro payment, **Customer Portal** (`POST /api/billing/portal`), and **webhooks** (`POST /api/billing/webhook`). Plan entitlements and usage limits are enforced in-app from Postgres (see [`src/lib/entitlements.ts`](src/lib/entitlements.ts)); webhooks keep the plan row in sync.\n\n**Env (see `.env.example`):** `STRIPE_SECRET_KEY`, `STRIPE_PUBLISHABLE_KEY` (server-only), `STRIPE_WEBHOOK_SECRET`, `STRIPE_PRICE_PRO` (one-time price ID), optional `ADMIN_TOKEN` for `POST /api/admin/grants`.\n\n**Local webhook testing:** create a one-time Pro price in [Stripe Test mode](https://dashboard.stripe.com/test/products), put the price ID in `.env`, then:\n\n```bash\nstripe listen --forward-to localhost:3000/api/billing/webhook\n```\n\nCopy the CLI signing secret into `STRIPE_WEBHOOK_SECRET` for that shell session. Trigger flows with `stripe trigger checkout.session.completed` (and exercise checkout from the app). **Go-live:** recreate the product and a Live webhook endpoint; rotate keys and webhook secret per environment.\n\n## Setup (local development)\n\n### Prerequisites\n\n- **Node.js:** recent LTS\n- **Package manager:** `pnpm` (this repo includes `pnpm-lock.yaml`)\n- **Postgres:** local or hosted (Supabase works well with pgvector)\n\n### 1) Install dependencies\n\n```bash\npnpm install\n```\n\n### 2) Configure environment variables\n\nCreate a `.env` file in the repo root. See `.env.example` for the full list; minimum for core behavior:\n\n```bash\nDATABASE_URL=\"postgresql://USER:PASSWORD@HOST:5432/DBNAME\"\n\n# AI (server-side): gateway for chat + embeddings; OpenAI key for Whisper only\nAI_GATEWAY_API_KEY=\"...\" # or use VERCEL_OIDC_TOKEN from `vercel env pull`\nOPENAI_API_KEY=\"sk-proj-...\"\n\n# Supabase Realtime — cross-device instant link list sync (same project as Postgres)\nNEXT_PUBLIC_SUPABASE_URL=\"https://YOUR_PROJECT.supabase.co\"\nNEXT_PUBLIC_SUPABASE_ANON_KEY=\"eyJ...\"\nSUPABASE_SERVICE_ROLE_KEY=\"eyJ...\"\n\n# Optional (used for email verification on signup)\nRESEND_API_KEY=\"re_...\"\nRESEND_FROM=\"Purl \"\n```\n\nNotes:\n\n- **`DATABASE_URL`** is required (Prisma + Better Auth).\n- **`AI_GATEWAY_API_KEY`** (or **`VERCEL_OIDC_TOKEN`** on Vercel / after `vercel env pull`) is required for **chat** and **embeddings** (ingest + semantic search) via AI Gateway. Enable **AI Gateway** in the Vercel project settings for OIDC-based auth. Optional: configure **per-user** limits in the project AI Gateway settings; the app passes the Better Auth user id on gateway calls.\n- **`OPENAI_API_KEY`** is required for **Whisper** transcription (audio ingest / URLs). Omit only if you do not use audio transcription.\n- **Supabase** env vars are required for realtime link list sync. Use **Project Settings → API** in the Supabase dashboard. The service role key must stay server-only.\n- **Resend** is optional for local dev: if `RESEND_API_KEY` is not set, signup can still work, but verification emails will not send.\n- **Better Auth** secrets and URLs are in `.env.example` — copy those keys for a working auth setup.\n\n### 3) Run database migrations\n\n```bash\npnpm prisma migrate dev\n```\n\n### 4) Generate Prisma client (if needed)\n\n```bash\npnpm prisma generate\n```\n\n### 5) Start the dev server\n\n```bash\npnpm dev\n```\n\nOpen `http://localhost:3000`.\n\n**PWA / install:** With `pnpm dev`, the service worker is not active. After a production build, `public/sw.js` exists locally; run **`pnpm start`** and open the app in Chromium to use **Install** or to test offline navigation to `/~offline`.\n\n## Testing\n\nTests use [Vitest](https://vitest.dev/) and focus on critical logic (formatters, link grouping, auth routing, API behavior, ingest pipeline). They intentionally avoid shallow UI-only wrappers.\n\n```bash\npnpm test # run once\npnpm test:watch # watch mode\n```\n\n## Useful commands\n\n```bash\npnpm lint\npnpm build\npnpm start\npnpm test\n```\n\nMore contributor notes (Prisma, Sentry, outbound proxy env): see [`AGENTS.md`](AGENTS.md).\n",
"bytes": 17131,
"sha": "12dba4c34095adc08958ce65b84a806c6b468d43e881a90c4a639c53cd8a9e7e",
"repo_slug": "nublson/purl",
"fonte": "repo",
"truncated": false,
"api": "https://agentalog.com/api/listings/mcp_io_github_nublson_purl_f77703e9/readme"
}
![\"Purl](\"thumbnail.jpeg\")
\n![\"GitHub](\"prCheckPipeline.png\")
\n![\"GitHub](\"releasePipeline.png\")
\n