{
  "markdown": "# News Buddy\n\n[![Daily News Digest](https://github.com/Harshagarwal06/buddy_agent/actions/workflows/daily-digest.yml/badge.svg)](https://github.com/Harshagarwal06/buddy_agent/actions/workflows/daily-digest.yml)\n[![CI](https://github.com/Harshagarwal06/buddy_agent/actions/workflows/ci.yml/badge.svg)](https://github.com/Harshagarwal06/buddy_agent/actions/workflows/ci.yml)\n\nNews Buddy is a daily AI news digest that fetches RSS feeds, filters for AI-relevant stories, deduplicates against prior runs, summarizes the best articles with an LLM, generates a visual explainer for each story, publishes a web archive, and can send the result by email, Telegram, or Slack.\n\nLive archive: https://harshagarwal06.github.io/buddy_agent/\n\nThe project started as a fully agentic `deepagents` experiment. After testing the daily workflow, the orchestration was moved to a deterministic LangGraph pipeline: fetch, filter, dedup, summarize, write, notify. That kept the system cheaper and easier to debug while preserving LLM judgment where it matters: article summarization, editorial importance, and article-specific image planning.\n\n## What It Does\n\n- Reads AI-focused RSS feeds from `config.yaml`.\n- Filters stories with source allowlists and AI keywords.\n- Deduplicates URLs with a local SQLite `state.db`.\n- Backfills the lookback window when too few fresh stories survive filtering.\n- Writes self-contained reader briefings through a provider-swappable LLM layer.\n- Requires each briefing to explain what happened, useful context, and why it matters, scored by a pure-heuristic rubric (`news_buddy/rubric.py`); thin summaries are retried once.\n- Generates cached 4:3 article-grounded explainers in one shared editorial system.\n- Writes Markdown and HTML digests to `~/news/YYYY-MM-DD.md` and `.html`.\n- Regenerates an archive index for GitHub Pages.\n- Sends non-empty digests through Telegram, Slack, and Buttondown when configured.\n- Supports safe manual verification with `--test-run`, which fetches and summarizes without mutating dedup state, writing RAG entries, deploying, or notifying subscribers.\n- Writes each embedded article once as an OKF (Open Knowledge Format) Markdown file (`news_buddy/knowledge_base.py`), then embeds that text into a local Chroma vector store for semantic search.\n- Exposes a separate read-only MCP server (`news_buddy_mcp/`) that lets an LLM search and fetch past digests over the public JSON archive.\n- Ships an offline evaluation harness (`scripts/eval_sub_model.py`) that scores candidate summarizer models against frozen article fixtures with the same rubric, so model swaps are decided on evidence.\n- Supports optional OpenTelemetry tracing to a local Arize Phoenix UI for full LLM-call visibility during development.\n\n## Architecture\n\n```mermaid\nflowchart TD\n    A[\"GitHub Actions or local CLI\"] --> B[\"news_buddy.__main__\"]\n    B --> C[\"LangGraph StateGraph\"]\n    C --> D[\"Fetch RSS feeds in parallel\"]\n    D --> E[\"Filter AI stories\"]\n    E --> F[\"Deduplicate against SQLite\"]\n    F --> G{\"Any articles?\"}\n    G -- \"yes\" --> H[\"Extract article text\"]\n    H --> I[\"Summarize with configured sub_model\"]\n    I --> J[\"Rubric score and retry\"]\n    J --> K[\"Mark seen and optionally write OKF file + embed in Chroma\"]\n    K --> L[\"Generate and cache article illustrations\"]\n    L --> M[\"Format Markdown digest\"]\n    G -- \"no\" --> X[\"Write empty digest\"]\n    M --> N[\"Write Markdown and HTML\"]\n    X --> N\n    N --> O[\"Update archive index\"]\n    O -. \"CLI, after graph\" .-> P[\"Notify Telegram, Slack, Buttondown\"]\n    O -. \"GitHub Actions, after CLI\" .-> Q[\"Deploy publication to gh-pages\"]\n```\n\nSee [DIAGRAM.md](DIAGRAM.md) for the node-level flow,\n[docs/rag-architecture.html](docs/rag-architecture.html) for the RAG/search\nview, and the [Code Brain](openwiki/index.md) for the source-linked maintainer\ndocumentation.\n\n## Code Brain\n\nThe [`openwiki/`](openwiki/) directory is a reviewed OpenWiki knowledge layer\ncovering runtime flow, providers, image generation, persistence, publishing,\nnotifications, safe run modes, and known gaps. Start with\n[`openwiki/quickstart.md`](openwiki/quickstart.md).\n\nIt is documentation only: News Buddy never imports or reads it at runtime.\nSource code, tests, `config.yaml`, and active workflows remain authoritative.\n[`openwiki/INSTRUCTIONS.md`](openwiki/INSTRUCTIONS.md) constrains generation so\nthe wiki preserves important boundaries such as deterministic filtering versus\nLLM summarization and local Chroma versus public JSON/MCP search.\n\nOpenWiki 0.2.4 is pinned in\n[`.github/workflows/openwiki-update.yml`](.github/workflows/openwiki-update.yml).\nThe workflow runs manually or weekly, uses the existing `NVIDIA_API_KEY`\nrepository secret, validates the result, and opens a documentation-only draft\npull request. It never runs the digest or deploys `gh-pages`.\n\nTo regenerate and validate locally with `NVIDIA_API_KEY` already exported:\n\n```bash\nnpm install --global openwiki@0.2.4 mermaid@11.16.0 jsdom@29.1.1\nOPENWIKI_PROVIDER=nvidia \\\nOPENWIKI_MODEL_ID=nvidia/nemotron-3-super-120b-a12b \\\nOPENWIKI_TELEMETRY_DISABLED=1 \\\nopenwiki code --update --print\npython scripts/validate_openwiki.py\n```\n\n## Repository Tour\n\n- `PROJECT_OVERVIEW.md` - the complete source-derived reference: technology inventory, subsystem deep dives, full configuration and environment reference, run-mode semantics, and project statistics.\n- `news_buddy/agent.py` - the LangGraph pipeline and node logic.\n- `news_buddy/__main__.py` - CLI, notification routing, and run summary output.\n- `news_buddy/llm.py` - the only place that constructs LLM clients.\n- `news_buddy/feeds.py` - RSS fetching and item normalization.\n- `news_buddy/extract.py` - article body extraction with RSS-summary fallback.\n- `news_buddy/image_generator.py` - NVIDIA/Hugging Face image generation, validation, WebP caching, and SVG fallback assets.\n- `prompts/image_style.md` - the single layout, style, grounding, and image-quality contract.\n- `news_buddy/state.py` - SQLite dedup state.\n- `news_buddy/rubric.py` - pure-heuristic summary quality scoring and the strict-retry decision.\n- `news_buddy/html_writer.py` and `news_buddy/archive_writer.py` - generated digest pages and archive index.\n- `news_buddy/knowledge_base.py` - writes each accepted article as an OKF-formatted Markdown file, the source of truth for embedding.\n- `news_buddy/rag.py` - ChromaDB-backed semantic search over saved articles; embeds the OKF file text.\n- `news_buddy/backfill_rag.py` - one-time backfill of the vector store from articles seen before RAG existed.\n- `news_buddy/observability.py` - opt-in OpenTelemetry tracing via Arize Phoenix.\n- `news_buddy/buttondown_notify.py`, `telegram_notify.py`, `slack_notify.py` - notification adapters.\n- `news_buddy_mcp/` - separate FastMCP server exposing read-only search/digest tools over the public JSON archive, with its own tests, lint, and Dockerfile.\n- `scripts/eval_sub_model.py` and `scripts/eval_report.py`/`eval_scoring.py`/`eval_store.py` - offline harness for comparing candidate summarizer models against frozen fixtures.\n- `.agents/skills/topicsearch/` - local agent skill for combined keyword and semantic archive search.\n- `.github/workflows/daily-digest.yml` - scheduled cloud run and GitHub Pages deploy.\n- `.github/workflows/openwiki-update.yml` - pinned manual/weekly Code Brain update that proposes a draft PR.\n- `openwiki/` - source-linked maintainer documentation generated with OpenWiki and reviewed against the code.\n- `scripts/validate_openwiki.py` - dependency-free Code Brain structure, link, and accuracy tripwire.\n- `tests/` - 124 tests covering notifications, package resource paths, archive signup behavior, CLI notification suppression, rubric scoring, RAG, and the evaluation harness.\n\n## Setup\n\nRequirements:\n\n- Python 3.11+\n- `uv` or `pip`\n- One LLM provider credential:\n  - `NVIDIA_API_KEY` for the default NVIDIA NIM summarizer/planner\n  - `GOOGLE_API_KEY` when Gemini is selected and for RAG embeddings\n  - `HF_TOKEN` when Hugging Face is selected instead\n  - local Ollama plus pulled models if `llm.provider: ollama`\n\nInstall:\n\n```bash\ngit clone https://github.com/Harshagarwal06/buddy_agent.git\ncd buddy_agent\npython -m venv .venv\nsource .venv/bin/activate\npip install .\ncp .env.example .env\n```\n\nEdit `.env` with provider and notification secrets. Edit `config.yaml` to tune feeds, keyword filtering, article limits, and model provider.\n\nThe default install contains everything needed for the configured NVIDIA\npipeline. Install only the extras you use for other providers or local tools:\n\n```bash\npip install '.[google]'          # Gemini provider\npip install '.[huggingface]'     # Hugging Face provider\npip install '.[ollama]'          # local Ollama provider\npip install '.[rag]'             # local Chroma semantic search + Gemini embeddings\npip install '.[observability]'   # OpenTelemetry client for a Phoenix collector\n```\n\nPackaged defaults include `config.yaml`, prompts, and web assets, so the\n`news-buddy` command also works when installed outside a source checkout. By\ndefault, writable state is kept in the repository root for a checkout and the\ncurrent directory for an installed package. Set `NEWS_BUDDY_HOME` to choose an\nexplicit writable data directory.\n\nThe `images` block in `config.yaml` controls the image model, output dimensions,\ncompression, concurrency, retries, and shared explainer system. Its\n`style_guide` points to `prompts/image_style.md`, which both the article planner\nand image renderer read. Production also sets `require_article_brief: true`, so\nan LLM outage or incomplete plan stops publication instead of producing generic\nplaceholder diagrams. The default NVIDIA FLUX.2-klein-4B integration uses\n`NVIDIA_API_KEY`. Normal `--test-run` executions skip image generation unless\n`images.generate_in_test_run` is explicitly set to `true`.\n\nTracing is opt-in and off by default. Install the `observability` extra, run an\n[Arize Phoenix](https://phoenix.arize.com/) collector separately, and set\n`OTEL_TRACING=true` before a run to trace every LLM call\n(`PHOENIX_COLLECTOR_ENDPOINT`, default `http://localhost:6006`); see\n`news_buddy/observability.py`.\n\n## Running Locally\n\nDry run with no network or file side effects:\n\n```bash\npython -m news_buddy run --dry-run --verbose\n```\n\nSafe live validation, recommended before changing scheduled delivery:\n\n```bash\npython -m news_buddy run --test-run --verbose\n```\n\nReal local run:\n\n```bash\npython -m news_buddy run --verbose\n```\n\nBy default, output is written to `~/news/`. The CLI prints article count, estimated token cost, duration, rubric failures, and notification status.\n\n## Deployment\n\nThe scheduled workflow in `.github/workflows/daily-digest.yml` runs daily in GitHub Actions. It:\n\n1. Sets up Python and uv.\n2. Checks `gh-pages` for today's digest and skips backup runs when it is already published.\n3. Installs from `uv.lock` with `uv sync --frozen --no-dev`.\n4. Restores `state.db` and generated images from Actions cache.\n5. Runs `uv run python -m news_buddy run --notify-at-utc 02:30`.\n6. Copies the generated HTML, JSON search record, and image assets to the `gh-pages` branch.\n7. Rebuilds the archive index for GitHub Pages.\n\nThe workflow has one primary morning schedule and two backup schedules. A concurrency group prevents overlapping digest jobs, and the `gh-pages` preflight keeps delayed backup schedules from sending duplicate notifications after the day's digest is already published.\n\nA separate CI workflow (`.github/workflows/ci.yml`) runs on pushes and pull requests with two jobs: lint, 124 tests, a runtime dependency audit, and an installed-wheel smoke test for the main package; plus lint, 15 tests, a runtime dependency audit, and a Docker build for `news_buddy_mcp/`.\n\nManual `workflow_dispatch` defaults to `test_run: true`, so a verification run does not mark stories seen, deploy pages, or notify subscribers.\n\nRequired GitHub secrets depend on enabled features:\n\n- `NVIDIA_API_KEY` for the default NVIDIA summarizer/image planner, FLUX.2 article images, and the OpenWiki update workflow.\n- `GOOGLE_API_KEY` when Gemini summarization is selected and for local RAG embeddings.\n- `HF_TOKEN` when Hugging Face summarization is selected.\n- `TELEGRAM_BOT_TOKEN` and `TELEGRAM_CHAT_ID` for Telegram.\n- `SLACK_WEBHOOK_URL` for Slack.\n- `BUTTONDOWN_API_KEY` for sending email.\n- `BUTTONDOWN_USERNAME` for the archive signup form.\n\n## Search\n\nKeyword search over the SQLite seen-state:\n\n```bash\npython news_buddy/search.py \"OpenAI\" --limit 10\n```\n\nSemantic search over the Chroma vector store:\n\n```bash\npython news_buddy/semantic_search_cli.py \"AI chip capacity\" --limit 10\n```\n\nThe daily workflow currently sets `NEWS_BUDDY_RAG_ENABLED=false`, so Chroma is best treated as a local/search experiment unless the workflow persistence is enabled.\n\nThe RAG extra uses Chroma's in-process `PersistentClient`. Do not expose a\nChroma HTTP server from this environment: ChromaDB 1.5.9 has an\n[unfixed pre-authentication server vulnerability](https://osv.dev/vulnerability/PYSEC-2026-311)\nin an API path that News Buddy does not use.\n\n## Public MCP Server\n\n`news_buddy_mcp/` is a separate FastMCP server that reads the public\n`index.json`/`YYYY-MM-DD.json` archive published to `gh-pages` and exposes it\nas three read-only MCP tools: `search_articles`, `get_digest`, and\n`list_digests`. It never touches `state.db` or Chroma, has its own\n`pyproject.toml`/`uv.lock`, test suite, and Dockerfile, and is built and\ntested by a dedicated job in CI.\n\n```bash\ncd news_buddy_mcp\nuv sync\nNEWS_BUDDY_ARCHIVE_URL=https://harshagarwal06.github.io/buddy_agent uv run python -m news_buddy_mcp.server\n```\n\n## Model Evaluation\n\n`scripts/eval_sub_model.py` captures a frozen set of real articles as\nfixtures, then replays them through candidate `sub_model` values and scores\neach with the pipeline's own `RubricMiddleware` and image-brief validation —\nso a model swap is judged by brief validity, rubric pass rate, latency, and\ntoken cost, not vibes. It is opt-in (never run by CI, since it makes real\nmodel calls):\n\n```bash\npython -m scripts.eval_sub_model --capture   # freeze fixtures once\npython -m scripts.eval_sub_model --run        # score candidates against them\n```\n\n[`docs/evals/2026-07-30-sub-model-baseline.md`](docs/evals/2026-07-30-sub-model-baseline.md)\nis a worked example: four candidates were compared against the production\ndefault (`meta/llama-3.1-8b-instruct`), and the incumbent was kept because\nevery candidate fell short on brief validity, the pass/fail gate fixed before\nthe run.\n\n## Current Gaps\n\n- Dedup is URL-based, so the same story from several outlets can still appear as separate entries.\n- RAG is not persisted in CI yet (`NEWS_BUDDY_RAG_ENABLED=false` in the daily workflow).\n- Image generation, caching, rendering, notifications, rubric scoring, RAG/knowledge-base writing, and the evaluation harness have focused tests; raw feed parsing (`news_buddy/feeds.py`) and the SQLite dedup mechanics (`news_buddy/state.py`) still need direct coverage.\n\nThese are intentionally visible because they make the next engineering steps clear: story-level clustering, live RAG persistence, state recovery, and broader tests.\n",
  "bytes": 15216,
  "sha": "185831d372920514718d12423f614b3b67748735b08b16e1249d9677fceaaeb3",
  "repo_slug": "harshagarwal06/buddy_agent",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/okf_harshagarwal06_buddy_agent_openwiki_inde_e4cbf131/readme"
}