{
  "markdown": "# batch-web-scraper\n\nTurn a list of company names into a CSV of their executive leadership.\n\nTwo frontends over one scrape pipeline, with **no LLM anywhere in the scrape\npath**: discovery and extraction are pure heuristics (link keyword scoring,\nrepeated-DOM-structure card detection, regex and `phonenumbers` validation),\nwith a Playwright Chromium fallback for JavaScript-rendered pages.\n\n- **Web app**: paste a company domain (or a direct team-page URL), get a\n  downloadable CSV of `name, title, email, linkedin_url, phone, bio,\n  source_url`.\n- **Batch CLI**: point it at a CSV of company names (a CRM export works\n  as-is), and it resolves each company to its website via search-engine\n  lookup, finds the leadership page, and appends every person into one\n  combined CSV with a resumable per-company run log.\n\nThis tool was extracted from a real contact-enrichment project, where it\nscraped 1,900+ executive profiles across ~200 companies in one resumable\nrun. It ships sanitized: no source data is included.\n\n## Local setup\n\nRequires Python 3.12+ and [uv](https://docs.astral.sh/uv/).\n\n    git clone git@github.com:edrod-oncology/batch-web-scraper.git\n    cd batch-web-scraper\n    uv sync\n    uv run playwright install chromium   # optional: only needed for JS-heavy sites\n\nRun the web app:\n\n    uv run uvicorn batch_web_scraper.web.app:app --port 8000\n\nOpen http://localhost:8000.\n\nRun the tests (fixture-based, no network access):\n\n    uv run pytest\n\n## Batch usage\n\nPut your company list in a CSV. The company-name column is auto-detected\n(`Company`, `Company name`, `Opportunity name`, `Account name`,\n`Organization`), or name it with `--column`. An example file is included at\n`examples/companies.example.csv`.\n\n    uv run python -m batch_web_scraper.batch --input companies.csv\n\nFlags:\n\n    --input PATH     CSV with a company-name column (default: companies.csv)\n    --column NAME    company column name (default: auto-detect)\n    --output PATH    combined leadership CSV (default: leadership_batch.csv)\n    --log PATH       per-company run log CSV (default: leadership_batch_log.csv)\n    --limit N        only process the first N company names (smoke test)\n    --no-resume      start fresh instead of skipping already-logged companies\n\nThe run is safe to kill and restart: on startup it skips every company\nalready present in the log CSV, and all output flushes after every row, so\na multi-hour run survives Ctrl-C, restarts, and crashes. Read the log CSV\nfirst when reviewing results: `domain_confidence` flags shaky\ncompany-to-website matches, and `status` explains every company that\nproduced no rows (`no_leadership_page`, `fetch_error`, `no_domain`).\n\n## Row hygiene (clean output on the first pass)\n\nBoth tools apply a shared filter (`core/rowfilter.py`) inside the scrape\npipeline, so corporate boilerplate never reaches the export:\n\n- **Non-person rows are dropped**: marketing taglines, nav labels, section\n  headers, disease/award/committee/geography/product tiles, even when they\n  carry an email or LinkedIn URL. They are not lost: the batch run writes\n  them to `<output>_rejected.csv`, and the web app reports a filtered count\n  with a \"Download filtered-out rows\" link.\n- **Generic role mailboxes** (`info@`, `ir@`, `media@`, ...) are blanked on\n  surviving rows; a shared inbox is not a contact's address.\n- **Placeholder titles** (\"Connect with me on LinkedIn\", \"Read Bio\") are\n  blanked, but the person is kept.\n\nThe heuristics, policies, and CSV column contracts are documented in the\n[OKF knowledge bundle](okf/index.md) (`okf/concepts/`, `okf/runbooks/`).\n\n## Deploying to a cloud service\n\nThe web app is deterministic Python + Playwright and deploys fine. The repo\ncarries a Dockerfile and a `render.yaml` blueprint for\n[Render](https://render.com)'s free tier: New > Blueprint > select this\nrepo > Apply. The container builds from the committed lockfile, installs\nChromium at build time, and runs as a non-root user. The free instance\nsleeps when idle; the first request after idle cold-starts in about a\nminute. Any Docker host (Fly.io, Railway, a VPS) works the same way; give\nit at least 512 MB so the Playwright fallback can start.\n\nThings to consider before running it in the cloud:\n\n- **Protect it.** The app is open by default. Set\n  `LEADERSHIP_AUTH_PASSWORD` (and optionally `LEADERSHIP_AUTH_USER`,\n  default `admin`) to require HTTP Basic Auth on every page and API call;\n  `/api/health` stays open for the platform health check. Do not run an\n  open scraper endpoint on the public internet.\n- **Batch mode is better run locally.** The batch CLI hits search engines\n  to resolve domains, and cloud data-center IPs get CAPTCHA-challenged and\n  rate-limited far more aggressively than residential IPs. It is polite by\n  design (2-4 s randomized delays between searches), but a long batch from\n  a cloud IP will still hit walls a home connection does not. The\n  deployed web app (one URL at a time, no search-engine step when you\n  paste a direct URL) is the cloud-friendly half.\n- **The free tier is small.** The web app enforces 30 pages / 5 minutes /\n  1 concurrent job to fit free-tier memory; raise those limits only with\n  more RAM.\n- **SSRF guard.** Private and internal hosts are refused, so a deployed\n  instance cannot be used to probe the hosting network.\n\n### What about the AI enrichment phase?\n\nIn the project this tool came from, a second phase filled missing LinkedIn\nURLs and email addresses using an AI agent (Claude Code) driving a real\ndesktop Chrome via the Claude in Chrome extension. That phase is **not part\nof this repo, and it cannot run in a cloud service**: the extension pairs\nwith a local, visible Chrome on the operator's machine, and there is no\nheadless or server-side mode. If you want enrichment in the cloud, the\nrealistic options are a commercial search API (Bing, Brave, SerpAPI) or an\nenrichment vendor with an API, each with its own terms and costs. The\nscrape pipeline in this repo stays deliberately LLM-free either way.\n\n## Custom domain\n\nIn Render: service > Settings > Custom Domains > Add, enter your subdomain\n(for example `scraper.example.com`), add the CNAME Render shows you at your\nDNS provider, then Verify; TLS is provisioned automatically.\n\n## Responsible use\n\nThis tool reads public corporate leadership pages: information companies\npublish about themselves. Use it accordingly: respect `robots.txt` and site\nterms, keep the polite delays in place, and comply with the privacy laws\nthat apply to you (GDPR, CCPA) when storing or using personal data such as\nnames, titles, and business contact details. Do not use it to harvest\npersonal emails at scale or to contact people who have opted out.\n\n## Limits\n\n- Heuristics, not magic: unusual page layouts produce partial rows.\n- Domain resolution can mismatch similarly named companies; check\n  `domain_confidence` in the run log before trusting low-scoring rows.\n- Emails and phones appear only when a site publishes them; most do not.\n\n## License\n\nMIT. See [LICENSE](LICENSE).\n",
  "bytes": 7026,
  "sha": "8d67abd37fdb45e58b2e1eb3cc2b8ed4fb6e89175f906f05f96696c9eb79ddf6",
  "repo_slug": "edrod-oncology/batch-web-scraper",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/okf_edrod_oncology_batch_web_scraper_okf_ind_ef08748d/readme"
}