batch_web_scraper: Knowledge Bundle
Bundle OKF 0.2 · 4 conceitos · edrod-oncology/batch-web-scraper
Open source Repository Open in the app JSON README (API)
About
# batch_web_scraper: Knowledge Bundle
This is an Open Knowledge Format (OKF) bundle for **batch_web_scraper**, a
web app that turns a company domain (or a direct team-page URL) into a
downloadable CSV of executive leadership, using pure heuristics and no LLM.
Each concept is one markdown file with YAML frontmatter; the directory tree
groups concepts by type and markdown links connect related concepts.
## Concept types
* [concepts](/concepts/index.md) - how the scraper works: the pipeline, the
discovery/extraction heuristics, and the CSV-driven batch scraper
* [runbooks](/runbooks/index.md) - operational procedures: run locally, run
the batch scraper, deploy to Render
* [apis](/apis/index.md) - the HTTP surface the UI and Render's health check
use
## Start here
* [scrape-pipeline](/concepts/scrape-pipeline.md) - the top-level flow and
module map; links out to everything else.
Details
- Kind
- OKF bundles
- Topic
- Files & documents
- Publisher
- edrod-oncology
- Origin
- okf_github
- Category
- dados
- Version
- 0.2
- Last push
- 2026-09-04T12:53:00Z
- Repository state
- ativo
- Language
- Python
- License
- MIT
- Added
- 2026-09-08 16:02:35
- Updated
- 2026-09-08 16:02:35
- Origin id
edrod-oncology/batch-web-scraper:okf/index.md
README
# batch-web-scraper
Turn a list of company names into a CSV of their executive leadership.
Two frontends over one scrape pipeline, with **no LLM anywhere in the scrape
path**: discovery and extraction are pure heuristics (link keyword scoring,
repeated-DOM-structure card detection, regex and `phonenumbers` validation),
with a Playwright Chromium fallback for JavaScript-rendered pages.
- **Web app**: paste a company domain (or a direct team-page URL), get a
downloadable CSV of `name, title, email, linkedin_url, phone, bio,
source_url`.
- **Batch CLI**: point it at a CSV of company names (a CRM export works
as-is), and it resolves each company to its website via search-engine
lookup, finds the leadership page, and appends every person into one
combined CSV with a resumable per-company run log.
This tool was extracted from a real contact-enrichment project, where it
scraped 1,900+ executive profiles across ~200 companies in one resumable
run. It ships sanitized: no source data is included.
## Local setup
Requires Python 3.12+ and [uv](https://docs.astral.sh/uv/).
git clone git@github.com:edrod-oncology/batch-web-scraper.git
cd batch-web-scraper
uv sync
uv run playwright install chromium # optional: only needed for JS-heavy sites
Run the web app:
uv run uvicorn batch_web_scraper.web.app:app --port 8000
Open http://localhost:8000.
Run the tests (fixture-based, no network access):
uv run pytest
## Batch usage
Put your company list in a CSV. The company-name column is auto-detected
(`Company`, `Company name`, `Opportunity name`, `Account name`,
`Organization`), or name it with `--column`. An example file is included at
`examples/companies.example.csv`.
uv run python -m batch_web_scraper.batch --input companies.csv
Flags:
--input PATH CSV with a company-name column (default: companies.csv)
--column NAME company column name (default: auto-detect)
--output PATH combined leadership CSV (default: leadership_batch.csv)
--log PATH per-company run log CSV (default: leadership_batch_log.csv)
--limit N only process the first N company names (smoke test)
--no-resume start fresh instead of skipping already-logged companies
The run is safe to kill and restart: on startup it skips every company
already present in the log CSV, and all output flushes after every row, so
a multi-hour run survives Ctrl-C, restarts, and crashes. Read the log CSV
first when reviewing results: `domain_confidence` flags shaky
company-to-website matches, and `status` explains every company that
produced no rows (`no_leadership_page`, `fetch_error`, `no_domain`).
## Row hygiene (clean output on the first pass)
Both tools apply a shared filter (`core/rowfilter.py`) inside the scrape
pipeline, so corporate boilerplate never reaches the export:
- **Non-person rows are dropped**: marketing taglines, nav labels, section
headers, disease/award/committee/geography/product tiles, even when they
carry an email or LinkedIn URL. They are not lost: the batch run writes
them to `<output>_rejected.csv`, and the web app reports a filtered count
with a "Download filtered-out rows" link.
- **Generic role mailboxes** (`info@`, `ir@`, `media@`, ...) are blanked on
surviving rows; a shared inbox is not a contact's address.
- **Placeholder titles** ("Connect with me on LinkedIn", "Read Bio") are
blanked, but the person is kept.
The heuristics, policies, and CSV column contracts are documented in the
[OKF knowledge bundle](okf/index.md) (`okf/concepts/`, `okf/runbooks/`).
## Deploying to a cloud service
The web app is deterministic Python + Playwright and deploys fine. The repo
carries a Dockerfile and a `render.yaml` blueprint for
[Render](https://render.com)'s free tier: New > Blueprint > select this
repo > Apply. The container builds from the committed lockfile, installs
Chromium at build time, and runs as a non-root user. The free instance
sleeps when idle; the first request after idle cold-starts in about a
minute. Any Docker host (Fly.io, Railway, a VPS) works the same way; give
it at least 512 MB so the Playwright fallback can start.
Things to consider before running it in the cloud:
- **Protect it.** The app is open by default. Set
`LEADERSHIP_AUTH_PASSWORD` (and optionally `LEADERSHIP_AUTH_USER`,
default `admin`) to require HTTP Basic Auth on every page and API call;
`/api/health` stays open for the platform health check. Do not run an
open scraper endpoint on the public internet.
- **Batch mode is better run locally.** The batch CLI hits search engines
to resolve domains, and cloud data-center IPs get CAPTCHA-challenged and
rate-limited far more aggressively than residential IPs. It is polite by
design (2-4 s randomized delays between searches), but a long batch from
a cloud IP will still hit walls a home connection does not. The
deployed web app (one URL at a time, no search-engine step when you
paste a direct URL) is the cloud-friendly half.
- **The free tier is small.** The web app enforces 30 pages / 5 minutes /
1 concurrent job to fit free-tier memory; raise those limits only with
more RAM.
- **SSRF guard.** Private and internal hosts are refused, so a deployed
instance cannot be used to probe the hosting network.
### What about the AI enrichment phase?
In the project this tool came from, a second phase filled missing LinkedIn
URLs and email addresses using an AI agent (Claude Code) driving a real
desktop Chrome via the Claude in Chrome extension. That phase is **not part
of this repo, and it cannot run in a cloud service**: the extension pairs
with a local, visible Chrome on the operator's machine, and there is no
headless or server-side mode. If you want enrichment in the cloud, the
realistic options are a commercial search API (Bing, Brave, SerpAPI) or an
enrichment vendor with an API, each with its own terms and costs. The
scrape pipeline in this repo stays deliberately LLM-free either way.
## Custom domain
In Render: service > Settings > Custom Domains > Add, enter your subdomain
(for example `scraper.example.com`), add the CNAME Render shows you at your
DNS provider, then Verify; TLS is provisioned automatically.
## Responsible use
This tool reads public corporate leadership pages: information companies
publish about themselves. Use it accordingly: respect `robots.txt` and site
terms, keep the polite delays in place, and comply with the privacy laws
that apply to you (GDPR, CCPA) when storing or using personal data such as
names, titles, and business contact details. Do not use it to harvest
personal emails at scale or to contact people who have opted out.
## Limits
- Heuristics, not magic: unusual page layouts produce partial rows.
- Domain resolution can mismatch similarly named companies; check
`domain_confidence` in the run log before trusting low-scoring rows.
- Emails and phones appear only when a site publishes them; most do not.
## License
MIT. See [LICENSE](LICENSE).