{
  "markdown": "# Job Search AI Agent\n\nAn AI-powered remote job search assistant. Type in your desired job keywords and the agent searches, filters, and presents the best matches for you. Talk to it instead, if you'd rather — see [Voice AI](#️-voice-ai).\n\n**[▶ Try it live](https://job-search-agent-blond.vercel.app)** — no signup, no API key.\n\n⚡ Instant results | 💰 $0 per search — searching and filtering are fully deterministic | 📊 [0.90 relevance](#evals) across 24 eval cases\n\nThe eval suite is itself under test. Two runs once scored the *same* nine listings 0.889 and 0.222, and the average came out identical both times because the disagreements cancelled out — so [the judge was audited and rewritten](#making-the-judge-repeatable) before any of these numbers were trusted.\n\n> The backend runs on a free Render instance that sleeps after 15 minutes of inactivity. If the demo has been idle, the first search takes about a minute — roughly 50s to wake the server, then 15s to query four job APIs. Every search after that is instant.\n\n---\n\n## What it does\n\nThe agent connects to 4 job APIs simultaneously:\n\n- [RemoteOK API](https://remoteok.com/api)\n- [Himalayas API](https://himalayas.app/jobs/api)\n- [Remotive API](https://remotive.com/api/remote-jobs)\n- [Jobicy.com](https://jobicy.com/api/v2/remote-jobs)\n\nResults are scored, ranked and filtered in plain Python — no model in the search path. An LLM is still used where judgement genuinely helps: turning your spoken feedback into filter keywords, reading your CV, and scoring listings for the n8n digest.\n\n### Why no LLM in the search path\n\nThe agent originally passed every fetched listing through Gemini to decide relevance. That worked, but it hid a bug: irrelevant results (Marketing, Janitor) were never filtered out at fetch time — the model just declined to print them. The moment you asked for them a second way, they came back.\n\nFiltering at the source instead fixed the bug, removed the cost, and made results reproducible. Quality is measured — see [Evals](#evals).\n\n---\n\n## How it works\n\nBuilt with **LangGraph** at its core. The graph uses a **fan-out architecture** — the agent spawns parallel fetch nodes using LangGraph's `Send` API. Results are deduplicated, scored for relevance against your query, ranked, and stored.\n\n```mermaid\nflowchart TD\n    START([START]) --> fan_out{fan_out}\n\n    fan_out -->|Send API| fetch_jobs[fetch_jobs]\n    fan_out -->|Send API| fetch_sjobs[fetch_sjobs]\n    fan_out -->|Send API| fetch_tjobs[fetch_tjobs]\n    fan_out -->|Send API| fetch_fjobs[fetch_fjobs]\n\n    fetch_jobs --> collect_results[collect_results\\ndedupe · score · rank]\n    fetch_sjobs --> collect_results\n    fetch_tjobs --> collect_results\n    fetch_fjobs --> collect_results\n\n    collect_results --> END([END])\n\n    classDef startEnd fill:#a78bfa,stroke:#7c3aed,stroke-width:2px,color:#000;\n    classDef nodeStyle fill:#f3e8ff,stroke:#9333ea,stroke-width:1px,color:#000;\n    classDef condStyle fill:#faf5ff,stroke:#c084fc,stroke-width:2px,color:#000;\n\n    class START,END startEnd;\n    class fetch_jobs,fetch_sjobs,fetch_tjobs,fetch_fjobs,collect_results nodeStyle;\n    class fan_out condStyle;\n```\n\n### Relevance scoring\n\nA job scores on where your query words appear. A hit in the **title** outweighs any number of hits in the description, so `title_hits * 10 + description_hits` sorts real matches to the top.\n\nFour details that matter in practice:\n\nShort query words have to match a whole word. Searching `ai` should not hit `p-ai-d media specialist`. Words of four characters or more still match inside a word, so `python` finds `python3`.\n\nA listing needs a title hit to qualify at all. Mentioning your keywords somewhere in the body text is not enough. Description hits still count, but only to break ties between listings that already earned their place.\n\nThe agent returns what matched, up to a cap of 12, and it will happily return three listings or none. An earlier version guaranteed a minimum of eight; on narrow queries, seven of those eight turned out to be noise.\n\nGeneric words are dropped. `developer`, `engineer`, `remote`, `role` and the rest show up in half of all job titles, so they can't rank anything. If a query is nothing but generic words, they get used anyway rather than matching the entire board.\n\nRemoteOK's tags are not trusted. The API is still queried by tag, but every listing that comes back is re-checked against its own title. Across 101 listings that's an average of 23.6 tags each, a nursing role tagged `python`, `sql`, `postgres` and `golang`, and three unrelated listings sharing an identical 36-tag list. The tags are SEO filler. The title isn't.\n\n### Feedback and memory\n\nTell the agent what to filter out — \"no MERN\", \"no senior roles\", \"no usa\" — and it stores your preferences in **PostgreSQL via LangGraph's PostgresSaver**. Filters persist across searches and across days.\n\nSeniority words (`senior`, `junior`, `lead`, `principal`, …) are matched against the **title only**. Every posting says \"work with senior engineers\" somewhere in its body; that shouldn't disqualify a mid-level role. Everything else — technologies, locations — is matched against title, description and location.\n\nActive filters are shown at the bottom of every response. Say **\"reset filters\"** to clear them.\n\nThe agent reads everything the four APIs return and shows the top 12 after ranking (`MAX_RESULTS`, at the top of `agent.py`). RemoteOK is the exception: it is asked to match on the title before it replies, so its own ten are already ten that matched.\n\n> **LangSmith tracing is enabled.** Graph runs are fully observable — every node execution, its latency and its output. Note that the search path no longer makes LLM calls, so token usage now appears only for feedback extraction, CV upload and the n8n evaluator.\n\n---\n\n## 🎙️ Voice AI\n\nTalk to the agent instead of typing. Same LangGraph brain (`agent.py`, unmodified) — a new interface built with [Pipecat](https://pipecat.ai): Deepgram (STT), ElevenLabs (TTS), Daily (real-time transport).\n\nFeedback works over voice too — say \"no senior roles\" mid-conversation and the agent filters the results it already found, live, with no repeated API calls. Ask \"tell me more about the first one\" for details on a specific listing. Voice sessions are intentionally stateless (in-memory only, no Postgres) since a single call is short-lived, unlike the text agent's persistent memory across days.\n\nSee [`voice/README.md`](./voice/README.md) for architecture details and setup instructions.\n\n---\n\n## 🔌 MCP Server\n\nA third interface onto the same brain: [`mcp_server.py`](./mcp_server.py) exposes the agent over the [Model Context Protocol](https://modelcontextprotocol.io), so Claude Desktop, Claude Code and any other MCP client can search jobs directly inside a conversation.\n\n**No API keys required.** Since the search path is fully deterministic, this server needs no `GEMINI_API_KEY`, no database and no network config — clone, install, point your client at it.\n\nOne read-only tool:\n\n```python\nsearch_remote_jobs(query: str, exclude_keywords: list[str] = []) -> list[dict]\n```\n\nIt returns the normalized job shape (`position`, `company`, `location`, `salary`, `description`, `apply_url`) rather than formatted text, so the calling model can filter and rank the results itself.\n\nThe tool is designed around **intent, not endpoints**: `exclude_keywords` is a parameter rather than a separate feedback call, which means no thread state and no Postgres here. The client's own conversation is the memory — say \"no senior roles\" and it simply calls the tool again with a fuller exclusion list, extracting the keywords itself instead of spending a Gemini call.\n\nFan-out results are cached in-process per query for 4 hours, so refinements filter locally and never re-hit the job boards.\n\n**Setup** — add to `claude_desktop_config.json` (Windows: `%APPDATA%\\Claude\\`, macOS: `~/Library/Application Support/Claude/`):\n\n```json\n{\n  \"mcpServers\": {\n    \"jobsearch\": {\n      \"command\": \"/absolute/path/to/.venv/bin/python\",\n      \"args\": [\"/absolute/path/to/mcp_server.py\"]\n    }\n  }\n}\n```\n\nUse absolute paths to the virtualenv's interpreter — the client starts the process directly, with no shell and no activated environment. Restart the client fully; logs land in `logs/mcp-server-jobsearch.log`.\n\n---\n\n## Tech Stack\n\n| Layer | Technology |\n|-------|-----------|\n| Agent | LangGraph (fan-out with `Send` API) |\n| Memory | PostgreSQL via `PostgresSaver` |\n| Voice | Pipecat (Daily transport, Deepgram STT, ElevenLabs TTS) |\n| MCP | Official Python SDK (`mcp`), stdio transport |\n| LLM | Google Gemini 2.5 Flash — feedback extraction, CV parsing, n8n scoring |\n| LLM Integration | LangChain `init_chat_model` |\n| Backend | FastAPI |\n| Frontend | Next.js 15 + ReactMarkdown + remark-gfm |\n| Job APIs | RemoteOK, Himalayas, Remotive, Jobicy |\n| Evals | LangSmith dataset + LLM-as-judge (Gemini 2.5 Flash) — 0.90 across 24 cases |\n\n---\n\n## Getting Started\n\n```bash\n# Backend\npip install -r requirements.txt\ncp .env.example .env  # add your API keys\nuvicorn main:app --reload\n\n# Frontend\ncd frontend\nnpm install\nnpm run dev\n\n# Voice AI (optional, see voice/README.md for details)\ncd voice/server\nuv sync\nuv run bot.py\n```\n\n---\n\n## Database Setup\n\nThis project uses PostgreSQL for persistent memory via LangGraph's `PostgresSaver`.\n\n**Option 1 — Supabase (recommended, free)**\n1. Create a free account at [supabase.com](https://supabase.com)\n2. Create a new project\n3. Copy the connection string from Connect → **Session pooler, port 5432**\n4. Add it to your `.env` as `DATABASE_URL`\n\nUse the session pooler, not the transaction pooler on port 6543. `PostgresSaver`\nrelies on prepared statements, which the transaction pooler does not keep across\nqueries — the failures are intermittent rather than clean.\n\n**Option 2 — Local PostgreSQL**\n1. Install PostgreSQL locally\n2. Create a database: `CREATE DATABASE jobsearch_memory;`\n3. Add connection string to `.env`\n\n## Environment Variables\n\nSee `.env.example` for all required variables:\n\n```env\nGEMINI_API_KEY=\nLANGSMITH_API_KEY=\nLANGSMITH_TRACING=true\nLANGSMITH_PROJECT=\nLANGSMITH_ENDPOINT=https://eu.api.smith.langchain.com  # if outside US\nDATABASE_URL=postgresql://postgres:yourpassword@localhost:5432/jobsearch_memory\nALLOWED_ORIGINS=http://localhost:3000  # comma-separated list of browser origins allowed to call the API\n```\n\nFor Voice AI environment variables, see [`voice/README.md`](./voice/README.md).\n\n---\n\n## Deployment\n\nThe backend deploys to Render from the checked-in `render.yaml` (New → Blueprint).\nIt declares the build and start commands, pins Python, and lists the secrets to\nfill in from the dashboard. The frontend deploys to Vercel with **Root Directory\nset to `frontend`** and `NEXT_PUBLIC_API_BASE` pointing at the Render URL.\n\nTwo details cost real time if you get them wrong:\n\n- `NEXT_PUBLIC_API_BASE` takes **no trailing slash** — the client appends `/ask`.\n- `ALLOWED_ORIGINS` takes **no trailing slash** either. The browser's `Origin`\n  header is scheme, host and port only, and the match is exact. A stray `/` makes\n  every request fail CORS while the page itself loads fine, so the only symptom\n  is an error hidden in the browser console.\n\n---\n\n## CV Upload\n\nUpload your PDF resume and the agent automatically extracts your skills and finds matching remote jobs — no keywords needed.\n\n- `POST /upload` — accepts a PDF up to 5MB and returns matching job listings\n\nOnly the first five pages reach the model, and only to compress the CV into a\nsingle line of keywords. That line becomes the search query; the listings that\ncome back are then filtered and formatted by the same deterministic code that\nserves `/ask`.\n\n## n8n Automation (Optional)\n\nThe included `n8n_workflow.json` adds automated job alerts every 12 hours — no manual searches needed.\n\n### What the workflow does\n![n8n Workflow](assets/n8n_workflow.png)\n```\nSchedule (every 12h)\n  → Search Settings   (query + thread_id — the only node you edit)\n  → Wake node         (GET /docs; expected to fail on a sleeping instance)\n  → Wait 60s\n  → POST /ask         (returns the formatted digest)\n  → Send digest email via Gmail\n```\n![Email Digest](assets/email_digest.png)\n\nThe workflow holds **no filtering logic of its own**. Earlier versions duplicated the keyword matching across five JavaScript Code nodes, which meant every fix to the Python scoring had to be made twice — and never was. Now n8n only schedules, calls the API and mails the result, so the digest gets the same deduplication, mojibake repair and relevance ranking as every other interface.\n\n### Setup\n\n1. Import `n8n_workflow.json` into your n8n instance\n2. Configure your Gmail credentials in the Gmail node and set your address in `Send a message`\n3. In **Search Settings**, set `query` to whatever you are looking for\n4. In **Search Settings**, replace `YOUR_UNIQUE_THREAD_ID` with a value only you know — `python -c \"import uuid;print(uuid.uuid4())\"`. Anyone who learns this value can change that thread's filters, so do not publish it\n5. Point both HTTP nodes at your own backend (`http://host.docker.internal:8002` from Docker, `http://localhost:8002` without it)\n\n**Why the wake node and the 60-second wait.** On a sleeping free-tier instance the first request is answered with an immediate `503`, not a slow response — so a longer timeout does not help, and n8n's retry ceiling (5 tries, 5s apart) covers only ~25s of a ~50s cold start. The wake node exists purely to trigger the boot and is set to `On Error: Continue`, since failing is its normal outcome. Drop both nodes if your backend is always on.\n\n**Optional — persistent filters.** The thread outlives the workflow, so exclusions only need to be set once: call `POST /feedback` with the same `thread_id` and something like `no senior, no staff`. Every later run reuses them. The response footer lists the filters that were actually stored, which is worth reading — the extraction is done by an LLM and may generalise more than you intended.\n\n---\n\n## API Endpoints\n\n| Endpoint | Method | Calls an LLM | Description |\n|----------|--------|--------------|-------------|\n| `/ask` | POST | no | Search jobs with keywords + applies memory filters |\n| `/reset` | POST | no | Clear all stored filters and return the unfiltered results |\n| `/feedback` | POST | yes | Turn feedback (\"no MERN\", \"no senior\") into filter keywords. Input capped at 100 characters |\n| `/upload` | POST | yes | Upload PDF resume — agent extracts skills and finds matching jobs. Max 5MB, first 5 pages |\n| `/evaluate` | POST | yes | AI scoring of job listings. Requires the `x-api-key` header |\n\nEvery endpoint is rate limited to 10 requests per minute per IP.\n\n**On the LLM surface.** Search costs nothing: since the scoring became deterministic, `/ask` and `/reset` make no model calls at all. Three endpoints still do, and they are constrained differently. `/evaluate` accepts arbitrary text and is therefore closed — set `EVALUATE_TOKEN` in `.env` and send the same value as `x-api-key`; the exported workflow carries a `YOUR_EVALUATE_TOKEN` placeholder, and without it the endpoint answers 401. `/feedback` and `/upload` stay open because the public demo needs them, so they are bounded by size instead: 100 characters of feedback, 5MB and 5 pages of PDF. Bounded is not the same as free — if you deploy this somewhere that matters, put a budget alert on the API key.\n\n---\n\n## Evals\n\nThere are two, and they answer different questions. The LangSmith eval asks *how relevant are the results in a live market*, and a model judges the answer. The [Harbor](https://www.harborframework.com) eval asks *does the filter still do exactly what I think it does*, and nothing judges anything — the listings are frozen and the output is asserted.\n\n### Relevance — LangSmith, ~0.9\n\nThe agent scores **0.90 across 24 test cases**, measured 16 August. Two of those cases are not mine: they are real queries typed by strangers into the live demo — `Remote React jobs` and `Fullstack react remote jobs` — added after the logs showed both returning almost nothing. Both now score 1.0, which is its own finding: the metric is precision, so a response of two correct listings scores perfectly while the person who typed it went away and tried again. What they hit is the [known limitation](#known-limitation) below, and this eval cannot see it.\n\nBefore those two were added the figure was roughly 0.85, and before the judge was fixed it was **0.82 across 22 test cases**, measured 6 August against a control run of the previous code the same morning, which scored 0.76.\n\nTreat that as a range, not a reading. The same code scored **0.765** five days later without a line changing — the job boards had moved. A single case can swing half a point on its own: `rust` went from 1.0 to 0.5 across those five days, and the query has no moving parts in the code at all.\n\nSo changes here are judged against a **control run of the unchanged code on the same day**, never against a number from last week. Comparing a new run to a stored baseline measures the market as much as the change, and the market is louder.\n\nThe market was not the only thing moving. Two runs an hour apart returned the **same nine listings** for `Python developer` and scored them 0.889 and 0.222; `react developer` returned the same eight and scored 1.0 and 0.625. The aggregate barely moved — 0.82 both times — because the disagreements cancelled out, which is worse than an obviously unstable number. Chasing it down is written up under [Making the judge repeatable](#making-the-judge-repeatable).\n\nThe setup is a LangSmith dataset where each query carries a written description of what a good answer looks like, plus a Gemini 2.5 Flash judge that counts how many returned listings meet it. Score is relevant divided by returned, so padding a response with weak matches costs you.\n\nThe queries include narrow niches (`rust`, `blockchain solidity`), vague ones (`remote job`), one where the right answer is probably nothing at all (`COBOL mainframe developer`), and misspellings. Typos are not corrected on purpose. Search for `pyton developer` and you get nothing back; the reference answer says that is correct.\n\n```bash\npython eval_runner.py   # posts to localhost:8002/ask, change the port if your backend runs elsewhere\n```\n\n#### How it got there\n\nMost of the gain came from taking things out. Every step below was measured against the same 22 cases:\n\n| Change | Score |\n|---|---|\n| Starting point | 0.558 |\n| Stopped trusting RemoteOK's tags, matched on titles | 0.575 |\n| Removed the guaranteed minimum of 8 results | 0.679 |\n| Treated `developer` and `engineer` as meaningful words | 0.584, reverted |\n| Fixed the reference answers for the typo queries | 0.751 |\n| Fixed a missing comma in the generic-word list | 0.812 |\n\nFive days later the same code scored 0.765, so everything after that point is quoted against a control run of the unchanged code on the same morning:\n\n| Change | Control | Score |\n|---|---|---|\n| Let the query's generic word break ties in the title | 0.765 | 0.780 |\n| Scored descriptions on how often a word appears, and stopped truncating each source to ten | 0.76 | **0.82** |\n\nThe big jump came from deleting a rule that guaranteed at least 8 results. On a query like `rust` the agent would find one genuine match and then pad the list with seven listings that happened to mention the word somewhere in their body text. Three good results beat twelve mediocre ones.\n\nThe missing comma is worth a mention because Python never complained about it. Two adjacent string literals in a set silently became one, which quietly dropped `engineer` and `remote` from the generic-word list. It only surfaced because a nonsense query started returning listings with \"Remote\" in the title.\n\n#### What the number doesn't cover\n\nIt measures the first response only. Users narrow results by talking to the agent (\"no support roles\", \"no senior\"), and none of the 22 cases exercise that path, so day-to-day use is better than the number suggests.\n\nThat path is no longer untravelled. The scheduled digest reuses one thread, so its stored exclusions are applied twice a day against whatever the boards are advertising, and the MCP tool takes exclusions as an argument on every call. It now has a number too, though not from here — see below.\n\nIt also moves. The agent queries live job boards, so two runs an hour apart see different listings and individual cases wobble by a lot. The aggregate is the signal, not any single row.\n\n#### Making the judge repeatable\n\nScoring the same listings twice and getting 0.889 and 0.222 makes every comparison meaningless, so the run was treated as the thing under test rather than the agent. Three changes, in order of how much they bought:\n\n**Stop asking the model questions that have one answer.** Five queries are ones where returning nothing is acceptable — the two deliberate misspellings, and niches like `COBOL mainframe developer` that the boards may simply not be advertising. Those examples now carry an `empty_ok` flag, and an empty response short-circuits to 1.0 before the judge is called. Four of the five already said so in their reference text; the model had the instruction and applied it unevenly. A rule you can state in one sentence does not need a model.\n\n**Say what the grader is grading.** The instructions asked it to \"count how many of the given jobs match the reference criteria\" without ever saying whether a bad listing costs one point or voids the response. Both readings were live, which is exactly the 0.889/0.222 split. The prompt now states that grading covers only what was returned, that a rejected listing subtracts one and nothing more, and that criteria which are silent about a listing count in its favour.\n\n**Write reference criteria that can be decided.** `Python developer` said \"mid-level or senior, no Junior or Entry-level\" and left \"Senior General QA (Python)\" undecidable. `remote job` said any \"legitimate remote software role\" for a query that names no field at all. Ten of the twenty-two were rewritten as \"a listing is relevant when… and is not relevant when…\", resolved against the listings the agent actually returns.\n\nTwo consecutive runs then agreed on eighteen of the nineteen cases whose criteria had not changed. The one that still moved — `django backend developer` — turned out to be another silent spec: nothing said what to do with a \"Backend Engineer\" listing that never names a language. The fix was to decide, and write it down.\n\nThe number moved from 0.82 to roughly 0.85 along the way. That is not the agent improving; it is the criteria finally saying what was always meant, and it makes every earlier number incomparable.\n\n#### Known limitation\n\nOne title match is enough to admit a listing. That is fine when the distinctive word in a query is unambiguous, and it falls apart when it isn't: `data` pulls in Data Analysts, `wordpress` pulls in WordPress Support Specialists.\n\nThe obvious fix, requiring two matching words, was tried and rejected because it threw away correct results like `Software Engineer (Go, Python, TS)`. Measurement showed why it could never have worked. Specific terms like `sql`, `aws` and `pytorch` appear in **0%** of returned job titles, because titles say \"DevOps Engineer\", not \"DevOps Kubernetes AWS Engineer\". There is only ever one word to match on.\n\nHalf of it has since been chased down, from the other end. Dropping the generic word does not only fail to admit the right listings — it makes the ones already admitted indistinguishable. Search `AI engineer` and the query becomes `ai`, so an AI Engineer and an AI Sales Executive score identically; measured on 47 listings, 18 of 19 that passed the gate scored the same, and the cap then cut them by the order the four APIs happen to sit in `fan_out`. A generic word now adds a point when it appears in the title, which cannot admit anything on its own but does separate the role from the industry. Same-day control: 0.765 to 0.780, three cases up and none down.\n\nThe other half was the description, and it needed two changes that each look worthless alone. Relevance to the body text was scored by asking *whether* a word appeared, never *how often*, so a listing naming Python once ranked level with one naming it forty times: across 32 listings for `AI engineer`, that produced exactly one distinct score. Counting occurrences produces sixteen. On its own, that changed one case out of 22 — the other 21 came back identical, because a better order cannot help when there is nothing spare to order. Each source was also truncated to ten listings before anything read them, a leftover from when an LLM read the results and every listing cost money; for that same query Jobicy returned 100, of which 32 were relevant, and eight of the twelve best-scoring were never reachable. Removing the truncation alone had been tried the day before and scored as noise, for the mirror-image reason. Together, against a same-day control: 0.76 to 0.82, five cases up and none down.\n\nThe description's contribution is capped below the weight of a title match, so it can only reorder listings, never admit one — the same discipline as the generic-word point.\n\nThe admission rule itself is untouched: one title match still admits a listing, and a Support Specialist can still get in on the strength of the word alone. It just no longer outranks a real match by accident.\n\n`WordPress developer` is the one case in the set that none of this moved, and checking why was more useful than fixing it would have been. On the morning of 6 August the four boards returned **no listing at all** with \"wordpress\" in its title, so the agent correctly returned nothing — and scored 0.0 for it. That is the same failure as `COBOL mainframe developer`: the judge is unreliable when the right answer is an empty list, which is why the next eval here is a deterministic check rather than another query.\n\nThe old 0.90 baseline is gone. It was measured against LLM-based filtering on a different dataset and was never comparable to this one.\n\n### Filtering — Harbor, asserted\n\nBoth problems above share a root: the number moves for reasons that have nothing to do with the code. Live listings shift hourly, and the same input scored 0.333 and then 0.167 because a model was doing the scoring.\n\nSo the filter is measured somewhere else. One capture of all four job boards is frozen into the repository, the container runs with its network disabled, and the expected result is five listings written out by hand. Same input, same number, every time.\n\n```bash\nuv tool install harbor\n\nPYTHONPATH=$(pwd) harbor run -p evals -i \"*filter-exclusion-senior*\" \\\n  -a evals.harbor_agents.pipeline_agent:PipelineAgent \\\n  -e docker -o evals/jobs \\\n  --extra-docker-compose evals/configs/no-network.yaml \\\n  --job-name local -y\n\npython evals/check_reward.py evals/jobs/local\n```\n\nDocker is the only requirement. No API key, because nothing in this path calls a model — the harness is the repository's own MCP tool, running unmodified against fixture files instead of the internet. That is also why it can sit on every pull request: it costs a runner minute and no tokens.\n\nThe query is `python developer`, excluding `senior`, `game` and `canonical`. Three terms rather than one, so that both halves of the filter are exercised: `senior` through the title-only seniority rule, `canonical` through the full-text rule, where the word appears in descriptions and in no title at all. Five listings survive, five don't. Leaving one in and dropping one too many both score zero.\n\nThe expected set is written by hand, which is the whole point. A verifier that recomputed it by calling `filter_jobs` would be comparing the code against itself and would pass forever.\n\nIt bites: collapsing the filter so every keyword is matched against titles only takes the score from 1.0 to 0.0, and the failing test names the two listings that leaked through. Notably, deleting `senior` from the seniority list does **not** — the word would still match the same title as an ordinary keyword. Controls that fail to fail are worth knowing about.\n\nWhat it does not cover: one query, one moment in the market, and no listing in the capture carries \"senior\" in its description alone — so the rule that seniority is judged on titles only is never tested in the one shape that separates it from ordinary matching. The fixtures are real captured data and were not edited to manufacture that case.\n\nDesign notes and the full reasoning live in [`evals/specs/`](./evals/specs).\n\n### Next\n\nRelevance is the remaining defect, and it now has somewhere to be measured. The scaffolding above is reusable as-is: freeze a capture, label which listings genuinely answer the query, assert. Whatever the fix turns out to be, it has to lean on descriptions — the measurement above shows there is rarely a second title word to ask for.\n\n## Project Structure\n\n```\njob-search-agent/\n├── agent.py           # LangGraph agent — fan-out, scoring, filtering\n├── voice_agent.py     # VoiceSession — stateless wrapper around the same graph\n├── mcp_server.py      # MCP server (stdio) — same graph, one read-only tool\n├── main.py            # FastAPI backend + all endpoints\n├── eval_runner.py     # LangSmith evaluation pipeline\n├── requirements.txt\n├── .env.example\n├── n8n_workflow.json  # n8n automation workflow\n├── evals/             # Harbor eval — frozen fixtures, deterministic verifier\n│   ├── filter-exclusion-senior/   # the task: instruction, environment, tests\n│   ├── harbor_agents/             # adapter that runs the pipeline as the agent\n│   ├── specs/                     # design notes: harness, environment, task\n│   └── check_reward.py            # turns a Harbor job into a build verdict\n├── frontend/          # Next.js 15 frontend\n└── voice/             # Voice AI interface (Pipecat) — see voice/README.md\n    └── server/\n        ├── bot.py\n        └── langgraph_processor.py\n```\n",
  "bytes": 30030,
  "sha": "bef43d2fd3b41e1defe69e5070ddf8293d1736b6d00e81a39a57ce7e9058a65f",
  "repo_slug": "tsouk88/job-search-agent",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/okf_tsouk88_job_search_agent_openwiki_index__9c30000e/readme"
}