{ "markdown": "# RAG Agent: question answering over PDF manuals\n\nUpload PDFs, ask questions in any language, get grounded answers together\nwith the exact excerpts they came from. Built for an ML Engineering\ninterview challenge ([brief](docs/challenge.pdf)), and built **eval-first**:\nevery retrieval and prompt change is measured against a hand-authored\ngolden dataset before it is kept. From day one the repo has carried a\n**wiki-style knowledge base for the AI coding agents** that helped develop\nit. The [documentation section](#documentation-a-wiki-for-the-agents-that-built-this)\nexplains how it is organized.\n\n![Python 3.12+](https://img.shields.io/badge/python-3.12%2B-3776AB?logo=python&logoColor=white)\n![FastAPI](https://img.shields.io/badge/FastAPI-0.141-009688?logo=fastapi&logoColor=white)\n![Qdrant](https://img.shields.io/badge/Qdrant-1.19-DC244C)\n![Models](https://img.shields.io/badge/models-OpenAI%20%2B%20Gemini%20via%20PydanticAI-412991?logo=openai&logoColor=white)\n![Docker Compose](https://img.shields.io/badge/run-docker%20compose-2496ED?logo=docker&logoColor=white)\n![TDD + pyright](https://img.shields.io/badge/quality-TDD%20%C2%B7%20pyright%20standard-brightgreen)\n\n## Quickstart\n\nYou need Docker, an OpenAI API key and a Gemini API key. Nothing else.\n\n```bash\ngit clone && cd rag-agent\ncp .env.example .env # put your keys in OPENAI_API_KEY and GEMINI_API_KEY\nmake up # builds the image, starts Qdrant + API in the foreground\n```\n\nWith the stack running, send any PDF to `POST /documents`. Repeat `-F`\nto upload several at once. The repo ships four real motor manuals in\n`case_files/` (WEG and Baldor, Portuguese and English) if you want\nsomething to try:\n\n```bash\ncurl -s -F \"files=@case_files/LB5001.pdf\" http://localhost:8000/documents\n```\n\nThe response reports how many documents and chunks were indexed, and the\n`make up` terminal logs each file's progress while it is ingested. Then\nask:\n\n```bash\ncurl -s -X POST http://localhost:8000/question \\\n -H 'Content-Type: application/json' \\\n -d '{\"question\": \"What grease should I use to relubricate the motor bearings?\"}'\n```\n\nInteractive OpenAPI docs live at .\n\n## The API\n\n| Endpoint | Request | Response |\n| ----------------- | ----------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |\n| `POST /documents` | `multipart/form-data`, one or more PDFs under `files` | `{\"message\", \"documents_indexed\", \"total_chunks\"}` |\n| `POST /question` | `{\"question\": \"...\"}` | `{\"answer\", \"references\": [verbatim excerpts the answer cites]}` |\n| `GET /health` | — | `{\"status\", \"vector_store\", \"indexed_chunks\", \"llm_model\", \"embedding_model\"}`; `503` when Qdrant is down |\n\nConfiguration mistakes fail before any request: `make check-env` refuses an\nempty key, and the API validates the models, the keys and the vector store\nat startup, so a wrong `.env` shows up in the `make up` terminal as\n`startup failed: …`. When something fails later, `GET /health` says which\ndependency is down. Re-uploading a file is idempotent: chunk ids are\ncontent-addressed, so the index never accumulates duplicates.\n\n### Example requests and responses\n\nReal outputs from the running stack (`openai:gpt-5-mini`, the four manuals\nfrom `case_files/` indexed, captured 2026-09-04). Each reference is a\npassage the model quoted verbatim from a page it read, verified by\ncontainment before it is returned. It is never a whole page, never invented.\n\n**The Quickstart question**, answered from the Baldor manual:\n\n```json\n{ \"question\": \"What grease should I use to relubricate the motor bearings?\" }\n```\n\n```json\n{\n \"answer\": \"Baldor motors are normally pregreased with Polyrex EM (Exxon Mobil); if other greases are preferred, check with a local Baldor Service Center. Also: “Keep grease clean. Mixing dissimilar grease is not recommended.”\",\n \"references\": [\n \"Baldor motors are pregreased, normally with Polyrex EM (Exxon Mobil). If other greases are preferred, check with a local Baldor Service Center for recommendations.\",\n \"Caution: Keep grease clean. Mixing dissimilar grease is not recommended.\"\n ]\n}\n```\n\n**The same question in Portuguese.** The answer follows the language of\nthe question, and the references keep the words of the source (an English\nmanual). Reading and answering are separate concerns:\n\n```json\n{ \"question\": \"Qual graxa devo usar para relubrificar os rolamentos do motor?\" }\n```\n\n```json\n{\n \"answer\": \"Use, preferencialmente, Polyrex EM (Exxon Mobil). Se optar por outra graxa, consulte um Centro de Serviço Baldor; e evite misturar graxas diferentes.\",\n \"references\": [\n \"Baldor motors are pregreased, normally with Polyrex EM (Exxon Mobil). If other greases are preferred, check with a local Baldor Service Center for recommendations.\",\n \"Caution: Keep grease clean. Mixing dissimilar grease is not recommended.\"\n ]\n}\n```\n\nWhen the indexed documents do not support an answer the agent refuses in\nthe question's language and returns an empty `references` list. It never\ninvents a source:\n\n```json\n{ \"question\": \"Qual é a capital da Austrália?\" }\n```\n\n```json\n{\n \"answer\": \"Desculpe, os documentos fornecidos não contêm essa informação.\",\n \"references\": []\n}\n```\n\nQuestions take a few seconds each (mean 6.4 s, p95 10.4 s over the 93-case\neval at 8 workers): one or two LLM calls plus retrieval, with the model\nwriting out the passages it quotes. The model reasons at low effort by\ndefault (`LLM_THINKING`).\n\n## How it works\n\n```mermaid\nflowchart LR\n subgraph ingest[\"POST /documents (write path)\"]\n P[PDF bytes] --> X[\"PdfExtractor
pymupdf4llm, page markdown
+ TOC breadcrumbs\"]\n X --> C[\"chunker
one chunk per page,
embedded as its blocks\"]\n C --> E[\"EmbeddingModel
pydantic-ai Embedder:
OpenAI or Google\"]\n E --> Q[(\"Qdrant
one point per chunk
payload = provenance\")]\n end\n subgraph ask[\"POST /question (read path)\"]\n U[question] --> R[\"Retriever
seed top-k\"]\n R --> A[\"AgentService
bounded tool loop\"]\n A --> L[\"LLM port
PydanticAI direct
structured reply\"]\n L -. \"query_knowledge(query)\" .-> R\n A --> O[\"answer +
quoted passages\"]\n end\n R --> Q\n```\n\nThe codebase is a **ports & adapters \"lite\"**: a framework-free domain\n(`src/domain`: dataclass entities, `typing.Protocol` ports, two domain\nservices) surrounded by adapters per pipeline stage, wired in one\ncomposition root at the API edge. The point is cheap experiments: swapping\nthe PDF extractor, the embedder, the retrieval strategy or the LLM provider\nis a one-line change, and the evals decide whether it stays.\n\n### Ingestion: from PDF bytes to searchable chunks\n\n`pymupdf4llm` extracts each page as markdown. Two cleaning passes wrap\nthat extraction; both replaced a naive baseline that scored worse:\n\n- **Font repair (before extraction).** Some PDFs embed fonts with no\n Unicode map, so those pages decode as runs of `�` instead of real text.\n We rebuild the missing map from Arial's standard glyph order before\n extraction runs. On one manual this took the garbled character count\n from 71,618 down to 41, with no OCR involved.\n- **Page cleaning (after extraction).** Running headers, page numbers and\n dot leaders are stripped before anything reaches the embedder, so they\n don't compete with real content for similarity.\n\nChunking stayed deliberately simple: **one chunk per page**, no\nfixed-size splitting, no overlap. Underneath that, each page is also\nsplit into small units (paragraphs and table rows), and each unit gets\nits own embedding. Qdrant stores all of a page's unit vectors on **one\nmultivector point**, scored by its best-matching unit (MaxSim). That\nmeans a page is *found* by its most specific sentence, but *returned*\nwhole, so the model gets full context without losing precision. It's\nsmall-to-big retrieval, without needing a separate parent index.\n\nEvery one of these choices replaced something that measured worse. The\n[scoreboard](#scoreboard) below shows what each one bought.\n\n### Retrieval: from a question to a grounded answer\n\nA question first gets a deterministic top-k search (`RETRIEVAL_K=5`,\nMaxSim over the stored multivectors). The retrieved chunks are rendered\nas XML in the model's system prompt, one `` per page. Here is a\ntrimmed example: this is what the model saw before answering the\nQuickstart's grease question above.\n\n```xml\n\n \n Baldor motors are pregreased, normally with Polyrex EM (Exxon Mobil). If\n other greases are preferred, check with a local Baldor Service Center\n for recommendations.\n\n Caution: Keep grease clean. Mixing dissimilar grease is not recommended.\n \n\n```\n\n(A `
` element is added above `` for pages that have one:\neither from the PDF's own outline, or from a markdown heading.)\n\nIf the seed chunks aren't enough, the model can call a `query_knowledge`\ntool, up to 3 rounds, to search again with a reformulated query\n(synonyms, the other language, a more technical term) before giving up.\n\nThe final turn is a **provider-enforced structured reply** with three\nfields: `answer`, `has_answer`, and `citations`. Citations are passages\nthe model must copy **verbatim** from the `` it read, character for\ncharacter, never paraphrased or translated. This is exactly how the two\nreferences in the grease example above were produced. We do not trust\nthe model's word for where a quote came from: every citation is resolved\nafterwards by checking, line by line, that it is actually contained in a\nchunk the model saw. A citation that fails that check is dropped rather\nthan guessed at, so `references` never carries invented or approximate\ntext, or a whole page when a sentence would do. The prompt is a\ndeliberate, reviewed artifact in `src/domain/services/prompts.py`. The\ndesign behind this citation scheme, including what it costs, is in the\n[answer layer](#answer-layer) below.\n\n```\nsrc/domain/ entities, ports (Protocols), AgentService, IngestionPipelineService, prompts (pure Python)\nsrc/ingestion/ pymupdf4llm extractor, chunker\nsrc/retrieval/ embedder (OpenAI or Gemini), Qdrant multivector store, VectorRetriever\nsrc/llm/ PydanticAiLLM adapter (structured output, function-derived tools)\nsrc/api/ FastAPI routes + composition root\nsrc/evaluation/ the eval harness (loader, matching, metrics, report, CLI)\nevals/ golden dataset (93 cases) and committed results\ntests/ domain services against fakes, adapters, routes and seam integration\ndocs/ the knowledge bundle (see below)\n```\n\n## Eval-first\n\nAccuracy is measured, not assumed.\n\n- **Golden dataset**: 93 hand-authored question → ideal-answer cases over\n the four manuals ([overview](evals/golden/golden-dataset.md)): operator\n and technical personas, table and figure lookups, cross-lingual cases\n (English manuals asked in Portuguese and vice-versa), and 8 unanswerable\n controls. Ground truth is verbatim excerpts plus page, never chunk ids,\n so it survives any change in chunking.\n- **Metrics.** Deterministic **gates** decide experiments: recall@5,\n hit_rate@5, MRR@5. Diagnostics (precision@5, per-slice breakdowns by\n document, language, persona and category) explain the numbers but never\n gate.\n- **The rule**: any change to chunking, embedding, retrieval or prompting\n ships with a before/after run committed to `evals/results/`.\n\n```bash\nmake install # local venv, Python >= 3.12\nmake eval label=my-experiment # runs against the eval collection, prints deltas vs the last run\nmake eval-fresh label=reindexed # drop the eval collection and re-ingest first (after ingestion changes)\nmake eval-answers label=agent # adds the answer layer: every case through the agent (LLM calls, a few minutes)\n```\n\n### Scoreboard\n\nThe table is alive: every kept experiment adds a row, with its committed\nresults file as evidence. The goal is to leave the best numbers we can\nreach here.\n\n| Iteration | Date | Results | recall@5 | hit_rate@5 | MRR@5 |\n| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ------------------------------------------------------------------------------------------------ | -------: | ---------: | -------: |\n| **Baseline**: pymupdf4llm extraction, fixed 1000/200 chunks, `text-embedding-3-small`, top-5 vector search | 2026-09-01 | [`20260901-190240-baseline.json`](evals/results/20260901-190240-baseline.json) | 0.65 | 0.66 | 0.60 |\n| **Font repair**: fonts lacking a ToUnicode map get one from Arial's glyph order before extraction; CESTARI stops indexing `�` (no OCR) | 2026-09-02 | [`20260902-035239-font-repair.json`](evals/results/20260902-035239-font-repair.json) | 0.78 | 0.80 | 0.70 |\n| **Page cleaning**: running headers, page numbers, dot leaders and picture-text markers stripped | 2026-09-02 | [`20260902-035640-page-cleanup.json`](evals/results/20260902-035640-page-cleanup.json) | 0.80 | 0.81 | 0.71 |\n| **Structured chunks**: markdown blocks packed to ~1200 chars, sentences and tables never split, sections from headings where the PDF has no outline | 2026-09-02 | [`20260902-041707-structured-chunks.json`](evals/results/20260902-041707-structured-chunks.json) | 0.79 | 0.81 | 0.71 |\n| **Contextualized embeddings**: document, section and heading prefixed to the text the embedder sees; stored chunk unchanged | 2026-09-02 | [`20260902-041913-embed-context.json`](evals/results/20260902-041913-embed-context.json) | 0.81 | 0.83 | 0.76 |\n| **Page chunks, small units**: one chunk per page; its paragraphs and table rows are embedded as separate vectors on the same Qdrant point (MaxSim), so a specific value is found and the whole page is read | 2026-09-02 | [`20260902-045635-page-multivector.json`](evals/results/20260902-045635-page-multivector.json) | 0.86 | 0.86 | 0.79 |\n| **Multilingual embedder**: `EMBEDDING_MODEL=google:gemini-embedding-001` (3072 dims) instead of `text-embedding-3-small`; six of the eleven Portuguese-question-over-English-manual misses recovered | 2026-09-02 | [`20260902-052352-gemini-embedding.json`](evals/results/20260902-052352-gemini-embedding.json) | **0.95** | **0.95** | **0.91** |\n\nGates are computed over the 83 gated cases (93 minus 8 unanswerable\ncontrols and 2 image-only diagnostics).\n\n#### Answer layer\n\nSame dataset, the whole `/question` path (seed retrieval → `gpt-5-mini`\n→ structured reply), scored deterministically: fact recall over the\ncases' `expected_facts`, citation precision and recall over the cited\n`(document, page)` pairs, refusal rate over the 8 unanswerable controls.\nNo LLM judge. Red cases are read by hand from the per-case JSON.\n\n| Iteration | Results | fact recall | citation precision | citation recall | refusals | latency (mean) | cost |\n| --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | ----------: | ------------------: | ---------------: | -------: | --------------: | ----------: |\n| **Chunk ids**: the model cites a chunk id per claim; provider-default reasoning effort | [`20260902-202721-agent-tool-on.json`](evals/results/20260902-202721-agent-tool-on.json) | **0.93** | 0.70 | **0.92** | 6/8 | 11.7 s | ≈ $0.22 |\n| **`query_knowledge` tool off**: same as above, the retrieval tool disabled; kept **on** going forward, it recovers cases the seed alone misses | [`20260902-203011-agent-tool-off.json`](evals/results/20260902-203011-agent-tool-off.json) | 0.92 | 0.73 | 0.91 | 7/8 | 10.5 s | ≈ $0.18 |\n| **Verbatim quotes**: citations become passages copied from ``, resolved by containment ([Decision 0013](docs/decisions/0013-citations-as-quotes.md)) | [`20260902-221750-citations-as-quotes.json`](evals/results/20260902-221750-citations-as-quotes.json) | 0.92 | 0.78 | 0.90 | 7/8 | 16.0 s | ≈ $0.29 |\n| **Low reasoning effort**: `LLM_THINKING=low` instead of the provider default | [`20260903-010828-thinking-low.json`](evals/results/20260903-010828-thinking-low.json) | 0.91 | 0.79 | 0.86 | 7/8 | **5.9 s** | $0.18 |\n| **Language reminder in the prompt**: explicit rule to answer in the question's language regardless of the chunks' language | [`20260904-033639-prompt-language-reminder.json`](evals/results/20260904-033639-prompt-language-reminder.json) | 0.91 | **0.81** | 0.90 | 6/8 | 6.4 s | **$0.14** |\n\nCosts marked ≈ are computed after the fact from each run's recorded\ntokens with the same price table; the last row's is recorded by the run\nitself (embedding calls excluded, three orders of magnitude smaller).\n\nFact recall and citation recall dip a little, 0.01 to 0.02, from the\nfirst row to the last. That's inside the ±0.03 run-to-run noise this\n93-case dataset carries (see [Decision 0013](docs/decisions/0013-citations-as-quotes.md)).\nCitation precision, latency and cost move well past that noise: verbatim,\ncontainment-checked citations raised precision from 0.70 to 0.81, and\ndropping the reasoning effort to `low` cut mean latency by more than\nhalf, at no real cost to the other metrics.\n\n## Engineering practices\n\n- **TDD for the code itself.** Distinct from the evals above, which\n measure whether the system answers well, the test suite checks that\n each module does what it was designed to do. Every module and every\n seam between modules was written red-green-refactor: domain services\n against fakes of their ports, adapters on their own, routes with\n dependency overrides, seams on an in-memory Qdrant. External services\n are faked here; their real behavior is the evals' job. `make test` runs\n the suite in seconds.\n- **Typed.** `make typecheck` runs pyright in `standard` mode: zero\n errors, no blanket ignores.\n- **Comment-free code, documented decisions.** Rationale lives in the\n knowledge bundle next to the code it explains, not in comments that\n drift.\n- **Runs on Python 3.12, 3.13 and 3.14** with the same pinned\n requirements; the image ships 3.14.\n\n## Configuration\n\nCopy `.env.example` to `.env`. Only the two API keys are required.\n\n| Variable | Default | What it does |\n| ------------------------- | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `OPENAI_API_KEY` | — | The primary LLM (`gpt-5-mini`) and the OpenAI embedding models. **Required.** |\n| `GEMINI_API_KEY` | — | Embeddings (`gemini-embedding-001`) and the fallback LLM. **Required.** |\n| `LLM_MODEL` | `openai:gpt-5-mini` | Any PydanticAI model string; `openai:` is the Responses API, `openai-chat:` Chat Completions. |\n| `LLM_FALLBACK_MODEL` | `google:gemini-3.5-flash` | Tried when the primary model fails with a provider error (4xx, 5xx, connection). Blank disables the fallback. |\n| `LLM_THINKING` | `low` | Reasoning effort of the LLM (`minimal`, `low`, `medium`, `high`, `xhigh`, `off`; blank keeps the provider default). Applies to the primary and the fallback model. At the provider default reasoning tokens were 85–94 % of the output and most of the latency; `low` cut the mean answer time by more than half on the eval. |\n| `EMBEDDING_MODEL` | `google:gemini-embedding-001` | `google:gemini-embedding-001` (the measured best, see the scoreboard), `openai:text-embedding-3-small` or `openai:text-embedding-3-large`; changing the model requires re-indexing (delete the collection, the store refuses a mismatched one). |\n| `RETRIEVAL_K` | `5` | Chunks per retrieval (seed and tool calls). |\n| `AGENT_MAX_TOOL_ROUNDS` | `3` | Cap on `query_knowledge` rounds per question; `0` disables the tool. |\n| `QUERY_KNOWLEDGE_ENABLED` | `true` | Offer the retrieval tool to the model at all. |\n| `QDRANT_URL` | `http://localhost:6333` | Host-side default; inside compose the API talks to the `qdrant` service. |\n| `QDRANT_COLLECTION` | `chunks` | Production collection. |\n| `EVAL_QDRANT_COLLECTION` | `eval_chunks` | Separate collection the eval harness indexes and reads. |\n\nThe LLM has a provider fallback: when the primary model fails with a\nprovider error, the same request is retried on `LLM_FALLBACK_MODEL`\n(PydanticAI's `FallbackModel`); if every model fails the API answers 502\nnaming each model's error. The OpenAI and Google extras are both installed.\n\n## Documentation: a wiki for the agents that built this\n\nThis repository was developed with AI coding agents, and we chose from the\nvery first commit to sustain it with a **wiki-style knowledge base written\nfor those agents**. The whole repo is one knowledge bundle in the\n[Open Knowledge Format](docs/okf-spec.md): every `.md` file carries typed\nfrontmatter, module knowledge sits next to the module's code, and every\nchange to the bundle is logged. It holds what the code cannot say: why\nthings are shaped this way, what was rejected, what was measured. That\nway each new agent session (and each human reader) starts with the same\ncontext instead of reverse-engineering it from git history.\n\nThe bundle is curated: the owner is its editor, approves every new\nconcept before it is written, and stamps what he has reviewed\n(`verified`). Agents propose, humans decide.\n\nSome of the documentation worth a look:\n\n- [`docs/architecture.md`](docs/architecture.md): the operating map of\n the codebase (shape, rules, how to extend it).\n- [`docs/decisions/`](docs/decisions/index.md): the decision records,\n each with context, alternatives rejected and consequences.\n- Module notes next to the code, such as\n [`src/ingestion/ingestion.md`](src/ingestion/ingestion.md) and\n [`src/evaluation/evaluation.md`](src/evaluation/evaluation.md).\n- [`log.md`](log.md): the bundle's changelog, newest first. The story of\n the project in one page.\n", "bytes": 27512, "sha": "7eec00a7ea37057ef0a5279c2fc2e36e83f8a898b243d83a4873d88378467bef", "repo_slug": "vipigal/rag-ai-agent", "fonte": "repo", "truncated": false, "api": "https://agentalog.com/api/listings/okf_vipigal_rag_ai_agent_index_md_3d3b337c/readme" }