{
  "markdown": "# myIR — Information Retrieval Laboratory\n\nmyIR is a Java 25 laboratory for rebuilding information-retrieval and web-ingestion systems from first principles. It combines a classical lexical engine, sparse-vector retrieval, concurrent crawling, product extraction, and a site-to-publication exporter.\n\nThe project is intentionally educational, but its boundaries are designed to support serious experiments. It does not attempt to replace Lucene or Elasticsearch.\n\n## Current Capabilities\n\n- Tokenization and composable normalization for English and Spanish.\n- In-memory corpus and positional inverted index.\n- Immutable corpus and index snapshots for consistent search reads.\n- Binary, TF-IDF, and BM25 ranking.\n- Sparse vectors, vocabulary-backed dimensions, TF/TF-IDF weighting, and cosine similarity.\n- Static HTML crawling with JDK `HttpClient` and Jsoup.\n- Optional Playwright-backed dynamic page fetching.\n- Queue-based breadth-first traversal using virtual threads.\n- URI canonicalization, URL filtering, metadata extraction, and sitemap parsing.\n- Page classification and product discovery for generic and WordPress/WooCommerce pages.\n- Site mirroring with portable JSON manifests.\n- Asset download and local link rewriting for PDF publication.\n- PDF, Markdown, and EPUB publication from a new or existing mirror.\n\n## Module Architecture\n\nThe project is a three-module Maven reactor. Every module is also a named JPMS module.\n\n```mermaid\ngraph TD\n    APP[\"codex-ir-app<br/>codex.ir.app\"]\n    WEB[\"codex-ir-web<br/>codex.ir.web\"]\n    CORE[\"codex-ir-core<br/>codex.ir.core\"]\n\n    APP --> WEB\n    APP --> CORE\n    WEB --> CORE\n```\n\n| Maven module | JPMS module | Responsibility |\n|---|---|---|\n| `codex-ir-core` | `codex.ir.core` | Domain-neutral IR engine: documents, indexing, snapshots, ranking, search, and sparse vectors |\n| `codex-ir-web` | `codex.ir.web` | Reusable ingestion and web primitives: crawling, canonicalization, classification, metadata, and product extraction |\n| `codex-ir-app` | `codex.ir.app` | Executable demos, discovery workflows, and the site-exporter application |\n\nDependency direction is `app -> web -> core`. Application-specific publication code stays in `codex-ir-app`; web concepts do not leak into `codex-ir-core`.\n\nThe `module-info.java` files are authoritative for JPMS visibility. In particular, fetcher implementations, crawler internals, and web utilities are not exported from `codex.ir.web`.\n\n## Core Engine\n\n### Document Processing and Indexing\n\n```mermaid\nflowchart TD\n    INPUT[\"Raw Document\"] --> PRE[\"DocumentPreprocessor\"]\n    PRE --> RESOLVE[\"Use structured field values, or rawContent as fallback\"]\n    RESOLVE --> TOK[\"Tokenizer + Normalizer\"]\n    TOK --> META[\"Normalized content + derived metadata\"]\n    META --> PIPE[\"PipelineIndexer\"]\n    PIPE --> LEX[\"Lexical stage\"]\n    PIPE --> VEC[\"Vector stage\"]\n    LEX --> CORPUS[\"Mutable Corpus\"]\n    LEX --> INDEX[\"Mutable InvertedIndex\"]\n    VEC --> STORE[\"Vocabulary + DocumentVectorStore\"]\n```\n\n`Document` is the central record. It preserves raw and normalized text, structured fields, and derived `DocumentMetadata`. When fields contain usable values, preprocessing aggregates those values instead of `rawContent`; blank fields fall back to raw content.\n\nMain factory pairs include:\n\n| Contract | Factory | Implemented strategies |\n|---|---|---|\n| `Corpus` | `Corpora` | Eager or debounced in-memory statistics |\n| `InvertedIndex` | `InvertedIndexes` | Positional in-memory postings |\n| `Indexer` | `Indexers` | Lexical, vector, or combined pipeline |\n| `Tokenizer` | `Tokenizers` | Whitespace tokenization |\n| `Normalizer` | `Normalizers` | Lowercase, accent folding, punctuation trimming, stop words, chains |\n| `Ranker` | `Rankers` | Binary, TF-IDF, BM25 |\n| `Searcher` | `Searchers` | Lexical and sparse-vector search |\n| `Vocabulary` | `Vocabularies` | Shared in-memory term dimensions |\n| `Vectorizer` | `Vectorizers` | Sparse document vectors |\n| `Similarity` | `Similarities` | Sparse cosine similarity |\n| `DocumentVectorStore` | `VectorStores` | In-memory vector storage |\n| `DocumentWeighter` | `Weighters` | Term frequency and TF-IDF |\n\n### Snapshot Read Boundary\n\nIngestion writes to mutable `Corpus` and `InvertedIndex` instances. Search and ranking consume immutable point-in-time views:\n\n```mermaid\nflowchart LR\n    INGEST[\"Indexing round\"] --> CORPUS[\"Corpus\"]\n    INGEST --> INDEX[\"InvertedIndex\"]\n    CORPUS --> CS[\"CorpusSnapshot\"]\n    INDEX --> IS[\"IndexSnapshot\"]\n    CS --> RANK[\"Ranker\"]\n    IS --> RANK\n    CS --> SEARCH[\"Searcher\"]\n    IS --> SEARCH\n```\n\nThis makes publication of a search-visible state explicit and prevents readers from observing a partially updated index.\n\n### Retrieval Paths\n\nLexical retrieval tokenizes and normalizes a query, resolves postings from an `IndexSnapshot`, scores matching documents with binary, TF-IDF, or BM25 ranking, and returns descending `SearchResult` values.\n\nVector retrieval weighs normalized query terms, creates a sparse query vector using the shared vocabulary, compares it with vectors in `DocumentVectorStore`, and returns matches above the configured similarity threshold.\n\nAll core storage remains in memory by design.\n\n### Score Explanation\n\n`SimpleSearcher` implements `ExplainableSearcher`, a capability interface that adds `explain(query, documentId) → Optional<ScoreExplanation>`. The explanation carries a `List<TermScoring>` — one entry per matched query term — exposing the formula intermediates (TF, IDF, normalization, field boost) that produced the final score.\n\nCheck `instanceof` before calling `explain`; `VectorSearcher` does not implement the interface.\n\n```java\nSearcher searcher = Searchers.lexical(indexSnapshot, corpusSnapshot,\n        tokenizer, normalizer, Rankers.bm25(corpusSnapshot, indexSnapshot));\n\nList<SearchResult> results = searcher.searchDetailed(\"java search\");\n\nif (searcher instanceof ExplainableSearcher es) {\n    results.stream().findFirst().ifPresent(r ->\n        es.explain(\"java search\", r.documentId()).ifPresent(explanation -> {\n            System.out.println(\"Score: \" + explanation.score());\n            for (TermScoring ts : explanation.contributions()) {\n                System.out.printf(\"  %-12s base=%.4f boost=%s contribution=%.4f%n\",\n                        ts.term(), ts.base(),\n                        ts.fieldBoost().map(fb -> String.format(\"%.2f\", fb.boostFactor())).orElse(\"none\"),\n                        ts.contribution());\n            }\n        })\n    );\n}\n```\n\n`explain` uses the same tokenization and normalization pipeline as `searchDetailed`, so the score it reports is numerically identical to the `SearchResult.score()` for any matching document.\n\n## Web Ingestion\n\n`codex-ir-web` exposes reusable crawling and extraction contracts while keeping implementations under internal, non-exported packages.\n\n```mermaid\nflowchart TD\n    SEED[\"Seed URI(s)\"] --> CANON[\"UriCanonicalizer\"]\n    CANON --> STRATEGY[\"WebPageSourceStrategy\"]\n    STRATEGY --> STATIC[\"Static HTML fetcher\"]\n    STRATEGY -. optional .-> DYNAMIC[\"Playwright dynamic fetcher\"]\n    STATIC --> PAGE[\"WebPage\"]\n    DYNAMIC --> PAGE\n    PAGE --> META[\"Metadata + classification\"]\n    PAGE --> PRODUCT[\"Product discovery\"]\n    PAGE --> MAP[\"DocumentMapper\"]\n    MAP --> IR[\"Core Indexer\"]\n```\n\nThe default traversal is queue-based breadth-first crawling with configurable depth, page count, domain policy, request delay, concurrency, content types, timeouts, and path/domain restrictions. Sitemap and robots parsing are implemented as reusable crawler internals. The site-exporter command currently starts from normal site traversal; it does not automatically switch to sitemap discovery.\n\nStatic fetching is the default path. `WebPageFetchers.dynamicHtml()` provides Playwright rendering, but applications must select that fetcher explicitly.\n\n## Site Exporter\n\nThe site exporter lives under `codex.apps.siteexporter` because it is an application workflow, not reusable IR or crawler infrastructure.\n\n```mermaid\nflowchart LR\n    SOURCE[\"Crawl or existing mirror\"] --> MIRROR[\"HTML mirror\"]\n    MIRROR --> MANIFEST[\"mirror-manifest.json\"]\n    MANIFEST --> DRIVER[\"PublicationDriver\"]\n    DRIVER --> PDF[\"PDF\"]\n    DRIVER --> MD[\"Markdown\"]\n    DRIVER --> EPUB[\"EPUB 3\"]\n    MIRROR --> ASSETS[\"Assets + link rewriting\"]\n    ASSETS --> PDF\n```\n\n### Mirror Contract\n\n`SiteMirrorService` writes one local HTML file per successful page and records every processed page in `mirror-manifest.json`. The manifest is read and written with Jackson through `ManifestReader` and `ManifestWriter`.\n\nImportant manifest guarantees:\n\n- `localHtmlPath` is relative to the manifest directory and uses portable `/` separators.\n- Successful entries resolve to local HTML files.\n- Failed writes remain visible as failed entries.\n- Counts are derived from the page list when the manifest is built or read.\n- `depth` remains `null` when the traversal source does not expose depth.\n- `discoveredOrder` provides deterministic publication order for a given source emission order.\n\n### Publication Formats\n\n| Format | Driver | Asset processing | Current behavior |\n|---|---|---|---|\n| PDF | `PdfPublicationDriver` | Yes | Downloads assets, rewrites local links, renders pages with OpenHTMLToPDF, and merges them with PDFBox |\n| Markdown | `MarkdownPublicationDriver` | No | Extracts readable text into one `.md` document and optional per-page Markdown files |\n| EPUB | `EpubPublicationDriver` | No | Produces an EPUB 3 archive with navigation and ordered XHTML chapters using `java.util.zip` |\n\nThe PDF path detects pdf2htmlEX output and routes it through a reader-oriented extraction step before rendering. Markdown and EPUB share `ReadablePageExtractor` for normal HTML and pdf2htmlEX pages.\n\nCurrent EPUB limitations: chapters are text-only, custom styling is minimal, heading hierarchy is flattened, and generated files have not yet been validated with `epubcheck`.\n\n## Build and Test\n\n### Prerequisites\n\n- Java 25.\n- Maven.\n- Playwright browser binaries only for tests or experiments that use dynamic fetching:\n\n```shell\nnpx playwright install\n```\n\n### Commands\n\n```shell\n# Compile the complete reactor\nmvn compile\n\n# Run every test\nmvn test\n\n# Run one core test without scanning unrelated modules\nmvn test -pl codex-ir-core -Dtest=codex.ir.ranking.RankersTest\n\n# Build an application module together with reactor dependencies\nmvn test -pl codex-ir-app -am\n\n# Full verification\nmvn compile && mvn test-compile && mvn test\n```\n\nWhen `codex-ir-core` or `codex-ir-web` has uninstalled local changes, include `-am` while working on `codex-ir-app`; otherwise Maven may resolve an older installed dependency.\n\n## Running Applications\n\n### IR and Crawling Demo\n\nThe primary demo entry point is `codex.scraper.Main`. Its current configuration performs live crawling, so inspect the configured seed URL before running it.\n\n```shell\nmvn exec:java -pl codex-ir-app \\\n  -Dexec.mainClass=\"codex.scraper.Main\"\n```\n\n### Product Discovery\n\n`DiscoveryRunner` accepts explicit product/category URLs or sitemap URLs:\n\n```shell\nmvn exec:java -pl codex-ir-app \\\n  -Dexec.mainClass=\"codex.scraper.DiscoveryRunner\" \\\n  -Dexec.args=\"--sitemap https://example.com/product-sitemap.xml --limit 50 --output both --out-dir ./reports\"\n```\n\n`codex.scraper.QuickDiscoveryRunner` is an IDE-oriented wrapper with arguments embedded in source.\n\n### Site Exporter\n\nMirror a site and publish it as PDF:\n\n```shell\nmvn exec:java -pl codex-ir-app \\\n  -Dexec.mainClass=\"codex.apps.siteexporter.SiteExporterCommand\" \\\n  -Dexec.args=\"--url https://example.com --out-dir ./mirror --format pdf --output ./site.pdf\"\n```\n\nResume from an existing mirror without network crawling:\n\n```shell\nmvn exec:java -pl codex-ir-app \\\n  -Dexec.mainClass=\"codex.apps.siteexporter.SiteExporterCommand\" \\\n  -Dexec.args=\"--from-mirror ./mirror --format markdown --output ./site.md\"\n```\n\nCreate an EPUB from an existing mirror:\n\n```shell\nmvn exec:java -pl codex-ir-app \\\n  -Dexec.mainClass=\"codex.apps.siteexporter.SiteExporterCommand\" \\\n  -Dexec.args=\"--from-mirror ./mirror --format epub --output ./site.epub\"\n```\n\n| Flag | Default | Description |\n|---|---|---|\n| `--url <url>` | Required unless resuming | Seed URL for a new mirror |\n| `--from-mirror <dir>` | None | Load an existing `mirror-manifest.json` and skip crawling |\n| `--out-dir <dir>` | `./mirror`, or the resumed mirror directory | Mirror HTML and manifest directory |\n| `--max-pages <n>` | `100` | Maximum pages for a new crawl |\n| `--max-depth <n>` | `3` | Maximum traversal depth |\n| `--no-same-domain` | Disabled | Permit links outside the seed domain |\n| `--format pdf|markdown|epub` | `pdf` | Publication format |\n| `--output <path>` | `./output.pdf`, `.md`, or `.epub` | Final artifact path, selected by format |\n\nTypical side outputs inside the mirror directory include:\n\n- `mirror-manifest.json` — mirrored page metadata.\n- `asset-manifest.json` — downloaded asset metadata for PDF runs.\n- `reader-pages/` — reader-oriented HTML generated for pdf2htmlEX inputs.\n- `markdown-pages/` — per-page Markdown generated by the Markdown driver.\n\n## Design Rules\n\n- Interface contracts are paired with static factories such as `Corpus`/`Corpora` and `ProductDiscoverer`/`ProductDiscoverers`.\n- Domain data is represented by records; builders are used when construction is incremental.\n- Mutable ingestion structures are separated from immutable search snapshots.\n- Core remains domain-neutral, web owns reusable crawling primitives, and concrete applications stay in app.\n- Persistence is intentionally deferred; current corpus, index, vocabulary, and vector stores are in memory.\n- Virtual threads are used for concurrent blocking work where they simplify ownership and limits.\n- Architectural decisions belong in ADRs under [`docs/adrs`](docs/adrs/).\n\nSee [`docs/CODING_IDENTITY.md`](docs/CODING_IDENTITY.md) for the project’s design philosophy and [`docs/Future-Forward.md`](docs/Future-Forward.md) for postponed work.\n\n## Current Limitations and Next Directions\n\n- Hybrid lexical/vector ranking is not implemented.\n- Field values are aggregated before indexing; true field-specific postings and BM25F remain future work.\n- Core storage is memory-only.\n- The site-exporter command does not yet expose sitemap-first or dynamic-rendering crawl modes.\n- Markdown and EPUB prioritize readable text over complete visual fidelity.\n- EPUB output still needs real-reader and `epubcheck` validation.\n\n## Module Documentation\n\n- [`codex-ir-core/README.md`](codex-ir-core/README.md)\n- [`codex-ir-web/README.md`](codex-ir-web/README.md)\n- [`codex-ir-app/README.md`](codex-ir-app/README.md)\n- [`docs/apps/site-exporter/ENGINEERING_LOG.md`](docs/apps/site-exporter/ENGINEERING_LOG.md)\n",
  "bytes": 14762,
  "sha": "3dabea37dcdbd463264d3d953faa926418b2db8dd648fde3b7a0a31e0c605ccf",
  "repo_slug": "jsanca/myir",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/okf_jsanca_myir_docs_knowledge_index_md_66329b31/readme"
}