{
  "markdown": "# mdcx\n\n<!-- mcp-name: io.github.jorgell23-sys/markdown-document-search -->\n\n[![PyPI](https://img.shields.io/pypi/v/mdcx)](https://pypi.org/project/mdcx/) [![tests](https://github.com/jorgell23-sys/mdcx/actions/workflows/tests.yml/badge.svg)](https://github.com/jorgell23-sys/mdcx/actions/workflows/tests.yml) [![License](https://img.shields.io/badge/license-Apache--2.0-blue)](https://github.com/jorgell23-sys/mdcx/blob/main/LICENSE) [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.22015991.svg)](https://doi.org/10.5281/zenodo.22015991)\n\nConvert a document collection to verified Markdown, package it into a single\nencrypted file, and query it from an agent through the Model Context Protocol.\n\n## Contents\n\n- [Overview](#overview)\n- [Requirements](#requirements)\n- [Installation](#installation)\n- [Quick start](#quick-start)\n- [Conversion](#conversion)\n- [Packaging and querying](#packaging-and-querying)\n- [Sent and received](#sent-and-received)\n- [Working incrementally](#working-incrementally)\n- [What the corpus knows about words](#what-the-corpus-knows-about-words)\n- [Something to keep that is not text to search](#something-to-keep-that-is-not-text-to-search)\n- [Writing often](#writing-often)\n- [MCP server](#mcp-server)\n- [When the client goes away](#when-the-client-goes-away)\n- [Reaching a word the transcription got wrong](#reaching-a-word-the-transcription-got-wrong)\n- [Language support](#language-support)\n- [Cross-language retrieval](#cross-language-retrieval)\n- [Portable paths](#portable-paths)\n- [Signing](#signing)\n- [Encryption](#encryption)\n- [Limitations](#limitations)\n- [Tests](#tests)\n- [Contributing](#contributing)\n- [Security](#security)\n- [Releases](#releases)\n- [Authorship](#authorship)\n- [Citation](#citation)\n- [Licence](#licence)\n\n## Overview\n\nmdcx converts a collection of documents to Markdown, verifies each conversion\nagainst its original, packages the corpus with its index and provenance into a\nsingle encrypted file, and serves that file to agents over the Model Context\nProtocol.\n\nIt addresses one constraint. An agent asked a question about a document\ncollection must either receive the documents in its context window, which is\nbounded in size and billed per token, or query a component that holds an index\nand returns only the passages that bear on the question. mdcx implements the\nsecond. Three properties distinguish it from an extraction script:\n\n- **Fidelity is measured, not assumed.** Every conversion is checked against the\n  text the original exposes, read by a library independent of the engine that\n  produced the conversion, and the coverage achieved is recorded per file.\n- **The corpus is a single encrypted artefact.** Passages, index and provenance\n  are held in one AES-256-GCM file whose header can be read without the key.\n- **Every passage carries its source.** An answer can be cited against a\n  document and a location rather than recalled.\n\n### Pipeline\n\n**Conversion.** Each document is attempted by the least expensive engine capable\nof reading it and escalated only where that engine falls short: direct text\nextraction, then a pass that recovers the tables a page draws, then full layout\nanalysis. Documents exposing no text are read by optical character recognition.\nContent the selected engine omitted is appended verbatim rather than reported as\nlost.\n\nOver the collection used during development — 99 documents, 1,144,553 reference\ntokens — 594 tokens were not recovered, a coverage of 99.948%. Of the 95\ndocuments that expose text, 70 were recovered in full and none fell below 99.5%.\nThe remaining four are scanned drawings holding no text in the file; they are\nmarked unverifiable, as no text original exists to measure them against.\n\n**Packaging.** The corpus, its search index and the provenance of every passage\nare written to a single `.mdcx` file. The development collection produced 3.9 MB\nfrom 8.8 MB of Markdown. A growing collection is not rebuilt from the start:\nvectors already computed are reused, and a corpus exceeding what can be\ndecrypted into memory is held as several packages queried as one.\n\n**Retrieval.** A query returns the passages that answer it, each with its source\ndocument and its position in the ranking. Word matching and dense retrieval are\nmerged by reciprocal rank, so a query reaches a document whether it shares that\ndocument's vocabulary or only its subject, including where the two are written\nin different languages. Over a corpus of 136 documents in 34 languages, the\nmerged engines rank the expected document first for 135 of the 136 queries.\nWhere no document in the corpus is about the question, the reply states this\nrather than presenting its nearest passage as an answer.\n\n### Measured cost\n\nOne query over the development collection — 99 documents, 180 MB — counted with\nthe `cl100k_base` tokenizer:\n\n| Method | Model tokens | Local tokens |\n|---|---|---|\n| Reading the originals | 2,265,488 | 2,265,327 |\n| Querying the package | 435 | 2,688,861 |\n\nThe 435 model tokens comprise 20 for the question, 274 for the retrieved passage\nand 141 for the answer.\n\nReading the originals costs the whole collection because a PDF is a binary\nformat: absent prior conversion there is no way to determine which of the 99\ndocuments holds the answer, so all of them are extracted and read.\n\nThis is a single measurement, not an average, and the saving depends on how much\ntext an answer requires. The work is not eliminated but relocated, from the\ncontext window, which is billed and finite, to local processing, which is\nneither. The local column rises for that reason.\n\n## Requirements\n\nPython 3.11 or later. No other component is required to query a package.\n\nThe floor is 3.11 because a package is held as one SQLite database and\nserialised in memory to be encrypted, and `sqlite3` gained the call that\ndoes so in that version. Earlier interpreters were declared supported and\nwere not: neither building a package nor opening one worked there.\nConversion and cross-language retrieval each add dependencies, listed under\n[Installation](#installation).\n\n## Installation\n\nQuerying and conversion are separated because their requirements differ by two\norders of magnitude.\n\n| Command | Provides | Approximate size |\n|---|---|---|\n| `pip install mdcx` | querying and reading `.mdcx` packages | 10 MB |\n| `pip install \"mdcx[mcp]\"` | the above and the MCP server | 50 MB |\n| `pip install \"mdcx[convert]\"` | document conversion (Docling, PyTorch) | 1.4 GB |\n| `pip install \"mdcx[tables]\"` | tables a page does not draw | 1.2 GB |\n| `pip install \"mdcx[multilingual]\"` | cross-language retrieval | 2.5 GB |\n| `pip install \"mdcx[all]\"` | all of the above, including OCR | 4 GB |\n| `pip install \"mdcx[all-gpu]\"` | the same, without pinning the CPU onnxruntime | 4 GB |\n\nConversion accounts for the heavy dependencies. A recipient who only queries an\n`.mdcx` file installs neither Docling nor PyTorch.\n\nThe `multilingual` extra is required for queries that cross languages. Most of\nits size is the embedding model, downloaded once on first use. A single-language\ncorpus does not require it.\n\nThe `tables` extra covers what a page does not draw. Tables in printed material\nare usually found from the rules drawn around them, which costs nothing and\nneeds no extra; borderless ones — a screenshot of a spreadsheet, a layout held\ntogether by alignment — are read by a small model that reports where the rows\nand columns run. It reads the shape only: the words still come from the text\nlayer of the document, so a cell cannot hold anything the page does not say.\nWithout it those pages are read by Docling instead, which is slower but already\npresent in the `convert` extra.\n\n### If the machine has a CUDA card\n\nInstall `mdcx[all-gpu]` rather than `mdcx[all]`, and install `onnxruntime-gpu`\nyourself.\n\n`onnxruntime` and `onnxruntime-gpu` are two distributions publishing the same\nmodule, so they cannot coexist: whichever pip wrote last wins, and it is usually\nthe CPU one. An extra that pins the CPU build therefore removes CUDA from an\nenvironment that had prepared it, on every upgrade — measured on three\nconsecutive releases, in two environments each time, with no error, nothing in\nany log, and optical recognition simply costing tens of times more. `all-gpu` is\n`all` without that pin.\n\npip cannot express \"either of these distributions\", so this cannot be settled by\ndeclaration alone. `mdcx-convert` therefore checks at startup: when the machine\nhas a card and the runtime does not offer it, it says so and gives the repair.\nThe check on its own is one line:\n\n```\npython -c \"import onnxruntime as o; assert 'CUDAExecutionProvider' in o.get_available_providers(), 'OCR WITHOUT THE CARD'\"\n```\n\n## Quick start\n\n```\npip install \"mdcx[convert]\"\n\nmdcx-convert --input ./Documents --output ./Documents_md\nmdcx pack --output ./Documents_md --target corpus.mdcx --key \"passphrase\"\nmdcx search corpus.mdcx \"where is the storage temperature stated\" --key \"passphrase\"\n```\n\n## Conversion\n\n```\nmdcx-convert --input ./Documents --output ./Documents_md\n```\n\nThe output mirrors the input directory structure, adds a global index, and\nrecords for each file the coverage achieved against its original.\n\n### Supported formats\n\nPDF, EPUB, Word, Excel, PowerPoint, HTML, Markdown, CSV and plain text.\n\nThe format of a file is determined from its first bytes rather than from its\nextension. Repositories are known to serve EPUB files from URLs ending in `.pdf`\nand declaring `application/pdf`, where only the content identifies the format\ncorrectly. Routing such a file by extension sends it to a reader that cannot open\nit, and the resulting failure is indistinguishable from a damaged document.\n\nPlain text carries no signature, so its extension determines the format. A file\nwhose content identifies no known format is skipped rather than assumed.\n\n### How much of the machine it uses\n\nConverting a library is the heaviest thing this package does, and it runs on a\nmachine somebody is working at. Nothing here is a constant: every figure is\nderived from the machine it finds, because the same number cannot be right on\nfour processors and on thirty-two.\n\n```\nmdcx-convert --input ./Documents --output ./Documents_md --max-cores 4\n```\n\n**Processors.** A fifth are left free and the rest are used: nine on twelve,\nthree on four. The cap is a budget for the whole run rather than a grant to\neach process, so it is divided among the workers and each is told its share.\nWithout that division the structured engine asks for four threads of its own\nand eight workers ask for thirty-two on a machine of twelve, spending inside the\npool the share that was carefully left outside it.\n\n**The card.** How many processes may use it at once is decided by three\nceilings, the smallest winning: the free video memory divided by what a worker\nholds with a full batch; how much of the material actually needs a model, which\nis the documents that expose no text and have nothing to extract; and leaving\nsomething for the processor. All three are needed. Without the first, asking for\nmore workers by hand does the opposite of what it looks like — twelve on a 6 GB\ncard ask for 15.7 GB and measured three times slower than three. Without the\nlast, a large card takes every worker and leaves one for the bulk of the work,\nwhich is processor work.\n\nThat limit is then held by a gate every worker shares, taken around the model\ncall rather than around the document, so a process reading text is not occupying\na place on the card while it does.\n\n**The batch.** What a page costs falls with the number of pages it travels\nwith — 150 ms sending one, 75 with eight, 46 with twenty-four — so the batch is\nas large as the card allows once every worker is seated on it, and no larger.\nIt is decided where both halves are known, because how much of the card a worker\nmay hold depends on how many may hold it; a worker deciding for itself reads the\nfree memory as though nobody else would.\n\n**Lanes.** Documents are dispatched to two of them. Both may reach every engine\nand both are counted against the same limit on the card: the lane decides what\nis worth dispatching where, not what a document is allowed to reach. The lane\nused to decide both, which meant that moving a document out of the crowded lane\nalso took away its structured engine.\n\nEvery one of these can be overridden — `--max-cores`, `--gpu-workers`,\n`--cpu-workers`, and `MDCX_TATR_BATCH` — and the derived figure is the default\nrather than a ruling. A machine that measures differently says so.\n\n### A document the engine does not finish\n\nThere is material the structured engine does not terminate on, and it cannot be\nrecognised beforehand: measured against documents that convert normally, the\nones that hang have fewer pages, the same size, the same images per page and\nslightly more text. About 4% of one real collection behaved this way — two to\neight ordinary A4 pages that ran for hours while their neighbours took seconds.\n\nWithout a bound, one such document holds its worker for the length of the run,\nand as many of them as there are workers stop the conversion altogether: the\nbatch waits for everyone. So `--timeout` gives up on a document after twenty\nminutes by default, records it with the status `TIMED OUT` — its own status,\nnot an error, because nothing was found wrong with it — and goes on to the next.\n`--timeout 0` waits indefinitely.\n\nThe limit is deliberately generous. Abandoning a good document loses all of its\nwork, while waiting too long for a bad one costs one worker for the excess, so\nit is sized for the first mistake being the expensive one.\n\nGiving up does not undo what was started. Layout analysis says as much — *the\nthread is likely stuck in a blocking call and will be abandoned* — and an\nabandoned thread keeps running: measured, three of them held a core each at 100%\nfor twenty minutes while producing nothing. A Python thread cannot be cancelled,\nso the only way to get the core back is to replace the process holding it, which\nis what `--recycle-after` does every fifty documents. `--recycle-after 0` never\nreplaces one.\n\nA permit for the card is squared up per document rather than only by the block\nthat took it, for the same reason: a block that is abandoned never returns its\npermit, and with two permits, two such documents left every worker waiting on a\nturn that never came — the card idle, the count at zero, and the run never\nadvancing again. And if a permit were still lost beyond recovery, the run goes\nahead without a turn rather than waiting forever: contention is a risk, waiting\nfor a turn that will not come is not.\n\n### Deciding whether a book is worth converting whole\n\nConverting a book to find out whether it is worth converting costs what the book\ncosts. `--sample-pages N` converts a spread sample instead — not the first N\npages, because a book opens with a cover, a blank verso and a title page, so a\nsample taken from the front describes the front matter rather than the book.\n\nThe sample is gathered into one document rather than converted page by page, and\nit keeps the headings that cutting pages would otherwise lose: a sample without\nthem is a wall of prose, and the section titles an author wrote are most of what\nsays whether the book is worth the rest. Its front matter says `sampled: true`\nand carries `pages_total`, so twenty pages of a book of six hundred cannot be\nmistaken for a short book.\n\n### Packing something that was never a folder\n\n`pack --output records.jsonl` reads one record per line — `name` and `text`, and\noptionally `pseudopath`, `folder` and `source` — instead of walking a directory.\nFor a collection that is generated rather than converted, writing it out as one\nfile per document only to read it back is work with nothing to show for it: on\n80,844 records, 1.4 minutes and 324 MB created, read once and deleted. A line\nthat cannot be read is skipped and named, because one bad record should cost\nthat record.\n\n### When a work is from\n\nA package records a date per document and, beside it, where that date came\nfrom. Both or neither: a date without its provenance confuses *when the work was\npublished* with *when the file was touched*, and whoever reads it cannot tell.\n\n    pack --dates dates.csv          # path,date[,provenance]\n    pack --date-from-mtime          # the file's time, recorded as `mtime`\n\nThe provenance is `source` when it came from the publisher, `sidecar` when\nsomebody supplied it, `front-matter` when the document carried it, `mtime` when\nit is the file's time and not the work's. Where nothing reliable is found the\ndate is absent, which is an honest answer and a different one from a guess. The\ncopyright year printed in the text is deliberately not used: a textbook reprints\nits front matter, so that year is the printing's rather than the edition's.\n\n`info` reports how many documents carry a date and the span they cover — *0 of 8\ndated* being the signal that the dates were lost on the way in. Every passage in\na reply carries `dated` and `dated_from`, so the age can be shown beside the\ncitation.\n\n`search --prefer recent` orders comparable answers newest first. It enters as a\nthird ranking fused by rank with the other two, never as a decay multiplying a\nscore: weighting values that share no scale is precisely what fusing by rank\navoids. It **orders rather than filters** — an older work that answers better\nstill comes back, which matters because a work from 1970 can be the right\nanswer, and in mathematics often is. Where both engines agree which passage is\nbest, the date does not move it; where they disagree, it decides.\n\nA preference can also be impossible to honour: it orders the fusion of two\nengines, so there has to be a fusion, and it orders by date, so something in the\nanswer has to carry one. Neither case is an error and neither changes the\nanswer, but from outside they look exactly like a preference that applied and\nfound nothing to move — so the reply says which happened. The command line\nprints a line only when it could not be applied, and the MCP reply carries\n`prefer_applied` and `prefer_reason` only then; an answer without them is one\nwhere the preference ran. The reason names what to do about it, because the\nremedies differ: a package with no meaning index is repacked with\n`--multilingual`, one with no dates with `--dates`.\n\n### Where works come from\n\nmdcx converts, packages and answers; it does not fetch, and depends on no\nnetwork of its own. A catalogue is a plugin, declared through the\n`mdcx.sources` entry point group and meeting the contract in `mdcx.sources`.\nAnswering questions over a package that already exists needs none of this — only\nbuilding a new corpus does, and where nothing is installed it says exactly that.\n\nKeeping the adapters out is deliberate rather than minimal. What looks like a\nsimple HTTP client is not: one catalogue answers 403 to its whole download\ncolumn and needs its handle resolved separately, another returns the same scrape\ncursor for every page. That knowledge belongs with whoever has it.\n\nWhat does not belong there is the part that has nothing to do with any\ncatalogue, and `mdcx.sources` now carries it:\n\n```\npython -m mdcx.sources --check <name>     # --help lists the rest\n```\n\nchecks a plugin against the contract — that `search` returns `Candidate`s with\nidentifiers it can be asked about again, that `fetch` returns bytes of a\nrecognisable type or raises rather than returning something else. A plugin had\nnothing to check itself against, and what that cost was measured: the first\nthing this reports is a source returning a cover thumbnail as though it were the\nbook. A catalogue named the attachment `9789819647453.pdf.jpg`, and 5,384 bytes\ncame back without an exception.\n\n`looks_like(data, \"pdf\")` is the check on its own — four bytes, no network — and\n`identify(data)` says what they were instead, because *this is not a PDF* is not\nactionable and *this is a JPEG* is. `patiently(call)` retries what raises\n`RateLimited`, honouring `Retry-After` when the server sent one: every catalogue\nrate limits, and none of that is knowledge about a particular one. It catches\nnothing else, because a 403 on a whole download column is not transient.\n\nA `Candidate` can also say **which server** its file would come from, in\n`download`, with `host` reading it back. The split between a cheap `search` and\nan expensive `fetch` exists so a caller can decide what is worth fetching, and\nwhich server it would be asking is one of the things worth deciding on: measured\nover 40 candidates from one catalogue, 25 of them — 62 per cent — resolve to a\nsingle host that answers 403 to everything, while another returns a 5 MB PDF in\ntwo seconds. Working down the ranking spends the whole budget on the first. Only\nthe catalogue knows this, so only the catalogue can say it; the check notes when\none host holds more than half.\n\n`tests/test_sources_kit.py` holds a source in twenty lines, against no network.\nAn executable example does not go quietly out of date.\n\n### Checking a conversion before packaging\n\n`mdcx-search` searches the converted Markdown directly, before there is a\npackage, and quotes each passage with the document and pseudopath it came from.\nIt is how a conversion is inspected while the folder is still open to\ncorrection.\n\n```\nmdcx-search \"movable type\" --output ./Documents_md\nmdcx-search --phrases ./questions.txt --output ./Documents_md --json found.json\n```\n\nPassages are ranked with BM25 aggregated per document rather than in isolation,\nso a long document that covers a subject across several fragments is not beaten\nby a short unrelated one that repeats a term. `--literal` requires the exact\nphrase and nothing else; `--bm25` ranks by relevance without literal matching.\n\nThis engine reads the Markdown folder. Retrieval over a built package, with\nmeaning and across languages, is `mdcx search` and the MCP server.\n\n## Packaging and querying\n\n```\nmdcx pack --output ./Documents_md --target corpus.mdcx --key \"passphrase\"\nmdcx info corpus.mdcx\nmdcx search corpus.mdcx \"where is the storage temperature stated\" --key \"passphrase\"\nmdcx export corpus.mdcx --target ./restored --key \"passphrase\"\n```\n\n`info` reads the header without the key, so the issuer and the integrity of a\nfile can be checked before it is opened. `export` reconstructs the original\nfolder, so a collection can be moved out of the format at any time.\n\n## Sent and received\n\nCorrespondence has a direction, and a question about it is usually about one\nside: what was asked of us, or what we answered. Where the top-level folder of\na collection states that direction, it is recorded per document and a query can\nbe restricted to it.\n\n| Top-level folder contains | Direction |\n|---|---|\n| `sent`, `emitido`, `outgoing` | sent |\n| `received`, `recibido`, `incoming` | received |\n| anything else | unclassified |\n\nThe names are recognised in English and Spanish, since a collection may be\norganised in either, and only the top-level folder is examined, so a subfolder\nnamed after a correspondent does not reclassify what it holds.\n\n```\nmdcx search corpus.mdcx \"what was agreed about the schedule\"     --key \"passphrase\" --only received\n```\n\nThe MCP `search` tool takes the same restriction as its `direction` argument.\nA collection organised in any other way is unaffected: every document is\nunclassified, and a query that names no direction is not narrowed.\n\n## Working incrementally\n\nA collection that is delivered once and a collection that grows every day place\ndifferent demands on the tool. The second must not pay for what it has already\ndone.\n\n### Conversion resumes\n\nConversion records the digest of each source in the Markdown it produces, and\nskips any file whose source is unchanged and whose output is present. This is\nthe default; `--force` disables it.\n\nThe unit is the chapter rather than the document, so a book split into 47\nchapters and interrupted at the 40th costs the remaining 7 on the next run.\nProgress is written after each unit and flushed to disk, so an interrupted run\nleaves a record that the next one reads.\n\nA run over a converted collection reports what it reused:\n\n```\nAlready converted and unchanged: 8 (reused)\nElapsed         : 0.4 min\n```\n\nA chapter is reconverted when its verification reported findings, since a result\nthat was not certified is not a result worth keeping.\n\n### Packaging costs what was added\n\nIndexing meaning dominates the cost of packaging. On one measured book: 505\nseconds of encoding against 4 seconds of compression and 0.2 of encryption.\nEncoding the whole corpus on every publication makes adding one document cost a\nreindex of every previous one.\n\nA passage whose text has not changed has the same vector. `--reuse` reads the\nvectors of an existing package and encodes only what is new:\n\n```\nmdcx pack --output ./Documents_md --target corpus-2.mdcx --key \"passphrase\" \\\n    --multilingual --reuse corpus-1.mdcx\n```\n\n```\n  meaning indexed with BAAI/bge-m3 (1024 dimensions)\n  passages encoded 395   reused 733\n```\n\nMeasured over the chapters of one book, where 733 of 1,128 passages were\nunchanged, packaging took 15.4 seconds against 37.8 without reuse.\n\nReuse also carries the calibration forward. A package given its questions with\n`--focus` hands them to the one built from it, so a corpus that grows keeps the\nthreshold it was calibrated with instead of reverting to one estimated from\npassages — a change that barely moves the stored number and shifts the margin\napplied to it from 0.95 to 0.60. Passing `--focus` again overrides what was\ninherited, and the summary says when it inherited rather than doing it quietly.\n\nThe vectors are read from the previous package, which already holds them and is\nalready encrypted with the same key. No intermediate store is created: a vector\nallows the text it represents to be approximated, so keeping vectors outside the\npackage would undo the encryption the format provides.\n\nReuse requires the same model. Vectors from two models occupy different spaces,\nso a package encoded by another model contributes nothing rather than\ncontributing values that cannot be compared.\n\n### Several packages as one corpus\n\n`MDCX_FILE` accepts more than one package, separated by the path separator of\nthe platform or by a comma. The server queries all of them and returns one\nranked list, with each result naming the package it came from.\n\n```json\n{\n  \"mcpServers\": {\n    \"mdcx\": {\n      \"command\": \"python\",\n      \"args\": [\"-m\", \"mdcx.mcp_server\"],\n      \"env\": {\n        \"MDCX_FILE\": \"/corpora/2026-01.mdcx:/corpora/2026-02.mdcx\",\n        \"MDCX_KEY\": \"package-key\"\n      }\n    }\n  }\n}\n```\n\nOne key serves every package; several keys are matched to the packages in order.\n\nThis makes each package immutable: it is indexed once and never rebuilt. A\ncorpus grows by adding packages rather than by enlarging one, which also keeps\neach of them within what can be decrypted into memory, since a package is\ndecrypted whole when it is opened.\n\nResults from different packages are merged by reciprocal rank. Their scores are\ncomputed over different corpus statistics — the frequency of a term depends on\nthe corpus it is measured in — so the scores are not comparable between\npackages, while positions within each are.\n\n## MCP server\n\nThe server requires Python and this package. It does not require the conversion\nstack; its footprint is approximately 50 MB.\n\n```json\n{\n  \"mcpServers\": {\n    \"mdcx\": {\n      \"command\": \"python\",\n      \"args\": [\"-m\", \"mdcx.mcp_server\"],\n      \"env\": {\n        \"MDCX_FILE\": \"/path/to/corpus.mdcx\",\n        \"MDCX_KEY\": \"package-key\"\n      }\n    }\n  }\n}\n```\n\nWith [uv](https://docs.astral.sh/uv/) the server runs without prior installation,\nwhich is the common arrangement for Python MCP servers:\n\n```json\n{\n  \"mcpServers\": {\n    \"mdcx\": {\n      \"command\": \"uvx\",\n      \"args\": [\"--from\", \"mdcx[mcp]\", \"python\", \"-m\", \"mdcx.mcp_server\"],\n      \"env\": {\n        \"MDCX_FILE\": \"/path/to/corpus.mdcx\",\n        \"MDCX_KEY\": \"package-key\"\n      }\n    }\n  }\n}\n```\n\nThree tools are exposed:\n\n| Tool | Returns |\n|---|---|\n| `search` | passages answering a question, each with its source document, portable path and rank; `direction` restricts it to one side of a correspondence |\n| `info` | the corpus record, including the fidelity of its conversion |\n| `document` | a complete document, when passages are insufficient |\n\nThe package is verified before the server begins listening, so an incorrect path\nor key is reported at startup rather than on the first query.\n\nA passage carries its `rank` and no score. The list is ordered by that rank and\nby nothing else: word matching and meaning score on scales with no common\nmeaning — one has no upper bound and depends on the corpus it was measured in,\nthe other runs from zero to one and does not — so there is no single number here\nthat can be compared, sorted or filtered by.\n\nWhat can be compared is reported once for the reply. `similarity` is how near\nthe corpus comes to the question, and a `warning` appears when nothing in it is\nabout the question. The passages are returned either way: the nearest passage is\nworth seeing even when it is not an answer, and a corpus that answers in another\nlanguage must not be hidden by this.\n\nHow near counts as near is measured from the corpus rather than fixed. Packing\nrecords `answerable_at`, how near this corpus comes to a question it does\nanswer, estimated by using its own passages as questions; the reply reports it,\nand the warning is judged against it.\n\nThat estimate is only as good as passages resembling questions, and on some\ncollections they do not. A catalogue of 80,844 records — all back-cover blurbs,\nall sharing a rhetorical shape — calibrated at 0.759 where a corpus of books\ncalibrates at 0.580, and at 0.759 the warning fires on questions the catalogue\nanswers well. `pack --focus \"<question>\"` is for that case: given the questions\na package exists to answer, the threshold is taken from them instead of\nestimated, and `info` reports which of the two it was. Repeat the option to give\nseveral; the cut goes just under the weakest of them. A fixed threshold could not do this: the\nsame questions reach 0.51 on one corpus and 0.55 on another, so any single cut\nfalls inside the answered range of one collection or below another's, which is\nhow it behaved before this was measured. A package built before this exists has\nno such number and is judged by the previous thresholds, unchanged.\n\nA package whose source material is gone can still be calibrated. The threshold\nused to be writable only by `pack`, and `pack` walks a folder of documents — so\na package that outlived its material fell back to a constant nobody measured on\nit, and would in every future version. Nothing about the measurement needs the\nmaterial: it needs the vectors, which are inside, and questions, which come from\noutside.\n\n```\nmdcx calibrate corpus.mdcx --key \"passphrase\"     --question \"how heat travels by conduction and radiation\"     --question \"what the equivalence of mass and energy means\"\n```\n\nThe package is rewritten in place — same documents, same passages, same vectors,\naround a changed measure — and `info` reports it as `focus-after` rather than\n`focus`, because measured while packing and measured afterwards describe the\ncorpus at different moments. A signed package needs its signing key; without it\nthe command refuses rather than handing back an unsigned package.\n\nThe same judgement is available per package, which is what a consumer serving\nseveral of them needs in order to decide which one answers:\n\n```python\narchive.closeness(connection, text)   # (nearest cosine, its clearance) or None\narchive.answers(connection, text)     # by that package's own calibration\n```\n\n`answers` applies the margin that matches how the package was calibrated, which\nis the part easiest to get wrong and which raises no error when it is: applying\nthe passage share to a threshold taken from questions was measured letting seven\nof eight unrelated queries through. Asking whether *anything* open is about a\nquestion is a different question — a property of the set — and stays where it\nwas, in the warning the MCP server raises.\n\n## What the corpus knows about words\n\n`vocabulary(connection)` returns the document frequency of every term together\nwith the rule that produced it, and `unknown_terms(connection, text)` names the\nterms of a text this corpus has genuinely never seen.\n\nThe rule matters more than it sounds. The index records terms of three\ncharacters or more — a single character where the script does not separate\nwords — so absence from the table has two meanings: the corpus never saw the\nterm, or the index was never going to record it. Weighting by rarity gives an\nabsent term the maximum weight, so reading absence as novelty makes the\nshortest, emptiest words the most informative ones. Measured: questions a corpus\nanswers well declared between 0.37 and 0.55 of unknown vocabulary, and fall to\nexactly zero once the rule is applied. `unknown_terms` applies it.\n\n`unknown_terms` is literal, and the index it reads is not the one that crosses\nlanguages. The meaning index reaches a Spanish question against an English\ncorpus; the word index cannot, so asked across languages it returns every term\nof the text — a measure of which language the corpus is in rather than of what\nit knows. `unfamiliar(connection, text)` returns the same terms with the share and a\n`cross_language` flag. Prefer it wherever the language of the question is not\nknown to match the corpus.\n\nThe flag is decided by the language and not by the share, which took two wrong\nshapes to arrive at. The share measures how *much* vocabulary is missing and\nnever why: it goes high both when the corpus is in another language and when it\nsimply does not cover the subject, and it is not even a property of the question\n— it rises as the package shrinks, so one query measured 0.17 against a package\nof 266 documents and 0.83 against one of 29, in the same language. A fixed cut\non it silences small packages systematically, which are the ones for which \"I\nhave never seen these words\" is the strongest thing they can say. And no cut\nworks anyway: a real crossing was measured at 0.60 and a same-language query at\n0.80.\n\nSo the detected language decides, the share only says that something is missing\nat all, and a detector that does not answer is not read as one that disagrees —\nfailing to identify a language is not evidence of a different one.\n\nFunction words are not removed on top of that, deliberately. `die` in die\ncasting and `les` in Les Misérables carry meaning in the language being\nsearched, and which words are empty depends on the question being asked. What\nmdcx can state is what it never recorded; what counts as uninformative is the\ncaller's.\n\nThe keys of `df` are normalised — folded case, folded accents — so a caller\ntokenising with `search.tokenize_text` gets `GPU` where the table holds `gpu`.\n`unknown_terms` handles that; anyone reading `df` directly must.\n\n### The two signals disagree\n\n`assess(connection, text)` returns both, because each is wrong where the other\nis right and neither said so.\n\nThe cosine cannot tell the senses of a homonym apart — a multilingual embedding\nplaces them together. Measured on a package of algebra: *graph coloring adjacent\nvertices different colors* came in at 0.6553 against a threshold of 0.5661 and\nreturned lessons on comparing graphs and on the ellipse. `graph` as the plot of\na function, not as a graph. The word the question turns on, `coloring`, the\ncorpus had never seen.\n\nNo quantity derived from the same vectors repairs that, which was measured\nrather than assumed: clearance does not separate — the false positive's falls\ninside the range of the questions the corpus answers *and* inside the range of\nthe unrelated ones, and its closeness sits above four of eight legitimate\nquestions — and neither does the minimum over windows of the query. Both are\nfunctions of a space that has already lost the distinction. What separates it is\nthe literal vocabulary, which does tell an absent `coloring` from a present\n`graph`.\n\nThe verdict is reported, not overruled. Whether an unfamiliar word should refuse\na query depends on what the query is for, and a word can be peripheral: *the\nslope of a line drawn in Patagonia* is answerable and `patagonia` is unknown.\nThe MCP `search` reply carries `unknown_terms` **per package** — a map from the\npackage to the words of the question it has never seen, so a reader can cross\nthe strange word with the package the passage they are about to cite came from.\nNot pooled: intersecting across packages emptied the signal as the library grew,\nbecause four packages each lacked something different and no term was missing\nfrom all four, while three of them had never seen the word the question turned\non. A package is absent from the map when it knows every word, and when the\nquestion is in another language than that package.\n\nWhere several packages are served, `similarity` is over all of them pooled and\n`answerable_at` is the **lowest** of their calibrated reaches, so the threshold\nin force comes from the narrowest package rather than from the one a passage\ncame from. `answerable_at_by_package` gives each of them. The criterion is left\nas it is on measurement rather than preference: against the alternative — the\nreach of the package the best passage came from — the minimum wins six to one\nover 42 queries, because a question one narrow package answers would otherwise\nbe condemned by a wider one's threshold.\n\nNote also that vectors are stored in half precision. They are renormalised when\nread, because rounding to half precision costs the normalisation and every\nquantity computed from them was slightly not a cosine.\n\n## Something to keep that is not text to search\n\nA corpus used as a memory sometimes has to hold an object — a certificate, a\ntable of coordinates — and everything in the folder became passages. One such\nartefact of 500 vertices measured 2,003 passages, 40.6 per cent of that corpus,\nand made every later write cost 1.57 times as much, because packing walks the\nwhole corpus even when one document changed.\n\nIt did not spoil the ranking, which was the fear and was wrong: coordinates\nresemble no question, so none of those passages reached a top five. The cost is\nweight and time.\n\n`indexed: false` in a document's front matter keeps it in the package — signed,\nencrypted, one file — and out of the index, the vectors and the passage count.\n`export` restores it verbatim, and the MCP `document` tool returns it by name,\nwhich is the only way in, since it cannot be searched for.\n\n`pack` also reports which document contributed the most passages, and says so\nwhen one holds a third or more of them.\n\n## Writing often\n\nCompressing and encrypting are properties of the whole file, so they cost the\nsame whether one document was added or the corpus was rebuilt. That is a fixed\nprice per write, and it grows with the corpus rather than with what was added.\n`pack` reports it as `seconds_compress` and `seconds_encrypt`, so a caller\nwriting often can decide how often to write.\n\n`--fast` trades size for time where that is the right trade — a package that is\nrewritten every few minutes and never leaves the machine, rather than one that\nis distributed and read many times. Measured on a 30 MiB database of 190\ndocuments: 1.03 s for 2,170 KiB against 6.22 s for 1,569 KiB. Six times the\nspeed for 38 per cent more bytes. Nothing else about the package changes.\n\n### When the client goes away\n\nThe server ends its own process rather than returning and letting the\ninterpreter decide when. Returning from `main` is not exiting: Python waits for\nevery non-daemon thread before shutting down, and the libraries under an encoder\nstart some. Ten server processes were measured alive at once, the oldest for two\nand a half hours, holding 7.25 GB between them and consuming no processor at\nall.\n\nAnd it lets go of the encoder after `MDCX_IDLE_UNLOAD_MINUTES` of silence — 30\nby default, `0` to disable — because the damage does not depend on the cause:\nthree gigabytes held by a process answering nobody is worth avoiding either way.\nThe next question reloads it in seconds.\n\n### Reaching a word the transcription got wrong\n\nA word that came out of optical recognition with one letter misread is a word\nthe index does not contain, and literal matching can never return it. There is\nno error to show for it: the reply is simply empty, and the passage sits in the\ncorpus unreachable.\n\nOptical recognition does not misread letters at random — it misreads the ones\ndrawn alike, `0` for `O`, `1` for `l`, `5` for `S` — so grouping letters by shape\nputs the misread word and its original in one bucket. The bucket is far too\ncoarse to answer with, holding hundreds of words, so it never answers: it\nproposes a handful of candidates and edit distance decides among them.\n\n```\nmdcx pack --output docs --target corpus.mdcx --key \"passphrase\" --shapes\nmdcx shapes corpus.mdcx --key \"passphrase\"      # to one written without it\n```\n\nMeasured on a corpus of 188 documents, against the methods that already exist\nfor the same problem: 76.5 % of misread words recovered, against 0 % for literal\nmatching and 45.3 % for trigram overlap, at a seventh of the trigram cost. It\ncosts 2.96 % of package there, and more on a corpus whose vocabulary is nearly\nall distinct — which is why it is asked for rather than assumed. A corpus that\nnever went through optical recognition pays and gets nothing.\n\nThree boundaries worth knowing, and the third is the one that decides how to\nread a reply.\n\nIt is **not a spell checker**: the same measurement over randomly substituted\nletters, the error a typist makes, falls to 49.6 % while trigrams do not move —\nthe advantage comes from the shape, and so does its limit.\n\nIt runs **only where the answer would otherwise be empty**: terms are matched\nwith OR, so one word the corpus does have is enough to return passages, and then\nnothing is widened.\n\nAnd **half of what it proposes is not a transcription error**. Precision was\nmeasured at 49.9 %, against 59.1 % for trigrams — for Latin script the table is\ncoarse, since all ten digits and 22 of 26 lowercase letters fall in one class, so\nin practice the sieve proposes any single substitution among them. `libre` finds\n`libro`, `precis` finds `precio`. That is survivable exactly because the reply\nsays what it did: a caller reads `read_as` and decides, where a caller shown a\nbare passage would have no way to know.\n\nWhere a passage was found under another spelling the reply says so, in\n`read_as`. Presenting it as a literal match would claim the document says what\nit does not, and a citation carrying the words of the document is what makes the\npackage worth having.\n\nThe technique is not new, and the attribution belongs here: it is the character\nshape codes Spitz described at Xerox in the nineties, used there over document\nimages to avoid a full recognition pass. What differs is where it is applied —\nover text already converted, for the errors the conversion left behind.\n\n## Language support\n\nRetrieval by word is script-aware. A query matches the words present in the\ndocuments, in any writing system, and the results of a single search may include\ndocuments in several languages. The predominant language of a corpus is recorded\nand reported by `info`; it describes the corpus and does not restrict what a\nquery returns.\n\nThe following are verified by `tests/test_languages.py`, which builds one corpus\nholding the same four subjects — algebra, botany, printing and baking — in every\nlanguage listed, then issues a query in each. Each query competes against the\nthree other documents in its own language and against the remainder of the\ncorpus. All 136 queries return the expected document in first position.\n\n| Script | Languages |\n|---|---|\n| Latin | English, Spanish, Portuguese, French, Italian, German, Dutch, Swedish, Danish, Norwegian, Finnish, Polish, Czech, Hungarian, Romanian, Turkish, Indonesian, Vietnamese, Catalan |\n| Cyrillic | Russian, Ukrainian, Bulgarian, Serbian |\n| Greek | Greek |\n| Arabic | Arabic, Persian |\n| Hebrew | Hebrew |\n| Devanagari | Hindi |\n| Bengali | Bengali |\n| Tamil | Tamil |\n| Thai | Thai |\n| Han | Chinese, Japanese |\n| Hangul | Korean |\n\nSupport is a property of the writing system rather than of the language, so a\nlanguage written in any of these scripts is covered whether or not it is listed.\nThree properties establish this:\n\n**Tokenisation.** A word is a run of letters, digits and the combining marks\nattached to them. The set of combining marks is derived from the Unicode\ncharacter database rather than enumerated, which keeps the vowel signs of\nDevanagari, Bengali, Tamil and Thai attached to the letters they modify.\n\n**Accent folding.** Folding is restricted to combining marks that represent an\naccent placed on a letter, so that `café` matches `cafe`. The vowel signs of\nIndic scripts and the points of Hebrew and Arabic are preserved, since in those\nscripts they carry the sound of the syllable.\n\n**Segmentation.** Writing systems that do not separate words with spaces —\nChinese, Japanese, Korean, Thai, Lao, Khmer, Burmese, Tibetan and Javanese — are\nindexed by character, at index time and query time alike. This is what a lexical\nindex can match without a segmenter trained on a single language.\n\nWord matching operates within a language: the words of a query must be present in\nthe document. A query written in one language reaches a document written in\nanother only where the two share a term, as proper names and loanwords often do.\nWhen a query returns no result and none of its terms appear in the index, the\nresponse states this and names the language of the corpus, distinguishing an\nempty answer from material the corpus does not hold.\n\nRetrieval across languages is a separate capability, described below. It is\noptional and requires a model. Where the query is written in the language of the\ndocuments the two are merged, each covering what the other cannot; where it is\nnot, word matching has nothing to contribute and is left out, because the few\nterms it does match there are accidents and they arrive first.\n\n## Cross-language retrieval\n\nWord matching operates within a language. A Spanish query and a German document\non the same subject share no term, so a word index has nothing to match. Measured\non a corpus written in 34 languages, a query retrieves 4.2% of the documents on\nits subject, comprising essentially those written in the language of the query.\n\nRetrieving the remainder requires representing meaning rather than spelling. A\nmultilingual embedding model places a sentence and its translation at nearby\npoints in a vector space, so a document can be retrieved through its content\nrather than its vocabulary. When built with `--multilingual`, a package stores a\nvector for each passage alongside the passage, under the same encryption, and the\nsame query retrieves 96.9% of the documents.\n\n```\npip install \"mdcx[multilingual]\"\nmdcx pack --output ./Documents_md --target corpus.mdcx \\\n    --key \"passphrase\" --multilingual\n```\n\nThe corpus is encoded once, when the package is built. A recipient encodes only\ntheir own queries.\n\n### Merged engines\n\nBoth engines are retained because their failure modes are complementary. Measured\non the same corpus of 136 documents in 34 languages:\n\n| Engine | Across languages | Expected document ranked first |\n|---|---|---|\n| Word | 4.2% | 136 of 136 |\n| Meaning | 98.5% | 126 of 136 |\n| Merged | 96.9% | 135 of 136 |\n\nThe dense engine retrieves across languages but ranks less precisely within the\nlanguage of the query. The lexical engine ranks precisely and does not retrieve\nbeyond that language. Merging by reciprocal rank retains both properties, at a\ncost of one document in 136 relative to the lexical engine alone.\n\nRanks are merged rather than scores, as a BM25 score and a cosine similarity have\nno common scale.\n\n`--mode lexical` and `--mode semantic` select a single engine.\n\n### Model selection\n\nThe default is `BAAI/bge-m3`. Models were compared on FLORES-200, a corpus of\nsentences translated by professionals into 200 languages. The task is to retrieve\na sentence given its translation, among candidates drawn from the same corpus, in\nboth directions of every language pair.\n\nThe selection criterion is the worst-performing language pair rather than the\nmean, since a mean can conceal a language on which a model performs poorly.\n\n| Model | Mean | Worst language | Worst pair |\n|---|---|---|---|\n| `BAAI/bge-m3` | 100.0% | 99.8% | 98.0% |\n| `sentence-transformers/LaBSE` | 99.5% | 97.7% | 96.0% |\n| `intfloat/multilingual-e5-large` | 98.9% | 96.5% | 92.0% |\n| `intfloat/multilingual-e5-small` | 97.0% | 94.4% | 92.0% |\n| `ibm-granite/granite-embedding-97m-multilingual-r2` | 95.4% | 91.3% | 84.0% |\n\nMeasured over 132 language directions covering 10 writing systems, with 50\ncandidates per query. In a larger run of 1,122 directions across all 34 languages\nwith 100 candidates, LaBSE reached a mean of 99.6% and 93.5% on its worst\nlanguage, with no pair below 90%.\n\nAn alternative model is selected by name:\n\n```\nMDCX_MODEL=sentence-transformers/LaBSE mdcx pack ...\n```\n\nA package records the model that encoded it. A query encoded with a different\nmodel occupies a different vector space, so a mismatch disables meaning-based\nretrieval rather than returning results that cannot be compared.\n\n## Portable paths\n\nNo output contains absolute paths. Each document is identified by a pseudopath\nbeginning with `@/`, resolved against the folder or package containing it, so a\ncorpus remains valid on local disk, network share or cloud storage.\n\n## Signing\n\nA package can be signed so that its issuer can be verified rather than declared.\nThe signature covers the digest of the encrypted body, attesting to both origin\nand content, and is verified without the encryption key.\n\n```\nmdcx keygen\nmdcx pack --output ./Documents_md --target corpus.mdcx --key \"passphrase\" \\\n          --issuer \"Acme Ltd\" --signing-key <private-key>\nmdcx verify corpus.mdcx --public-key <public-key>\n```\n\nVerification requires the body to be intact. A signature covering only the\nrecorded digest would accept a package whose contents had been replaced while its\nheader was left unmodified.\n\nThe issuer field is free text and is not evidence of origin on its own.\n\n## Encryption\n\nPackages are encrypted at rest and decrypted in memory when opened; no plaintext\nis written to disk. This protects a file in transit and at rest. It is not\nequivalent to searching over encrypted data without decryption, which is a\ndistinct field with documented leakage attacks and per-query costs measured in\nseconds.\n\nThe key is derived with scrypt at N = 2^15, r = 8, p = 1. Those parameters\nrequire 32 MB of memory per attempt, which is what resists the parallelisation a\nGPU would otherwise bring to a search: memory, unlike arithmetic, does not\nbecome cheap by adding cores. One derivation takes 286 ms single-threaded on the\ndevelopment machine, approximately 3.5 attempts per second per core.\n\nThat cost falls on an attacker and on the legitimate opening of a package alike.\nIt multiplies the work of a search; it does not make a weak passphrase safe. The\nstrength of the encryption is the strength of the passphrase, and one drawn from\na dictionary stays within reach of an offline search whatever the derivation\ncosts.\n\n## Limitations\n\nRetrieval returns documents whose content is close to the query. It does not\ntranslate them: passages are returned in the language in which they were written.\n\nDocuments that expose no text, such as scanned drawings, are read by optical\ncharacter recognition and counted as unverifiable rather than as findings, since\nno text original exists against which to measure fidelity. Coverage is computed\nover the documents that could be measured, so an unverifiable document neither\nraises nor lowers it.\n\nCoverage measures the tokens preserved by a conversion. It does not measure the\npreservation of table structure, which is reported separately.\n\nDeciding that a grid of drawn rules is a table, rather than prose someone framed\nfor emphasis, is done by how many of its rows run to more than one line. That\ntest is not exact in either direction: prose laid out in short lines can pass it,\nand a table whose cells wrap can fail it. A separating signal was looked for in\nthe material at hand and not found — the longest cell in the sample belongs to a\nlegitimate table, so cell length does not divide them — and no further threshold\nwas added on the strength of one collection. Where the decision goes wrong, the\ntext is still present and still counted in coverage; what is lost is its shape.\n\nA package is decrypted in full when it is opened, so its size is bounded by the\nmemory available. A corpus larger than that is held as several packages and\nqueried together, as described under\n[Working incrementally](#working-incrementally).\n\n## Tests\n\n```\npip install pytest\npython -m pytest tests/ -v\n```\n\n254 tests. Most of them exist because something failed once; the file that\ncovers it says which, so a correction that is undone is noticed.\n\n**Retrieval**\n\n| File | Scope |\n|---|---|\n| `test_languages.py` | retrieval in 34 languages across 11 writing systems, and the requirement that a term shared by several languages returns the documents of all of them |\n| `test_multilingual.py` | retrieval across languages, and the requirement that merging engines preserves the precision of the lexical engine |\n| `test_multipackage.py` | querying several packages as one corpus, including key configuration and the reporting of a missing package |\n| `test_relevance.py` | that serving several packages does not bury the answer among the ones that hold nothing about it |\n| `test_direction.py` | restricting a search to one side of a correspondence, and the requirement that both engines filter by the same form of the value |\n| `test_answer_quality.py` | what the server says when it cannot answer, and the places where a number meant something other than it appeared to |\n| `test_describes_itself.py` | that every field a reply carries and every argument a tool accepts are accounted for in the description the caller reads, and that no field is promised after it stopped arriving |\n| `test_encoder.py` | how the encoder spends the accelerator, and that batching leaves the vectors unchanged |\n| `test_package_identity.py` | that a cache belongs to a package rather than to a memory address, so a package cannot answer with the vectors or the corpus statistics of one that was closed |\n\n**Conversion**\n\n| File | Scope |\n|---|---|\n| `test_formats.py` | identification of a file by content rather than extension, in both directions, and the extraction of reference text from EPUB |\n| `test_conversion_order.py` | which engine reads a document, when the search for a better one stops, and what rejects a table that is not one |\n| `test_single_extraction.py` | that a document is extracted once however many engines read it, and that what each is given is its own to edit |\n| `test_table_cells.py` | that a character of a drawn table lands in exactly one cell, including a glyph the outer rule cuts |\n| `test_table_shapes.py` | the reading of a table the page does not draw, where the model supplies the shape and the text layer the words |\n| `test_headings.py` | that a chapter keeps the section titles its book already carried, whichever engine converted it |\n| `test_reporting.py` | that the summary separates a document measured and found short from one that could not be measured at all |\n| `test_server_leaves.py` | that a server whose client has gone ends its own process rather than waiting on a thread nobody will join, and that it lets go of the encoder after a long silence |\n| `test_shapekey.py` | recovering a word optical recognition misread: that letters drawn alike share a key, that the sieve proposes and never decides, that it runs only where the answer would be empty, and that the reply says what was read as what |\n| `test_sources_kit.py` | the source contract and what a plugin should not have to write again: reading four bytes, waiting when a server asks, and checking a plugin against the contract — plus a source in twenty lines, which does not go out of date the way a paragraph does |\n| `test_incremental.py` | reuse of vectors between packages: that unchanged passages are not encoded again, that an edited one is, and that reuse produces the same ranking |\n\n**The package as it is installed**\n\n| File | Scope |\n|---|---|\n| `test_stress.py` | hostile inputs: empty and corrupted files, names in other alphabets, malformed queries including SQL injection, truncated and tampered packages, concurrent access, and compaction against content loss |\n| `test_entrypoints.py` | that every command the package declares can be started and can report its version, and that every MCP tool publishes the signature of the function that answers it |\n| `test_machine_share.py` | that a conversion leaves a share of the machine free, counting threads as well as processes; that the lanes are sized from the video memory, the work and the processors rather than from constants; and that a turn on the card is given back even when the model raises |\n| `test_console.py` | that a document name the console cannot represent does not stop the conversion, in the parent process and in the workers |\n\nTests that need a model skip themselves when it is absent, so the suite passes\non a plain `pip install mdcx` as well as on `[all]`. Seven files depend on the\n`multilingual` extra and one on `tables`.\n\nContinuous integration runs the suite on Python 3.11 and 3.13, on Linux and\nWindows.\n\n## Contributing\n\nIssues and pull requests are accepted at\n[github.com/jorgell23-sys/mdcx](https://github.com/jorgell23-sys/mdcx).\n\nA change needs a test that has been seen to fail without it, and a change about\ncost or quality needs the measurement that justifies it. [CONTRIBUTING.md](CONTRIBUTING.md)\nsets out what makes a report act on itself and what a pull request is expected\nto carry; [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) covers the rest.\n\n## Security\n\nTo report a vulnerability, open a security advisory at\n[github.com/jorgell23-sys/mdcx/security/advisories](https://github.com/jorgell23-sys/mdcx/security/advisories)\nrather than a public issue.\n\nPackages are encrypted with AES-256-GCM and keys derived with scrypt. The\nencryption protects a package at rest and in transit; it does not protect against\na compromised host, where the key is present in memory while the package is open.\n\n## Releases\n\nVersion history and release notes:\n[github.com/jorgell23-sys/mdcx/releases](https://github.com/jorgell23-sys/mdcx/releases).\n\nVersioning follows [Semantic Versioning](https://semver.org/). The `.mdcx` format\nis read backwards-compatibly: a package written by an earlier version remains\nreadable by a later one.\n\n## Authorship\n\nConceived and directed by Jorge Ellena G., implemented with the assistance of\nClaude (Anthropic).\n\nDesign decisions in this package are recorded alongside the measurements that\njustify them, including the ones that were rejected. Several constants here are\nthe third value that was tried, and the two that failed are written down beside\nthem so the next attempt starts somewhere new. Where no measurement separated\ntwo options, that is recorded too, rather than settled by a heuristic that\nhappened to fit the material at hand.\n\n## Citation\n\nArchived on Zenodo with a permanent identifier. The concept DOI resolves to the\nlatest version:\n\n    https://doi.org/10.5281/zenodo.22015991\n\n## Licence\n\nApache 2.0. See\n[LICENSE](https://github.com/jorgell23-sys/mdcx/blob/main/LICENSE). Third-party\ncomponents and their licences are listed in\n[NOTICE](https://github.com/jorgell23-sys/mdcx/blob/main/NOTICE).\n\nPyMuPDF is not used. Its AGPL licence would require software incorporating this\npackage to be published under AGP",
  "bytes": 60000,
  "sha": "71b2d011b09b6483256aa1ec624c0ef93cb210d3fe4f2835bca0cf0632ae8844",
  "repo_slug": "jorgell23-sys/mdcx",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_jorgell23_sys_markdown_documen_8e3e1bdf/readme"
}