{
"markdown": "# memtrust\n\n\n\n\n\n[](https://github.com/RudrenduPaul/memtrust/actions/workflows/ci.yml)\n[](https://github.com/RudrenduPaul/memtrust/blob/main/LICENSE)\n[](https://pypi.org/project/memtrust-cli/)\n[](https://www.npmjs.com/package/memtrust-cli)\n[](https://pypi.org/project/memtrust-cli/)\n\n![\"memtrust](\"https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=1219788&theme=light\")
\n\nAgent memory backends each publish their own benchmark numbers, on different tests, measured\ndifferent ways. memtrust runs the same evals against all four and publishes the raw logs. Run\nagainst the vendors, not by them.\n\n\n\n```bash\npip install memtrust-cli\nmemtrust run --backends mempalace,mem0,zep,openviking --eval all\n```\n\n(For contributing to this repo instead of just running it, see [Development](#development) --\n`pip install -e \".[dev]\"` from a clone.)\n\n**Contents:** [Why this exists](#why-this-exists) · [What it does](#what-it-does) ·\n[Commands](#commands) · [How this differs](#how-this-differs-from-trusting-a-vendors-own-numbers) ·\n[Contradiction detection](#the-eval-that-actually-matters-contradiction-detection) ·\n[Compression fidelity](#the-eval-built-for-the-other-headline-overclaim-compression-fidelity) ·\n[Temporal-KG boundary](#the-eval-built-from-mempalaces-own-bug-temporal-kg-boundary-detection) ·\n[The landscape](#the-landscape-verified-not-benchmarked) · [Benchmarks](#benchmarks) ·\n[GitHub Actions usage](#github-actions-usage) · [Self-host](#self-host) · [Install](#install) ·\n[Hosted layer](#what-a-hosted-trust-layer-would-add) · [Backend coverage](#backend-coverage) ·\n[Development](#development) · [FAQ](#faq) · [License](#license) · [Success stories](#success-stories)\n\n## Why this exists\n\nIf you've compared agent-memory backends recently, you've probably noticed each one leads with a\ndifferent accuracy number, on a different benchmark, measured a different way. MemPalace's own\ncommunity already flagged the problem in public. Issue [#27](https://github.com/MemPalace/mempalace/issues/27)\non the MemPalace repository, opened April 7, 2026 and still open, documents that a headline 100%\nLongMemEval figure, measured with Haiku reranking, wasn't reproducible from the repository's own\nbenchmark scripts and was pulled from the README as unverifiable. A separate 96.6% figure people\ncite everywhere turns out to be mostly ChromaDB's default embeddings doing the work in raw mode,\nnot MemPalace's own architecture. A \"lossless\" compression claim (the \"AAAK\" mode) drops the same\nLongMemEval score from 96.6% to 84.2% in practice, a 12.4 percentage point gap. Two internal pull\nrequests attempting to fix the reporting problem, #433 and #729, were both closed without merging\non April 12, 2026 -- #729 within seven minutes of being opened. As of this writing, the issue has\n232 thumbs-up reactions and 39 comments.\n\nNone of that means MemPalace, or any other backend, doesn't work. It means nobody outside the\nvendor had run the same test, the same way, against every option, and published the raw logs.\n\nmemtrust does that. It runs LongMemEval, LoCoMo, and a growing set of evals built specifically for\nthis project -- 17 of them as of this writing, all registered in the CLI's `--eval` flag. The two\nthat matter most for understanding what this project is actually for:\ncontradiction detection, because neither LongMemEval nor LoCoMo tests the question that actually\nmatters once a memory system sits underneath a production agent -- what happens when a new fact\ncontradicts an old one? Does the backend flag the conflict? Silently overwrite the old fact with no\naudit trail? Serve whichever version it happens to retrieve first? None of the four backends this\nproject tracks publish a number for that. And compression/round-trip fidelity, built to directly\ntest claims like the \"lossless\" one above: it stores content, retrieves it, and scores literal\nreconstruction fidelity rather than semantic accuracy, per operating mode a backend exposes (see\n`MemoryBackendAdapter.supported_modes`) -- the mechanism that would let a contributor with live\nMemPalace credentials actually reproduce the 12.4-point compressed-mode accuracy drop\nmempalace/mempalace#27 documents, instead of just citing it. **Neither has been run against a live\nMemPalace instance as of this writing** -- both have, however, been run against a live\nself-hosted `mem0ai` install; see \"Benchmarks\" below. The other evals -- ranking quality,\ncrash recovery, extraction quality, embedding drift, scale/volume stress, lock contention, stats\naccuracy, orphan cleanup, result consistency, migration rollback, filter injection, resource-sync\nsafety, and temporal-KG boundary detection -- each grew out of a specific real bug report against\none of the four tracked backends; see \"Success stories\" below for the full list.\n\n**Where this stands right now, in one place:** [live benchmark results](#benchmarks) for one\nbackend (`mem0_direct`, self-hosted `mem0ai`), including a real bug this project's own attempt to\nget those numbers surfaced in mem0's default configuration; and [197 real GitHub issues and PRs](#success-stories)\nfiled against MemPalace, Mem0, Zep/Graphiti, and OpenViking independently root-caused against this\ncodebase -- 55 (28%) PASS, 16 (8%) PARTIAL, 42 (21%) a genuine capability gap, 84 (43%) not\napplicable, every verdict re-verified by a reviewer independent of whoever built the fix.\n\n## What it does\n\nEvery command below was actually run against this repo, with zero vendor API keys configured, to\nproduce the output shown. Nothing here is simulated.\n\n```\n$ memtrust run --backends mempalace,mem0,zep,openviking --eval all\nmemtrust 0.3.4 -- run_id=mt_2026-08-04T061759Z\nBackends: mempalace, mem0, zep, openviking Evals: longmemeval, locomo, contradiction,\nresource_sync_safety, compression, ranking_quality, scale_stress, embedding_drift, crash_recovery,\nextraction_quality, migration_rollback, filter_injection, lock_contention, stats_accuracy,\norphan_cleanup, result_consistency, temporal_kg_boundary\n\nmempalace: SKIPPED (not configured) -- mempalace is not configured: environment variable\nMEMPALACE_STORAGE_PATH is not set. Skipping this backend. See docs/methodology.md for setup\ninstructions.\nmem0: SKIPPED (not configured) -- mem0 is not configured: environment variable MEM0_API_KEY is not\nset. Skipping this backend. See docs/methodology.md for setup instructions.\nzep: SKIPPED (not configured) -- zep is not configured: environment variable ZEP_API_KEY is not set.\nSkipping this backend. See docs/methodology.md for setup instructions.\nopenviking: SKIPPED (not configured) -- openviking is not configured: environment variable\nOPENVIKING_API_KEY is not set. Skipping this backend. See docs/methodology.md for setup\ninstructions.\n\nCost: $0.00 (no LLM-judged evals ran -- structural evals only, or judge not configured)\n\nFull report: memtrust-report-2026-08-03.json\n```\n\nThat's the real, reproducible behavior of a fresh clone with no credentials: every backend reports\nSKIPPED, the command exits cleanly, and a valid JSON report is still written. `memtrust --version`\nnow correctly prints the installed version, matching `pip show memtrust-cli`. Earlier releases printed\n`0.0.0+unknown` even when properly installed, because `src/memtrust/__init__.py` read\n`importlib.metadata.version(\"memtrust\")` while the installed distribution is actually named\n`memtrust-cli` -- kept in the FAQ below for the record rather than deleted, since silently erasing\na bug the moment it's fixed is exactly the kind of curation this project exists to push back on in\nother people's benchmarks. Set the relevant environment variable for any backend you want to\nactually test (`MEM0_API_KEY`, `ZEP_API_KEY`, `OPENVIKING_API_KEY`, `MEMPALACE_STORAGE_PATH`) and\nthat backend runs for real against its live API instead of being skipped.\n\nThe eval logic itself is proven offline, against the bundled synthetic fixtures and, for several\nadapters, the real installed vendor packages with only the network boundary mocked, by the test\nsuite:\n\n```\n$ pytest --cov=memtrust --cov-report=term-missing\n... (33 module rows total; the 11 most relevant to this README are shown below)\nName Stmts Miss Cover\n-------------------------------------------------------------------------------------\nsrc/memtrust/adapters/base.py 290 1 99%\nsrc/memtrust/adapters/mempalace_adapter.py 265 15 94%\nsrc/memtrust/adapters/mem0_adapter.py 140 12 91%\nsrc/memtrust/adapters/mem0_direct_adapter.py 281 34 88%\nsrc/memtrust/adapters/openviking_adapter.py 178 18 90%\nsrc/memtrust/adapters/zep_graphiti_adapter.py 63 3 95%\nsrc/memtrust/adapters/zep_graphiti_selfhosted_adapter.py 165 24 85%\nsrc/memtrust/evals/contradiction.py 127 2 98%\nsrc/memtrust/evals/compression.py 86 1 99%\nsrc/memtrust/evals/temporal_kg_boundary.py 90 3 97%\nsrc/memtrust/receipt.py 118 10 92%\n-------------------------------------------------------------------------------------\nTOTAL 4167 265 94%\n\n590 passed, 8 skipped in 5.92s\n```\n\nThis is an excerpt, not the full table -- the weakest-covered module in the repo,\n`evals/mempalace_metadata_scale.py` (70%), isn't one of the 11 shown above; run the command\nyourself for the complete per-module breakdown.\n\n590 passing tests across 33 source modules, 94% overall statement coverage, 98% on the\ncontradiction-detection eval, 99% on compression/round-trip fidelity, 97% on the temporal-KG\nboundary eval, 85-99% across the adapter layer. The 8 skips are live-`mempalace`-package tests that\nonly run with the optional `mempalace-direct` extra installed (`pip install -e\n'.[dev,mempalace-direct]'`). Every test mocks its HTTP or wire\nlayer, or uses an in-memory fake backend -- none of them touch a real network, though a meaningful\nshare of the adapter tests now import and exercise *real installed vendor classes* directly\n(`mem0ai==2.0.12`'s embedder and vector-store modules, and -- gated behind the optional\n`mempalace-direct` extra -- the real `mempalace.mcp_server` functions), mocking only the outermost\nnetwork or wire-client boundary rather than the whole library. `graphiti-core` is not installed in\nthis environment, so its self-hosted adapter's tests still run against a hand-written Protocol\ndouble built to match the real package's confirmed method signatures, not the real classes -- see\n`docs/methodology.md`'s adapter confidence table for exactly which claim rests on which kind of\nverification.\n\n\n\n## Commands\n\n```\n$ memtrust --help\nUsage: memtrust [OPTIONS] COMMAND [ARGS]...\n\n memtrust: an independent, reproducible benchmark harness for agent-memory\n backends.\n\nOptions:\n --version Show the version and exit.\n --help Show this message and exit.\n\nCommands:\n keygen Generate a new Ed25519 keypair for signing `memtrust run`...\n report Read a prior `memtrust run` JSON report and print a formatted...\n run Run the eval suite against the requested backends.\n verify Verify a signed receipt produced by `memtrust run --sign`.\n```\n\n| Command | Flags | What it does |\n|---|---|---|\n| `memtrust run` | `--backends TEXT` comma-separated list or `all` (default `all`) · `--eval TEXT` comma-separated from `longmemeval,locomo,contradiction,resource_sync_safety,compression,ranking_quality,scale_stress,embedding_drift,crash_recovery,extraction_quality,migration_rollback,filter_injection,lock_contention,stats_accuracy,orphan_cleanup,result_consistency,temporal_kg_boundary`, or `all` (default `all`) · `--output FILE` (defaults to `./memtrust-report-.json`) · `--locomo-dataset-path FILE` points the LoCoMo eval at a real, downloaded `locomo10.json` instead of the bundled synthetic fixture (memtrust does not bundle or auto-fetch the real dataset) · `--locomo-exclude-question-ids-file FILE` excludes known-bad-ground-truth LoCoMo question IDs from scoring · `--scale-stress-n-records INTEGER` (default `500`) sets how many synthetic records the scale-stress eval stores and re-queries · `--sign FILE` writes a signed `