\n\n**OPTIMUZ is an open-source agentic AI inference runtime that we optimized specifically for Arm Neoverse V2 (GCP Axion).** We took a standard CPU-based agent stack (llama.cpp + MCP + naive tool routing) and applied six Arm-native optimizations — KleidiAI kernel acceleration, SVE2 BF16 matrix math, NUMA-aware thread pinning, semantic MCP context compression, CPU-CPU speculative cascade decoding, and a reasoning-token governor. The result: **63.8% higher throughput, 92.7% fewer wasted context tokens, and 95% lower per-request cost** than the unoptimized baseline on the same Arm hardware.\n\n> This project was built for the **Arm AI Optimization Challenge 2026 — Cloud AI Track**. Every optimization is reproducible, benchmarked with Arm Performix, and packaged as reusable artifacts for the Arm developer community.\n\n---\n\n## The Crisis This Solves\n\n**Agentic AI on the cloud is broken.** Most of the $0.40 - $2.00 per request is waste. Cloud agents waste money on unused MCP tool schemas, duplicated KV caches, excess reasoning tokens, and expensive GPU prices for memory-bound decode tasks. \n\n**Optimuz fixes this** with a three-tier CPU-CPU speculative cascade (0.5B → 3B → 8B) on **KleidiAI-optimized llama.cpp**, a semantic MCP tool router, a reasoning-token governor, and an evolution loop driven directly by **Arm Performix**.\n\n
\n\n| Pain | The Optimuz Fix |\n|---|---|\n| **Tool-schema flood** | Semantic MCP router (TurboVec + Top-K schema injection) |\n| **Reasoning-token burn** | RTG governor tied to confidence and KV pressure |\n| **KV duplication** | Shared KV path; CXL pooling when topology is present |\n| **GPU lock-in for decode** | CPU-CPU cascade + KleidiAI optimized exclusively for Axion |\n| **Invisible cost** | Grafana + Prometheus RMF dashboards |\n| **Untuned stacks** | AROP + Performix Instruction Mix recipes in the loop |\n\n
\n\n> [!IMPORTANT]\n> **Hardware Honesty:** Our live demo runs on **GCP Axion `c4a-standard-8`** (Neoverse-V2, SVE2/I8MM/BF16). Optimuz auto-detects NUMA/CXL/MTE at runtime and degrades safely on single-NUMA VMs like Axion, while activating NUMA-split cascades and CXL KV pooling natively on multi-socket Neoverse hosts.\n\n---\n\n## Interactive Architecture Explainer\n\nWe have built a fully interactive, self-contained architecture explainer documenting the 6 core pillars of Neuroswarm's design.\n\n\n[](https://omkarchaithanya.github.io/OPTIMUZ/explainer/index.html)\n\n### 1. Closed-Loop Optimization (AROP)\nDriven by `neuroswarm_arm/arop/tuner.py`, using `ARM Performix` to clamp parameters like `cascade_draft_k` and `governor_thinking_cap` based on live telemetry such as `tier1_hit_rate` and latency.\n\n### 2. Speculative Cascade (ASCR)\nDriven by `neuroswarm_arm/runtime/dipa/speculative/engine.py`, masking tool-call latency by predicting tool calls (B2) and running `executor.speculate(pred)` (B3) in parallel with cascade generation.\n\n### 3. Semantic MCP Router\nDriven by `neuroswarm_arm/tools/semantic_mcp_router.py` (TurboVec ANN), evaluating incoming queries alongside a `RouteContext` to route to the most relevant tools efficiently without bloating the LLM context window.\n\n### 4. CXL-Aware KV Cache (MAKS)\nDriven by `neuroswarm_arm/runtime/okf_slot_affinity.py` (`OkfSlotAffinity`), tracking `okf_block_hashes` and mapping them to a specific `id_slot` for zero-copy KV cache reuse across inference requests.\n\n### 5. Reasoning-Token Governor (RTG)\nDriven by `neuroswarm_arm/governor.py`, dynamically capping Chain-of-Thought output via a computed `thinking_token_cap` and `system_prompt` derived from `PlanState` signals (like `slo_remaining_ms` and `tool_confidence_top1`).\n\n### 6. Adaptive Quantization (AQR)\nDriven by `neuroswarm_arm/aqr.py` (`pick_quant`), matching workload profiles (`agent_role` and `workload_class`) to precision formats (like `Q5_K_M` or `Q4_0`) to balance reasoning quality against execution latency.\n\n---\n\n## The Optimization Story (Baseline → Change → Result)\n\n> The hackathon organizers explicitly reminded us: *\"Show what was optimized, what technical changes were made, and how those changes helped the project run better on Arm.\"* Here is that story, optimization by optimization.\n\n### Optimization 1: KleidiAI + SVE2 BF16 Kernel Acceleration\n\n| | Baseline | Optimized | Delta | How We Verify |\n|---|---|---|---|---|\n| **Throughput (xLAM-2-1B)** | 60.5 tok/s | **99.08 tok/s** | **+63.8%** | `benchmarks/kleidiai_baselines.json` |\n| **NEON Instruction Share** | 2.14% | **3.41%** | **+59%** | `03-instruction_mix_dynamic_kleidi.json` |\n| **Time-to-First-Token** | 1.12 s | **0.38 s** | **-66%** | Arm Performix Code Hotspots recipe |\n\n**What the baseline was:** Stock llama.cpp built with generic `cmake` flags on GCP Axion `c4a-standard-8`. No KleidiAI, no SVE2, no BF16. The model ran via standard GGML CPU backend with scalar FP32 fallback.\n\n**What we changed:**\n1. Rebuilt llama.cpp with KleidiAI micro-kernels via XNNPack:\n ```bash\n cmake -B build -DGGML_NATIVE=OFF -DGGML_CPU_ALL_VARIANTS=OFF \\\n -DGGML_OPENMP=ON -DGGML_BF16=ON -DGGML_NEON=ON \\\n -DCMAKE_BUILD_TYPE=Release \\\n -DCMAKE_C_FLAGS=\"-march=armv8.2-a+sve2+bf16\"\n cmake --build build --config Release -j$(nproc)\n ```\n2. Enabled KleidiAI `I8MM` / `SVE2` / `BF16` kernel paths at runtime via `GGML_KLEIDIAI=1`.\n3. Pinned threads to Neoverse V2 performance cores and disabled hyper-threading contention.\n\n**Why this is Arm-specific:** The `+sve2+bf16` flags and KleidiAI kernel fusion are **only available on Arm Neoverse V2/V3**. On x86 (Intel/AMD), these code paths do not exist — the same optimization is impossible. This is not a generic speedup; it is an **Arm-native competitive advantage**.\n\n**Evidence:** Arm Performix Instruction Mix report shows `libggml-cpu` consuming ~79% of CPU time in optimized SME2/SVE2 GEMM kernels vs. scalar fallback in baseline. See [`docs/evidence/performix/OPTIMIZATIONS.md`](./docs/evidence/performix/OPTIMIZATIONS.md).\n\n---\n\n### Optimization 2: Semantic MCP Tool Router (Context Bloat Elimination)\n\n| | Baseline | Optimized | Delta | How We Verify |\n|---|---|---|---|---|\n| **Tool-schema tokens per request** | ~143,000 (full catalog) | **~10,500 (Top-3)** | **-92.7%** | `work/benchmarks/router_mcpga.json` |\n| **Tool-selection accuracy** | 88.0% (naïve injection) | **100%** | **+12.0 pp** | `latest/layer-verify/06-router_accuracy.json` |\n| **MCP latency per call** | 600–3000 ms | **~150 ms** | **-75%** | End-to-end benchmark suite |\n\n**What the baseline was:** Standard MCP agent behavior — every tool schema from every connected server is injected into the system prompt. On a 40-tool deployment, this consumes 72% of the context window before the user sends a single token (Apideck/MCPGA benchmark).\n\n**What we changed:**\n1. Embedded tool descriptions with `nomic-embed-text-v1.5` (256-dim, Arm64-optimized NumPy backend).\n2. Built a TurboVec ANN index (4-bit quantized when active, exact NumPy fallback) for sub-2ms retrieval.\n3. At query time, embed user intent → retrieve Top-K=3 tools → inject only those 3 schemas into the LLM context.\n4. Fallback: if top-1 confidence < 0.42, trigger a BGE cross-encoder rerank.\n\n**Why this matters for Arm:** The embedding model (90M params) and ANN index fit entirely in Neoverse L2 cache. On x86 with lower L2-per-core density, the same router adds ~8ms latency. On Arm Neoverse V2, it adds **<2ms** — making real-time semantic routing viable for the first time.\n\n---\n\n### Optimization 3: CPU-CPU Speculative Cascade Decoding\n\n| | Baseline | Optimized | Delta | How We Verify |\n|---|---|---|---|---|\n| **DeepSeek-R1-Distill-Qwen-7B decode** | ~10 tok/s (single model) | **~18 tok/s** | **+80%** | `benchmarks/run_kpis.py --p1-spec-decode` |\n| **Draft-model acceptance rate** | N/A | **~72%** | — | Cascade trace logs |\n| **Cost per 1K tokens** | $0.0308 | **$0.00154** | **-95%** | `latest/layer-verify/08-economics.json` |\n\n**What the baseline was:** Single-model inference. Every token is generated by the full 8B parameter model. No speculative decoding, no cascade.\n\n**What we changed:**\n1. **Tier-0 Draft:** `DeepSeek-R1-Distill-Qwen-1.5B` (Q4_K_M, ~1.1 GB) generates K=5 candidate tokens.\n2. **Tier-1 Target:** `DeepSeek-R1-Distill-Llama-8B` (Q5_K_M, ~5.0 GB) verifies all 5 in parallel.\n3. **Thread affinity:** Draft pinned to cores 0–3, target pinned to cores 4–7 — zero L2 cache contention.\n4. Both models run **entirely on CPU** via llama.cpp + KleidiAI. No GPU required.\n\n**Why this is novel:** Published speculative-decoding work (Dovetail, DuoDecoding) assumes GPU draft + CPU target. We **inverted the assumption**: both draft and target run on Arm Neoverse cores, exploiting the 6 GB/s-per-core memory bandwidth and SVE2 BF16 MMLA. This is the first open-source CPU-CPU heterogeneous speculative decoder for agentic workloads.\n\n---\n\n### Optimization 4: Reasoning-Token Governor (RTG)\n\n| | Baseline | Optimized | Delta | How We Verify |\n|---|---|---|---|---|\n| **Thinking tokens per request** | ~5,000 (unbounded) | **~2,000** | **-60%** | `benchmarks/reasoning_cost.py` |\n| **GSM8K accuracy at cap=512** | 86.0% (unbounded) | **85.2%** | **-0.8 pp** | `benchmarks/reasoning_cost.py` |\n| **Cost per agent task** | ~$0.10–$1.00 | **~$0.0015–$0.02** | **-95%** | Economics trace |\n\n**What the baseline was:** Standard DeepSeek-R1 inference with no token budget. The model generates 3,000–8,000 \"thinking\" tokens before every visible answer, burning 80%+ of the inference budget.\n\n**What we changed:**\n1. Streaming observer on the target model output.\n2. Dynamic budget formula: `max_thinking = min(4096, 256 + 4 * confidence * 1024)`.\n3. Confidence signal comes from the semantic MCP router (Optimization 2). High-confidence tool calls get capped at 256 thinking tokens.\n4. Early-exit injection: when budget expires, inject `<|im_end|>` to force commitment.\n\n**Why this is unique:** No other open-source project ties reasoning-token budget to **tool-routing confidence**. The router signal makes the governor \"aware\" of task complexity — a connection only possible because OPTIMUZ owns both layers.\n\n---\n\n### Optimization 5: Multi-Agent KV-Cache Deduplication (MAKS)\n\n| | Baseline | Optimized | Delta | How We Verify |\n|---|---|---|---|---|\n| **KV pool size (3 agents, shared docs)** | 671 KB | **83 KB** | **-87.5%** | `latest/layer-verify/14-maks-dedup.json` |\n| **Concurrent agents per c4a-standard-8** | ~8 | **~24** | **+3×** | Load-test benchmark |\n\n**What the baseline was:** Each agent instance maintains an independent KV cache. Shared documents are re-encoded for every agent, causing 40–70% memory duplication.\n\n**What we changed:**\n1. Shared KV-Cache pool across all agents in a swarm.\n2. Content-addressed deduplication: identical prompts hash to the same KV pages.\n3. Arm MTE (Memory Tagging Extension) secures zero-copy sharing between agent processes.\n4. CXL-aware migration: when KV working set exceeds RAM threshold, cold pages migrate to CXL-attached memory (activated on multi-socket Neoverse hosts; gracefully degrades to NVMe pager on single-NUMA Axion).\n\n---\n\n### Optimization 6: HAOE Task-Graph Runtime with NUMA Awareness\n\n| | Baseline | Optimized | Delta | How We Verify |\n|---|---|---|---|---|\n| **Orchestration latency (5-turn agent)** | ~1,970 ms | **~1,116 ms** | **-43%** | `docs/evidence/performix/OPTIMIZATIONS.md` |\n| **HAOE fast-path hit rate** | 0% | **~68%** | — | Runtime metrics |\n\n**What the baseline was:** Standard LangChain-style Python orchestration — GIL-bound, single-threaded agent loop with blocking MCP calls.\n\n**What we changed:**\n1. **HAOE (Heterogeneous Agentic Orchestration Engine):** Task-graph runtime that schedules agent work as async DAGs.\n2. **NUMA-aware scheduling:** Tool calls, JSON parsing, and state serialization are distributed across Neoverse cores using SVE2-optimized string kernels.\n3. **Fast-path:** High-confidence chat requests bypass the full DAG and go straight to DIPA inference.\n4. **MCP process pool:** Warm stdio servers eliminate 600ms+ cold-start per tool call.\n\n\n---\n\n## The Optimuz Architecture\n\n
\n\n| Optimization | Description |\n|---|---|\n| **Matryoshka Embeddings Truncating** | Dynamically scales embedding dimensions for optimal latency and accuracy, minimizing redundant computations. |\n| **Turbovec Indexing** | Blazing-fast ANN (replaces FAISS) natively optimized for ARM architecture to perform rapid semantic searches. |\n| **Speculative Decoding** | Generates multiple draft tokens in parallel, vastly increasing throughput for language model outputs. |\n| **Speculative Tool Calling** | Overlaps draft tool prediction with main cascade generation (predict → overlap MCP → ToolOutputCache). |\n| **Model Cascading** | Intelligent multi-tier CPU cascade routing (Inspired by Google Cascade) that shifts workloads based on reasoning demands. |\n| **KV Cache Optimization** | Zero-waste MAKS multi-agent memory KV session deduplication, dropping memory bottlenecks at scale. |\n| **xLAM Model Integration** | Powered by Qwen model finetuned (xLAM) for highly effective instruction following and reasoning. |\n| **Mem0 (Zero) Memory Management Layer** | Highly efficient memory management layer that aggressively orchestrates active context windows. |\n| **Open Knowledge Format (OKF)** | Structured ontology and knowledge files seamlessly compiled and injected into agent contexts. |\n| **Quantization & Auto-Truncation** | Models are natively **Q4_0** quantized and automatically auto-truncated to **Q4_0_4_8** for extreme execution performance on Arm. |\n\n
\n\n```\n┌─────────────────────────────────────────────────────────────────────────────┐\n│ Plane 1: HAOE — Task-Graph Orchestration Runtime (NUMA + SVE2 aware) │\n│ ┌─────────────────────────────────────────────────────────────────────┐ │\n│ │ Plane 2: DIPA — Inference Kernel (KleidiAI + Speculative Cascade) │ │\n│ │ ┌─────────────────────────────────────────────────────────────┐ │ │\n│ │ │ Plane 3: ASCR — Adaptive Speculative Cascade Router │ │ │\n│ │ │ (0.5B draft → 3B verifier → 8B target, CPU-CPU only) │ │ │\n│ │ └─────────────────────────────────────────────────────────────┘ │ │\n│ └─────────────────────────────────────────────────────────────────────┘ │\n├─────────────────────────────────────────────────────────────────────────────┤\n│ Plane 4: MAKS — Shared KV-Cache Pool (MTE-secured, CXL-aware) │\n├─────────────────────────────────────────────────────────────────────────────┤\n│ Plane 5: RTG — Reasoning-Token Governor (confidence-aware budget) │\n└─────────────────────────────────────────────────────────────────────────────┘\n```\n**Data Flow:**\n```\nUser Query → HAOE Router → Semantic MCP Tool Selection (Top-K) → RTG Budget Set\n → DIPA Planner → ASCR Cascade (draft → verify) → llama.cpp + KleidiAI\n → KV Checkpoint in MAKS → Tool Call via MCP Pool → Response Stream\n```\n\n## Reusable Artifacts (For the Developer Community)\n\n> **Judges evaluate: \"Does it create reusable artifacts — optimized models, migration templates, prompt assets, or learning-ready content?\"** Here is our inventory.\n\n| Artifact | What It Is | How to Reuse | Location |\n|---|---|---|---|\n| **Optimized GGUF Models** | DeepSeek-R1-Distill variants quantized with KleidiAI-verified Q4_0/Q4_0_4_8) | Drop into any llama.cpp project on Arm | [`models/`](./models) |\n| **KleidiAI Build Script** | One-command cmake with correct `-march=armv8.2-a+sve2+bf16` flags | Copy to any llama.cpp project | [`scripts/build-kleidi.sh`](./scripts/build-kleidi.sh) |\n| **Migration Template** | Step-by-step guide: x86 GPU stack → Arm CPU stack | Follow for LangChain/CrewAI projects | [`templates/migration-x86-to-arm.md`](./templates/migration-x86-to-arm.md) |\n| **Helm Chart** | Kubernetes deployment for Graviton4/Axion/Cobalt clusters | `helm install optimuz ./helm/` | [`helm/`](./helm) |\n| **MCP Server Templates** | 3 reference MCP servers (echo, calc, weather) with Arm-optimized stdio | Copy and modify for your tools | [`mcp_servers/`](./mcp_servers) |\n| **Benchmark Suite** | Reproducible KPI scripts with Arm Performix integration | Run on your own Arm hardware | [`benchmarks/`](./benchmarks) |\n| **Performix Recipes** | Pre-configured `apx_recipe_run` configs for agent inference | Import into Performix GUI | [`performix/`](./performix) |\n| **Quantization Configs** | Q4_0 → Q4_0_4_8 auto-repacking specs for Arm Neoverse where llama.cpp doesn't support Q4_0_4_8 so it will repack after running llama.cpp function | Apply to your own models | [`configs/quantization/`](./configs/quantization) |\n| **OpenAPI Spec** | Full REST API spec for the inference gateway | Generate clients in any language | [`docs/openapi.yaml`](./docs/openapi.yaml) |\n\n---\n\n## Migration Value: From x86 GPU to Arm CPU (Track 2 Alignment)\n\n> **Track 2 judges specifically look for \"migration/adoption value.\"** This section proves OPTIMUZ is a migration enabler, not just a greenfield project.\n\n### The Problem with Current x86/GPU Agent Stacks\nA typical production agent stack today:\n- **Inference:** vLLM on NVIDIA A10G / H100 (cost: $1.00–$3.50/hr)\n- **Orchestration:** LangChain on x86 CPU (cost: $0.20/hr, but 90% of latency)\n- **MCP:** Naïve tool injection (72% context waste)\n- **Memory:** Per-agent KV cache (40–70% duplication)\n- **Total:** $0.10–$1.00 per agent task, 5–30× more tokens than necessary\n\n### The OPTIMUZ Migration Path\n| Step | Action | Time | Evidence |\n|---|---|---|---|\n| 1 | Replace vLLM with llama.cpp + KleidiAI on Axion | 30 min | [`templates/migration/01-inference.md`](./templates/migration/01-inference.md) |\n| 2 | Add semantic MCP router (drop-in middleware) | 15 min | [`templates/migration/02-mcp-router.md`](./templates/migration/02-mcp-router.md) |\n| 3 | Enable speculative cascade (config change) | 5 min | [`templates/migration/03-cascade.md`](./templates/migration/03-cascade.md) |\n| 4 | Activate RTG governor (env var) | 2 min | [`templates/migration/04-rtg.md`](./templates/migration/04-rtg.md) |\n| 5 | Deploy with Helm on Axion/Graviton/Cobalt | 10 min | [`helm/README.md`](./helm/README.md) |\n\n**Result:** Same agent capabilities, **$0.0015–$0.02 per task**, running on **$0.15/hr** Axion CPU instead of **$1.50/hr** GPU+x86.\n---\n\n---\n\n## Hardware Targets & Graceful Degradation\n\n| Platform | Cores | SVE2 | BF16 | KleidiAI | NUMA | CXL | OPTIMUZ Mode |\n|---|---|---|---|---|---|---|---|\n| **GCP Axion c4a-standard-8** | 8 | ✅ | ✅ | ✅ | Single | ❌ | **Primary Demo** — all optimizations active |\n| **AWS Graviton4 r8g.4xlarge** | 16 | ✅ | ✅ | ✅ | Single | ❌ | **Secondary** — higher concurrency, same code |\n| **AWS Graviton4 r8g.24xlarge** | 96 | ✅ | ✅ | ✅ | Dual | ❌ | **NUMA-split cascade** — draft on Node 0, target on Node 1 |\n| **Arm AGI CPU (dev kit)** | 136 | ✅ | ✅ | ✅ | Multi | ✅ | **Full CXL KV pooling** — rack-scale shared memory |\n| Azure Cobalt 100 | 64 | ✅ | ✅ | ✅ | Dual | ❌ | **Supported** — auto-detects topology |\n\n**Auto-Detection:** At startup, OPTIMUZ probes `/proc/cpuinfo`, `numactl`, and `cxl-list` to select the optimal configuration. No manual tuning required.\n\n\n## The 5-Plane Acronym Map\n\n
![\"Optimuz\"](\"./Optimuz_logo.png\"){kind=logo}
\n![\"Demo](\"https://img.shields.io/badge/Watch_Demo-YouTube-red?style=for-the-badge&logo=youtube\"){kind=icon}
![\"Python](\"https://img.shields.io/badge/Python-3.11%2B-3776AB?style=for-the-badge&logo=python&logoColor=white\"){kind=icon}
![\"FastAPI\"](\"https://img.shields.io/badge/FastAPI-API-009688?style=for-the-badge&logo=fastapi&logoColor=white\"){kind=icon}
![\"MIT](\"https://img.shields.io/badge/License-MIT-blue.svg?style=for-the-badge\"){kind=icon}
\n \n![\"Matryoshka](\"https://img.shields.io/badge/Matryoshka%20Embeddings-Truncating-9B59B6?style=for-the-badge&logoColor=white\"){kind=icon}
\n ![\"TurboVec\"](\"https://img.shields.io/badge/TurboVec-Indexing-F39C12?style=for-the-badge&logo=github&logoColor=white\"){kind=icon}
\n ![\"Speculative](\"https://img.shields.io/badge/Speculative-Decoding-E74C3C?style=for-the-badge&logoColor=white\"){kind=icon}
\n ![\"Speculative](\"https://img.shields.io/badge/Speculative%20Tool-Calling-3498DB?style=for-the-badge&logo=arxiv&logoColor=white\"){kind=icon}
\n ![\"Speculative](\"https://img.shields.io/badge/Speculative-Cascading-1ABC9C?style=for-the-badge&logo=google&logoColor=white\"){kind=icon}
\n ![\"xLAM](\"https://img.shields.io/badge/xLAM%20Model-Integration-34495E?style=for-the-badge&logo=github&logoColor=white\"){kind=icon}
\n ![\"Open](\"https://img.shields.io/badge/Open%20Knowledge-Format-2ECC71?style=for-the-badge&logo=google-cloud&logoColor=white\"){kind=icon}
\n ![\"Mem0\"](\"https://img.shields.io/badge/Mem0-Memory%20Management-16A085?style=for-the-badge&logo=github&logoColor=white\"){kind=icon}
\n