{
  "markdown": "<div align=\"center\">\n\n# AISIX AI Gateway\n\n### The open-source, Rust-native AI gateway for LLMs and AI agents\n\n**One OpenAI-compatible API in front of every model.** Route, govern, secure, cache, and\nobserve all your LLM and AI-agent traffic from a single control point — shipped as one\nstatic binary with low per-request overhead. Run it in your infrastructure for free,\nforever.\n\n*Built by the original creators of [Apache APISIX](https://apisix.apache.org/).*\n\n[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)\n[![Built with Rust](https://img.shields.io/badge/Built%20with-Rust-orange.svg)](https://www.rust-lang.org/)\n[![Docs](https://img.shields.io/badge/docs-read-3aa757.svg)](https://docs.api7.ai/ai-gateway/)\n[![Discord](https://img.shields.io/badge/Discord-join-5865F2.svg)](https://discord.gg/dUmRZ7Rvf)\n[![Website](https://img.shields.io/badge/website-api7.ai-1a73e8.svg)](https://api7.ai/ai-gateway)\n\n[**Start free**](https://api7.ai/ai-gateway?utm_source=github&utm_medium=readme&utm_campaign=ai-gateway) ·\n[**Documentation**](https://docs.api7.ai/ai-gateway/) ·\n[**Quickstart**](https://docs.api7.ai/ai-gateway/getting-started/gateway-quickstart) ·\n[**AISIX Cloud**](https://api7.ai/ai-gateway?utm_source=github&utm_medium=readme&utm_campaign=cloud) ·\n[**Roadmap**](ROADMAP.md)\n\n<br>\n\n<img src=\"assets/aisix-architecture.svg\" alt=\"AISIX AI Gateway architecture — one OpenAI- or Anthropic-compatible API in front of OpenAI, Anthropic, Gemini/Vertex, Bedrock, Azure OpenAI, and DeepSeek, with API key auth, rate and token limits, guardrails, caching, routing and failover, and observability in between\" width=\"100%\">\n\n</div>\n\n---\n\n**AISIX AI Gateway** is a Rust-native gateway that puts a single, OpenAI-compatible API in\nfront of every LLM provider — OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure OpenAI,\nDeepSeek, and any OpenAI-compatible endpoint. It gives platform teams one place to route,\ngovern, secure, and observe LLM traffic, with first-class SSE streaming and low gateway\noverhead.\n\nIt runs as a **single static binary** — low cold-start, lock-free config reads, and hot\nconfiguration reloads with no restarts: declare resources in one `resources.yaml` and\nreload on `SIGHUP`, or point the gateway at etcd for a multi-replica cluster. Run the\nopen-source gateway in your infrastructure, or connect it to\n**[AISIX Cloud](https://api7.ai/ai-gateway?utm_source=github&utm_medium=readme&utm_campaign=cloud)**\nfor centralized management with team governance, budgets, audit, and a dashboard.\n\n> **AISIX AI Gateway (this repo)** is the open-source product. It runs without a control\n> plane using declarative configuration or etcd. When connected to\n> **[AISIX Cloud](https://api7.ai/ai-gateway?utm_source=github&utm_medium=readme&utm_campaign=cloud)**,\n> the same gateway serves as the data plane. AISIX Cloud adds a commercial control plane,\n> either hosted by API7 (**Hybrid Cloud**) or hosted by you in your infrastructure\n> (**On-Premises**). In both options, the gateway runs in your environment and calls\n> providers directly; live AI traffic does not pass through the control plane or API7.\n> The proxy API is identical throughout.\n> **[Talk to us about AISIX Cloud →](https://api7.ai/contact?utm_source=github&utm_medium=readme&utm_campaign=cloud)**\n\n## ⚡ Quickstart\n\nOne container. No control plane, no database, no configuration store — the gateway reads\nevery dynamic resource from one declarative `resources.yaml`.\n\n```yaml\n# config.yaml\nresources_file: /etc/aisix/resources.yaml\nproxy:\n  addr: \"0.0.0.0:3000\"\nadmin:\n  enabled: false          # a declarative gateway needs no admin listener\nobservability:\n  metrics:\n    prometheus:\n      enabled: true\n      addr: \"0.0.0.0:9090\"\n```\n\n```yaml\n# resources.yaml\n_format_version: \"1\"\n\nprovider_keys:\n  - display_name: openai-main\n    provider: openai\n    api_key: ${OPENAI_API_KEY}        # interpolated from the environment\n\nmodels:\n  - display_name: my-model\n    provider: openai\n    model_name: gpt-4o-mini\n    provider_key: openai-main\n\napi_keys:\n  - display_name: local-dev\n    key_env: CALLER_API_KEY           # hashed at load; the plaintext is never stored\n    allowed_models: [\"my-model\"]\n```\n\n```bash\nexport OPENAI_API_KEY=\"YOUR_PROVIDER_KEY\"\nexport CALLER_API_KEY=\"YOUR_CALLER_KEY\"\n\ndocker run -d --name aisix \\\n  --platform linux/amd64 \\\n  -v \"$(pwd)/config.yaml:/etc/aisix/config.yaml:ro\" \\\n  -v \"$(pwd)/resources.yaml:/etc/aisix/resources.yaml:ro\" \\\n  -e OPENAI_API_KEY -e CALLER_API_KEY \\\n  -p 3000:3000 -p 127.0.0.1:9090:9090 \\\n  ghcr.io/api7/aisix:latest        # proxy → :3000, metrics + status → :9090\n#                                  ^ the metrics/status listener is unauthenticated;\n#                                    keep it on loopback or a private network\n```\n\nThen call the gateway exactly like OpenAI:\n\n```bash\ncurl http://localhost:3000/v1/chat/completions \\\n  -H \"Authorization: Bearer $CALLER_API_KEY\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"model\":\"my-model\",\"messages\":[{\"role\":\"user\",\"content\":\"hello\"}]}'\n```\n\nEdit `resources.yaml` and send `SIGHUP` (`docker kill -s HUP aisix`) to apply changes with\nno restart — an invalid file is rejected whole and the last good configuration keeps\nserving. Check a file before booting with `aisix validate --resources resources.yaml`.\n\nFull walkthrough: the\n[Gateway Quickstart](https://docs.api7.ai/ai-gateway/getting-started/gateway-quickstart) ·\nevery field: the [resources file reference](https://docs.api7.ai/ai-gateway/reference/resources-file).\nFor a multi-replica cluster, point the gateway at etcd instead — `resources_file` and\n`etcd` are mutually exclusive.\n\n## ✨ Why AISIX\n\n- **One API, every model.** Speak the OpenAI *or* Anthropic wire format in; the gateway\n  translates to whichever provider each model points at. Point an OpenAI or Claude SDK at\n  one `base_url` and switch models without changing code.\n- **A real gateway, in Rust.** Single static binary, low cold-start, lock-free config reads\n  on the hot path, native streaming.\n- **Open source, free forever.** Apache-2.0 licensed and built to run in your\n  infrastructure. Choose AISIX Cloud when you want centralized management through a\n  control plane and dashboard.\n- **Production controls built in.** Routing & failover, rate limits, guardrails, caching,\n  and observability ship in the box. (Budgets and spend caps are an AISIX Cloud feature —\n  the gateway enforces the control plane's decisions.)\n\n## 🧩 Features — available today\n\nCovered by 183 end-to-end scenario files (496 cases) that run against real gateway processes.\n\n- **OpenAI-compatible proxy** (`:3000`) — `chat/completions`, `completions`, `responses`,\n  `embeddings`, `rerank`, `images/{generations,edits}`, `audio/{speech,transcriptions,translations}`,\n  `videos` (submit → poll → fetch), `files`, `batches`, `fine_tuning/jobs`, `realtime`,\n  `GET /v1/models`, plus a root-level `/passthrough/:provider/*` escape hatch. Native SSE streaming,\n  tool/function calling, JSON mode, vision/multimodal input, and reasoning-content support.\n- **Anthropic Messages API** — `POST /v1/messages` as a first-class route, working against\n  **any** configured upstream: requests and responses (including streaming) are translated\n  both ways when a model points at a non-Anthropic provider.\n- **Routing & failover** — virtual/routing models with six strategies: `round_robin`\n  (smooth weighted round-robin), `consistent_hash` (session affinity keyed by header /\n  cookie / API key / client IP), `failover`, plus metric-based `least_cost`,\n  `least_latency`, and `least_busy`. Per-target `priority` tiers (active/backup pools),\n  retry budgets, cooldowns, tag-conditional targets, and per-attempt timeouts.\n- **Ensemble models** — fan one request out to a panel of models concurrently, then have a\n  judge model synthesize a single answer, with a minimum-successful-responses threshold.\n- **Semantic routing** — one virtual model that dispatches by the *meaning* of each\n  request: it embeds the prompt, scores it against per-route example utterances, and routes\n  to the best match (or a default). See the\n  [semantic routing docs](https://docs.api7.ai/ai-gateway/routing/semantic-routing).\n- **Rate limiting & concurrency** — RPS/RPM/RPH/RPD + TPM/TPD + concurrency caps,\n  AND-combined across caller keys, models, and policy scopes (`api_key` / `model` / `team` /\n  `member` / `team_member`). Counters are per-process by default, or shared across replicas\n  with the Redis backend.\n- **Guardrails** — content-policy enforcement on input and output, in-process or through a\n  provider: keyword/regex, built-in PII detection and redaction, Presidio, Lakera, OpenAI\n  Moderation, AWS Bedrock Guardrails, Azure AI Content Safety (Prompt Shield + text\n  moderation), and two Alibaba Cloud services. A block returns `422 content_filter`;\n  monitor mode records what would have happened without blocking.\n- **Caching** — exact-match response cache with per-policy TTL and model/key scope matchers;\n  memory and Redis backends; cost-saved telemetry on every hit. Separately, **automatic\n  prompt caching** can be enabled per direct Anthropic model to inject cache breakpoints, so\n  callers get provider-side prompt discounts without changing their requests.\n- **MCP gateway** — front registered upstream MCP servers at `/mcp` with gateway-held\n  credentials, per-server tool namespaces, and per-caller access. It serves every\n  Streamable HTTP revision from `2025-03-26` through stateless `2026-07-28` without\n  downstream sessions. Upstreams use `initialize` by default or `server/discover` with\n  `protocol_version: \"2026-07-28\"`. CI runs the official MCP suite's applicable tools-only\n  protocol scenarios. Also exposes a REST API as MCP tools from its OpenAPI description.\n- **A2A agent gateway** — front A2A (Agent-to-Agent) agents at `/a2a/:agent`, serving each\n  agent's card with URLs rewritten to the gateway, over JSON-RPC 2.0.\n- **Inbound authentication** — caller API keys (SHA-256 hashed, model allowlists, expiry,\n  rotation), or OIDC/JWT bearer tokens validated against registered providers (Entra ID,\n  Okta, Google Workspace, or any OIDC issuer) with JWKS caching.\n- **Observability** — Prometheus `/metrics`, structured per-request access logs, usage\n  events, OTLP/GenAI span export (Langfuse, Honeycomb, Grafana Cloud, or any OTLP receiver),\n  plus dedicated Datadog and Aliyun SLS log exporters and object-storage (S3/GCS/Azure Blob)\n  telemetry.\n- **Declarative configuration** — one `resources.yaml` carries all ten resource collections\n  (provider keys, models, caller keys, guardrails, MCP servers, A2A agents, cache policies,\n  observability exporters, rate-limit policies, OIDC providers), validated against the same\n  JSON Schemas the gateway uses at runtime. `aisix validate` checks a file offline; `SIGHUP`\n  reloads it atomically.\n- **Operational endpoints** — `/livez` and `/readyz` on the proxy listener; `/status/config`,\n  `/status/ready`, `/status/models`, and Prometheus `/metrics` on a dedicated metrics\n  listener (`:9090`). The admin listener (`:3001`) additionally serves a **read-only**\n  resource surface, OpenAPI 3 with a Scalar UI, and a playground. Resources are managed\n  declaratively — through the `resources_file` (reloaded on SIGHUP) or direct etcd\n  writes — not through the admin listener; its former write endpoints were removed.\n\n## 🔌 Supported providers\n\nAISIX dispatches through **five native adapter families** — distinct wire-protocol bridges,\nnot one generic relabel. Whatever the upstream protocol, the client-facing API stays\nOpenAI-shaped.\n\n| Adapter family | Reaches | Wire shape · auth |\n|---|---|---|\n| `openai` | OpenAI **+ any OpenAI-compatible vendor** — DeepSeek, Groq, Mistral, Together, Fireworks, Perplexity, vLLM, Ollama, or self-hosted OpenAI-compatible endpoints | OpenAI chat completions · Bearer |\n| `anthropic` | Anthropic Claude | Anthropic Messages · `x-api-key` |\n| `bedrock` | AWS Bedrock — Anthropic, Meta Llama, Mistral, Cohere, Amazon Titan/Nova, AI21 | Bedrock Converse + `/invoke` · SigV4 |\n| `vertex` | Google Vertex AI (Gemini) | Vertex `:generateContent` · OAuth2 |\n| `azure-openai` | Azure OpenAI | Azure deployments · api-key / Entra ID |\n\nPlus specialized handling for vendor quirks (e.g. DeepSeek reasoning content) and dedicated\n**rerank / embeddings** vendors (Cohere, Jina). Details in\n[adapter protocol families](https://docs.api7.ai/ai-gateway/providers/adapters).\n\n## ☁️ Open source vs AISIX Cloud\n\nSame gateway binary, same proxy API — in every form the gateway runs in your environment.\n**AISIX Cloud** adds a commercial control plane, either hosted by API7\n(**Hybrid Cloud**) or hosted in your infrastructure (**On-Premises**).\n\n<table>\n  <tr>\n    <td width=\"50%\" valign=\"top\">\n      <img src=\"assets/console-overview.png\" alt=\"AISIX Cloud overview — requests, latency p50/p99, error rate and cost today, with a 7-day request-and-cost trend and data-plane health\" width=\"100%\"><br>\n      <sub><b>Overview</b> — traffic, latency, error rate &amp; spend at a glance</sub>\n      <br><br>\n      <img src=\"assets/console-models.png\" alt=\"AISIX Cloud models — alias an upstream LLM per provider (OpenAI, Anthropic, AWS Bedrock, DeepSeek) with model IDs and per-model rate limits\" width=\"100%\"><br>\n      <sub><b>Models</b> — one alias per upstream: OpenAI, Anthropic, Bedrock, DeepSeek…</sub>\n      <br><br>\n      <img src=\"assets/console-guardrails.png\" alt=\"AISIX Cloud guardrails — pre-input and post-output content policies (keyword blocklist, Azure Content Safety, AWS Bedrock) that block on violation\" width=\"100%\"><br>\n      <sub><b>Guardrails</b> — pre-input &amp; post-output policies, block on violation</sub>\n    </td>\n    <td width=\"50%\" valign=\"top\">\n      <img src=\"assets/console-playground.png\" alt=\"AISIX Cloud playground — pick a model, set system and user prompts, run, and read the response with live token and cost metering\" width=\"100%\"><br>\n      <sub><b>Playground</b> — test any model with live token &amp; cost metering</sub>\n      <br><br>\n      <img src=\"assets/console-observability.png\" alt=\"AISIX Cloud observability exporters — fan out chat-completion telemetry to OTLP, Datadog and object storage, with per-target delivery health\" width=\"100%\"><br>\n      <sub><b>Observability</b> — fan out traces &amp; logs to OTLP, Datadog, object storage</sub>\n      <br><br>\n      <img src=\"assets/console-budgets.png\" alt=\"AISIX Cloud budgets — organization and per-environment spend caps with progress bars, hard-stop versus warn-only, including an over-budget policy\" width=\"100%\"><br>\n      <sub><b>Budgets</b> — hard-stop spend caps with warn-only tiers</sub>\n    </td>\n  </tr>\n</table>\n\n<p align=\"center\">\n  <em>The AISIX Cloud dashboard — overview metrics, multi-provider models, guardrails, budgets (with hard-stop spend caps), and observability exporters, across all your gateways.</em>\n  <br><br>\n  <a href=\"https://aisix-demo.api7.ai/\"><b>▶ Try the live dashboard demo — aisix-demo.api7.ai</b></a>\n</p>\n\n| | Open-source gateway (this repo) | [AISIX Cloud](https://api7.ai/ai-gateway?utm_source=github&utm_medium=readme&utm_campaign=cloud) (Hybrid Cloud or On-Premises) |\n|---|---|---|\n| Price | Free · Apache-2.0 · forever | Commercial — [talk to us](https://api7.ai/contact?utm_source=github&utm_medium=readme&utm_campaign=pricing) |\n| Configuration | Declarative `resources.yaml`, or etcd for a cluster | Dashboard + Cloud Admin API, multi-environment |\n| Tenancy | Single instance / namespace | Org → Team → Member → Environment |\n| Provider keys | In the resources file as `${VAR}` env references, or in etcd | Envelope-encrypted at rest, write-only, in-place rotation |\n| Inbound auth | Caller keys (SHA-256 hashed, model allowlists, expiry), or OIDC/JWT bearers | Same, plus masked reveal, key ownership, and PATs |\n| Budgets | — (rate and token limits only) | Per key / provider / env / org / team, hard-stop & alerts |\n| RBAC | Admin key = read-only resource surface | Org roles (owner / admin / member), invites |\n| Audit log | — | Full org-scoped audit with diff viewer |\n| Usage & cost | Export logs, metrics, and usage events yourself | Managed usage views, model pricing catalog, spend reporting |\n| Surface | Status endpoints, OpenAPI read surface, playground | Full dashboard + per-environment playground |\n\n→ **Want the AISIX Cloud control plane, governance, budgets, and dashboard?**\n**[Talk to API7](https://api7.ai/contact?utm_source=github&utm_medium=readme&utm_campaign=cloud)** about\nHybrid Cloud or On-Premises, or **[book a demo](https://api7.ai/contact?utm_source=github&utm_medium=readme&utm_campaign=demo)**.\n\n## 🏗️ Architecture\n\nA single Cargo workspace; the `aisix-server` crate builds one binary named `aisix` that\nwires the crates together.\n\n```text\ncrates/\n├── aisix-core           Config, snapshot, resource model, resources.yaml source, errors\n├── aisix-etcd           Config provider + watch supervisor\n├── aisix-gateway        Hub & bridge, SSE parser, provider trait\n├── aisix-proxy          /v1/*, /mcp, /a2a handlers, routing, middleware\n├── aisix-admin          Read-only resource surface + playground + OpenAPI\n├── aisix-provider-*     openai · anthropic · azure-openai · bedrock · vertex\n├── aisix-mcp            MCP gateway — server registry, tool ACL, transports\n├── aisix-a2a            A2A agent gateway — agent cards, JSON-RPC bridge\n├── aisix-ratelimit      fixed-window + token accounting + concurrency (local | redis)\n├── aisix-cache          memory + redis backends\n├── aisix-redis          shared Redis connection for cache + rate limits\n├── aisix-guardrails     pre/post content-policy hooks\n├── aisix-obs            tracing, metrics, access log, exporters\n└── aisix-server         the `aisix` binary — bootstrap + CLI\n```\n\n## 🗺️ Roadmap\n\nHighlights on the [roadmap](ROADMAP.md); tracked live in\n[issues](https://github.com/api7/aisix/issues):\n\n- Semantic (embedding-similarity) response caching\n- More observability sinks — Langsmith, Helicone, Slack alerts\n- Prompt templates managed as gateway resources\n- Llama-Guard as a guardrail provider\n\nShipped since this list was last written: the MCP gateway, the A2A agent gateway,\nOIDC/JWT inbound auth, Redis-backed distributed rate limiting, and the Lakera, Presidio,\nPII, and OpenAI Moderation guardrails — see **Features** above.\n\n## 🛠️ Development\n\nPrerequisites: the Rust toolchain pinned in `rust-toolchain.toml`. Docker is only needed\nfor the tests that exercise etcd, Redis, or provider emulators.\n\n```bash\ncargo check --workspace\ncargo fmt --check\ncargo clippy --workspace -- -D warnings\ncargo test --workspace\n\n# Coverage (matches the CI gate)\ncargo llvm-cov --workspace --lcov --output-path lcov.info\n\n# Run locally against a resources.yaml (no etcd needed). Copy the Quickstart's two files\n# and change resources_file to the local path, e.g. resources_file: ./resources.yaml\ncargo run -p aisix-server --bin aisix -- --config config.local.yaml\n\n# Check a resources file without starting a listener\ncargo run -p aisix-server --bin aisix -- validate --resources resources.yaml\n```\n\n## 💬 Community\n\n- **Discord** — [discord.gg/dUmRZ7Rvf](https://discord.gg/dUmRZ7Rvf)\n- **Issues & discussions** — [github.com/api7/aisix/issues](https://github.com/api7/aisix/issues)\n- **Contributing** — [CONTRIBUTING.md](CONTRIBUTING.md)\n- **Website** — [api7.ai/ai-gateway](https://api7.ai/ai-gateway?utm_source=github&utm_medium=readme)\n\nIf AISIX is useful to you, a ⭐ helps other engineers find it.\n\n## 📄 License\n\n[Apache 2.0](LICENSE).\n",
  "bytes": 19609,
  "sha": "3f79d086fcc3e50ab9bc23303d5148b06c9c50e765b9bf7649111a5e90a74aab",
  "repo_slug": "api7/aisix",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_api7_aisix_0b397d36/readme"
}