{
  "markdown": "# ARDoCo LLM Access\n\nA small, reusable Java library for accessing Large Language Models (LLMs) and embedding models through\n[LangChain4j](https://docs.langchain4j.dev/), with a pluggable caching layer for LLM requests and\nembeddings.\n\nIt is framework-neutral. Model settings are passed as plain configuration objects,\nwhile credentials and hosts are read from the environment. The\ncode was extracted and generalized from the [LiSSA](https://github.com/ardoco/lissa) project so that\nLiSSA, [ardoco](https://github.com/ardoco), and other tools can share one implementation.\n\n## Features\n\n- **Chat models** for OpenAI, Ollama, Blablador, DeepSeek, and Open WebUI, created lazily and\nconfigured via a typed builder.\n- **Cached requests**: single or n-fold LLM calls, or a transparent `CachingChatModel` decorator, backed by a cache.\n- **Embeddings** for OpenAI, Ollama, ONNX, Open WebUI, and a mock (for when embeddings are not required),\nwith automatic caching and token-length handling.\n- **Pluggable cache** with local-file, Redis, and REST-Redis backends, hierarchical layering, and\nconflict-resolution strategies. The on-disk format is compatible with LiSSA's existing caches.\n\n## Requirements\n\n- Java 21+\n- Maven\n\n## Installation\n\n```xml\n<dependency>\n<groupId>io.github.ardoco</groupId>\n<artifactId>llm-access</artifactId>\n<version>VERSION</version>\n</dependency>\n```\n\n## Usage\n\n### Chat model\n\n```java\nimport edu.kit.kastel.mcse.ardoco.llm.chat.*;\nimport dev.langchain4j.model.chat.ChatModel;\n\nLlmConfiguration config = LlmConfiguration.builder(ChatModelPlatform.OPENAI)\n\t\t.modelName(\"gpt-4o-mini\") // required\n\t\t.seed(133742243)          // optional\n\t\t.temperature(0.0)         // optional\n\t\t.build();\n\nChatModel model = new ChatModelProvider(config).createChatModel();\nString answer = model.chat(\"Hello!\");\n```\n\n### Cached requests\n\nWrap requests in a cache so repeated prompts are not re-sent to the model:\n\n```java\nimport edu.kit.kastel.mcse.ardoco.llm.cache.CacheManager;\nimport edu.kit.kastel.mcse.ardoco.llm.cache.Cache;\nimport edu.kit.kastel.mcse.ardoco.llm.cache.chat.ChatCacheKey;\nimport edu.kit.kastel.mcse.ardoco.llm.chat.*;\n\nCacheManager.setCacheDir(\"cache\");\nChatModelProvider provider = new ChatModelProvider(LlmConfiguration.of(ChatModelPlatform.OPENAI, \"gpt-4o-mini\"));\n\nCache<ChatCacheKey> cache = CacheManager.getDefaultInstance().getCache(provider, provider.cacheParameters());\nChatModel model = provider.createChatModel();\n\nString once = ChatModelUtils.cachedRequest(\"Summarize X\", model, cache);\nvar many = ChatModelUtils.nCachedRequest(\"Summarize X\", model, cache, 5); // 5 samples\n\ncache.flush(); // persist\n```\n\nAlternatively, wrap any `ChatModel` in a `CachingChatModel` decorator to cache transparently (including\nmulti-message chats) without changing call sites:\n\n```java\nChatModel cached = new CachingChatModel(provider.createChatModel(), cache);\ncached.chat(List.of(UserMessage.from(\"Summarize X\"))); // response cached by message content\n```\n\n### Embeddings\n\n```java\nimport edu.kit.kastel.mcse.ardoco.llm.cache.CacheManager;\nimport edu.kit.kastel.mcse.ardoco.llm.embedding.*;\n\nCacheManager.setCacheDir(\"cache\"); // required for the caching creators\nEmbeddingCreator creator = EmbeddingCreator.create(EmbeddingConfiguration.of(EmbeddingPlatform.OPENAI, \"text-embedding-3-large\"));\n\nfloat[] vector = creator.calculateEmbedding(\"some text\");\nvar vectors = creator.calculateEmbeddings(List.of(\"a\", \"b\", \"c\"));\n```\n\nONNX models need local files:\n\n```java\nEmbeddingCreator creator = EmbeddingCreator.create(\n\t\tEmbeddingConfiguration.onnx(\"bge-small\", \"/path/model.onnx\", \"/path/tokenizer.json\"));\n```\n\n## Configuration\n\nCredentials and hosts are read via `Environment`, which loads a `.env` file from the working directory\n(falling back to system environment variables). See [`sample.env`](sample.env) for a template.\n\n| Platform   | Chat env vars                                                      | Embedding env vars                                                |\n| ---------- | ------------------------------------------------------------------ | ----------------------------------------------------------------- |\n| OpenAI     | `OPENAI_API_KEY` (`OPENAI_ORGANIZATION_ID` optional)               | `OPENAI_API_KEY` (`OPENAI_ORGANIZATION_ID` optional)              |\n| Ollama     | `OLLAMA_HOST` (`OLLAMA_USER`+`OLLAMA_PASSWORD`, or `OLLAMA_TOKEN`) | `OLLAMA_EMBEDDING_HOST` (`OLLAMA_EMBEDDING_USER`, `..._PASSWORD`) |\n| Blablador  | `BLABLADOR_API_KEY`                                                | —                                                                 |\n| DeepSeek   | `DEEPSEEK_API_KEY`                                                 | —                                                                 |\n| Open WebUI | `OPENWEBUI_URL`, `OPENWEBUI_API_KEY`                               | `OPENWEBUI_URL`, `OPENWEBUI_API_KEY`                              |\n\n## Caching\n\nCaching is central to how this library is meant to be used: with a fixed `seed` and `temperature`,\nidentical requests are served from the cache instead of being re-sent to the model, which makes runs\nreproducible and keeps API cost and latency down. A `CacheManager` owns the configured backend(s) and\nhands out `Cache` instances; the model wrappers (`CachingChatModel`, the caching embedding creators) read\nand write through them automatically.\n\nBefore using the default manager, set the cache directory once:\n\n```java\nCacheManager.setCacheDir(\"cache\"); // getDefaultInstance() throws until this is called\n```\n\nAll cache behaviour (which backends, layering, conflict handling, connection details) is driven by\nenvironment variables, read when the `CacheManager` is constructed.\n\n### How entries are identified\n\nEvery entry is keyed by the model configuration (`model`, `seed`, `temperature`), the mode (`CHAT` vs\n`EMBEDDING`), and the request content. Because the mode is part of the key, chat and embedding entries\nnever collide, and different models/seeds/temperatures are kept apart automatically.\n\n- **Local (file) cache:** one JSON file per caller and model configuration, named\n`<Origin>_<model>_<seed>[_<temperature>].json` in the cache directory. `<Origin>` is the simple class\nname of the object that requested the cache, and `<temperature>` is omitted when it is `0.0` (kept for\nbackward compatibility with LiSSA's existing caches). Within a file, content is mapped to values via a\nUUID derived from the content.\n- **Redis / REST-Redis:** each entry is a Redis hash whose key is the JSON form of the cache key, with\nfields `data` (the stored value) and `timestamp`.\n\n### Backends\n\n| Type         | Storage                                | What you deploy                              | Use when                                                                                                  |\n| ------------ | -------------------------------------- | -------------------------------------------- | --------------------------------------------------------------------------------------------------------- |\n| `LOCAL`      | JSON files in the cache directory      | nothing                                      | single-machine or developer runs (the default)                                                            |\n| `REDIS`      | a Redis server, direct TCP (via Jedis) | a Redis server, reachable on its port        | a cache shared across machines/runs on a trusted network                                                  |\n| `REST_REDIS` | a Redis server behind an HTTP proxy    | a Redis server **and** the REST-Redis server | Redis is not reachable directly (firewall / HTTP-only egress) or you want HTTP auth in front of the cache |\n\nRedis and REST-Redis **fail fast**: if the backend cannot be reached at start-up (its `PING` fails), cache\ncreation throws instead of silently falling back. Pair a remote backend with `LOCAL` (see below) if you\nwant a local fallback layer.\n\n### Hierarchy and conflict resolution\n\n`CACHE_HIERARCHY` is a comma-separated list of backends, **primary first**. A single entry (e.g. `LOCAL`)\nmeans no layering; multiple entries stack the caches, with the first as the primary layer and the rest as\nfallbacks:\n\n```\nCACHE_HIERARCHY=REDIS,LOCAL        # read/write Redis first, fall back to a local file layer\nCACHE_HIERARCHY=REST_REDIS,LOCAL   # same, but reach Redis over HTTP\n```\n\nWrites are **write-through**: every `put` stores the response in all layers at once. On a read, all layers\nare consulted, and a value present in one layer but missing from another is copied into the layer that lacks\nit (**backfill**) — in either direction. `CACHE_REPLACEMENT_STRATEGY` decides what happens when two layers\nhold **different** values for the same key:\n\n| Strategy    | Behaviour                                                                |\n| ----------- | ------------------------------------------------------------------------ |\n| `NONE`      | (default) return the primary value; backfill layers that are missing it  |\n| `ERROR`     | throw `IllegalStateException` if two layers disagree on a key            |\n| `OVERWRITE` | overwrite the secondary layer with the primary value on conflict         |\n\n### Configuration reference\n\n| Variable                     | Applies to     | Default                  | Description                                                               |\n| ---------------------------- | -------------- | ------------------------ | ------------------------------------------------------------------------- |\n| `CACHE_HIERARCHY`            | all            | `LOCAL`                  | Comma-separated backends, primary first (`LOCAL`, `REDIS`, `REST_REDIS`)  |\n| `CACHE_REPLACEMENT_STRATEGY` | layered caches | `NONE`                   | Conflict handling between layers: `NONE`, `ERROR`, `OVERWRITE`            |\n| `REDIS_URL`                  | `REDIS`        | `redis://localhost:6379` | Redis connection URL                                                      |\n| `REST_REDIS_URI`             | `REST_REDIS`   | `http://localhost:8080`  | Base URL of the REST-Redis server (or a proxy in front of it)             |\n| `REST_REDIS_USERNAME`        | `REST_REDIS`   | —                        | HTTP Basic-auth username (optional; sent only when set)                   |\n| `REST_REDIS_PASSWORD`        | `REST_REDIS`   | —                        | HTTP Basic-auth password (optional; sent only when set)                   |\n\nThe cache **directory** is set in code via `CacheManager.setCacheDir(...)`, not through an environment\nvariable — pick where that value comes from in your own runner (e.g. a `LLM_CACHE_DIR` variable you read\nand pass in).\n\n### Deployment\n\n#### Local file cache\n\nNothing to deploy: the default `CACHE_HIERARCHY=LOCAL` writes JSON files into the directory passed to\n`CacheManager.setCacheDir(...)`. These files are self-contained, which makes `LOCAL` the format used for\n**replication packages** — commit or share the directory and others can reproduce a run offline (see\n[Replication packages](#replication-packages) below).\n\n#### Deploying Redis\n\nUse the `REDIS` backend when several machines or runs should share one cache over a trusted network.\n\n`docker-compose.yml`:\n\n```yaml\nservices:\nredis:\n\timage: redis:7.4\n\tcommand: [\"redis-server\", \"--appendonly\", \"yes\"] # persist to disk so the cache survives restarts\n\tports:\n\t- \"6379:6379\"\n\tvolumes:\n\t- redis-data:/data\n\nvolumes:\nredis-data:\n```\n\nThen point clients at it:\n\n```env\nCACHE_HIERARCHY=REDIS,LOCAL\nREDIS_URL=redis://redis-host:6379\n```\n\nRedis has no authentication by default. If it is reachable beyond a trusted network, enable a password\n(`--requirepass`) / TLS and put credentials in the URL (`rediss://user:pass@host:6380`).\n\n#### Deploying REST-Redis\n\nREST-Redis exists for the case where clients can only reach the cache over **HTTP**, not over the raw Redis\nTCP port (e.g. a shared team cache behind a reverse proxy, or restricted egress). It is a thin HTTP server\n([`org.fuchss:rest-redis`](https://central.sonatype.com/artifact/org.fuchss/rest-redis)) that proxies the\nhandful of operations the cache actually uses (`ping`, `exists`, `hget`, `hset`) to a real Redis. The\nmatching client is already bundled in this library, so applications only need `REST_REDIS_URI`.\n\nThe topology is:\n\n```\nclient (this library) --HTTP--> [reverse proxy: TLS + auth] --HTTP--> REST-Redis server --TCP--> Redis\n```\n\nThe server is published as a Docker image\n([`ghcr.io/dfuchss/rest-redis`](https://github.com/dfuchss/rest-redis)), so deployment is just Docker:\n\n1. **Write the server config** as `server_config.json`. It points the server at your Redis and picks the\nHTTP port to serve on:\n\n```json\n{\n\t\"redis_host\": \"redis\",\n\t\"redis_port\": 6379,\n\t\"http_port\": 8080\n}\n```\n\n2. **Run Redis and the REST-Redis server** together with Docker Compose:\n\n```yaml\nservices:\n\tredis:\n\timage: redis:7.4\n\tcommand: [\"redis-server\", \"--appendonly\", \"yes\"] # persist so the cache survives restarts\n\tvolumes:\n\t\t- redis-data:/data\n\n\trest-redis:\n\timage: ghcr.io/dfuchss/rest-redis\n\tdepends_on:\n\t\t- redis\n\tvolumes:\n\t\t- ./server_config.json:/app/server_config.json:ro\n\tports:\n\t\t- \"8080:8080\" # expose directly only on a trusted network — otherwise front it with a proxy (see below)\n\nvolumes:\n\tredis-data:\n```\n\nTo run only the server against an existing Redis, use the image directly:\n\n```bash\ndocker run -p 8080:8080 -v \"$(pwd)/server_config.json:/app/server_config.json:ro\" ghcr.io/dfuchss/rest-redis\n```\n\n3. **Point clients at it:**\n\n```env\nCACHE_HIERARCHY=REST_REDIS,LOCAL\nREST_REDIS_URI=http://rest-redis-host:8080\n# REST_REDIS_USERNAME / REST_REDIS_PASSWORD — only when a proxy in front enforces Basic auth\n```\n\nThe server has **no built-in authentication or TLS**. When exposing it beyond a trusted network, put a\nreverse proxy (nginx, Caddy, Traefik, …) in front to terminate TLS and Basic auth, point `REST_REDIS_URI`\nat the proxy, and set `REST_REDIS_USERNAME` / `REST_REDIS_PASSWORD` — the client sends them only when set.\n\n### Replication packages\n\nBecause `LOCAL` cache files are self-contained, the cache directory _is_ the replication artifact: ship it\nand anyone can reproduce a run **offline** — no API keys, no Redis, no model access — by pointing their\n`CacheManager` at it with `CACHE_HIERARCHY=LOCAL`.\n\nTo fill that local cache while running experiments against a shared Redis, layer `LOCAL` underneath it:\n\n```env\nCACHE_HIERARCHY=REDIS,LOCAL   # or REST_REDIS,LOCAL\n```\n\nWith this layering every response is write-through to both Redis and the local files, and any entry already\nin Redis (e.g. from an earlier run or a teammate) is backfilled into the local files the first time this run\nreads it. So after the experiment the local cache directory holds every request the run touched. Flush at\nthe end (`CacheManager.getDefaultInstance().flush()`) to make sure everything is on disk, then ship that\ndirectory as the replication package. Replicators unpack it, set `CACHE_HIERARCHY=LOCAL`, and re-run.\n\n### Choosing a backend\n\n- **Just running locally / developing, or shipping a replication package?** Keep the default `LOCAL`.\n- **Sharing a cache across machines on a trusted network (and/or building up a replication package)?**\n`REDIS,LOCAL`.\n- **Clients can only talk HTTP, or you need auth/TLS in front of the cache?** `REST_REDIS,LOCAL`.\n\n## Package overview\n\n| Package                                          | Contents                                                     |\n| ------------------------------------------------ | ------------------------------------------------------------ |\n| `edu.kit.kastel.mcse.ardoco.llm.chat`            | Chat model providers, platforms, lazy model, cached requests |\n| `edu.kit.kastel.mcse.ardoco.llm.embedding`       | Embedding creators and configuration                         |\n| `edu.kit.kastel.mcse.ardoco.llm.cache`           | Cache abstraction, backends, hierarchy, and manager          |\n| `edu.kit.kastel.mcse.ardoco.llm.cache.chat`      | Typed cache keys/parameters for chat requests                |\n| `edu.kit.kastel.mcse.ardoco.llm.cache.embedding` | Typed cache keys/parameters for embeddings                   |\n| `edu.kit.kastel.mcse.ardoco.llm.util`            | Environment/.env access, key generation, helpers             |\n\n## Building\n\n```bash\nmvn verify\n```\n\nTests that require Docker (the Testcontainers-based REST-Redis integration test) are skipped\nautomatically when no Docker environment is available.\n\n## License\n\nLicensed under the MIT License. See [LICENSE](LICENSE).\n",
  "bytes": 16532,
  "sha": "314a4c6cac5eb5a32f4c6ee58ae2caac2a726cf7ed083ec3ee938653b0f0aaa6",
  "repo_slug": "ardoco/llm-access",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/okf_ardoco_llm_access_openwiki_index_md_b50069a8/readme"
}