Use llamay models with MAQPNA

Serve a local GGUF model with llamay, route it through the MAQPNA gateway, and get a per-session inference context that forks with the session, checkpoints, and llamay's request audit anchored in the MAQPNA ledger.

llamay is the inference engine from Kosa8 Labs. It serves GGUF models on your own hardware, CPU or CUDA, with an OpenAI-compatible API. Its contexts (the model's KV and inference state) are objects: you can fork, snapshot and restore them. This page connects llamay to MAQPNA step by step, first on a laptop with maqpna dev, then in a cluster. When you finish, your agent makes governed model calls to a local model, and you have seen its inference context fork when its session forks.

What llamay adds to MAQPNA#

Any OpenAI-compatible server works as a MAQPNA model route. A route with provider: llamay gets three more things:

Everything else on the model path applies unchanged: scopes, budgets, token limits, data loss prevention (DLP), residency and failover. See Call models through the gateway.

sequenceDiagram
    autonumber
    participant A as Agent (session s1)
    participant G as MAQPNA gateway
    participant L as llamay
    A->>G: POST /llm/llamay/v1/chat/completions<br/>Bearer session token
    G->>G: verify token, scope models:llamay,<br/>budget, token limits, DLP
    G->>G: strip agent's llamay_session
    alt first call of s1 on this route
        G->>L: POST /v1/contexts {model}
        L-->>G: 201 {id: ctx_...}
    end
    G->>L: POST /v1/chat/completions<br/>gateway key, llamay_session=ctx_...
    L-->>G: completion + usage (cached_tokens)
    G->>G: DLP on completion, meter tokens,<br/>audit record ext.llamayContext
    G-->>A: completion

Before you begin#

• The maqpna CLI, version 2.1.0 or later. See Install.
• About 1 GB of free disk and memory for a small model such as qwen2.5:0.5b.
• jq, to read the ledger in the checks below.
• Optional, for a live sandbox fork: kosa8 on an Apple silicon Mac or x86-64 Linux with KVM.

Step 1: Install llamay#

Install llamay with the method llamay.com documents:

curl -fsSL https://llamay.com/install.sh | sh

LLAMAY_INSTALL_DIR="$HOME/.local/bin" curl -fsSL https://llamay.com/install.sh | sh

curl -fsSL https://llamay.com/install.sh | sh

irm https://llamay.com/install.ps1 | iex

Check it worked.

llamay version

It prints the version and build details. maqpna dev up --llamay looks for llamay next to the maqpna binary first, then on PATH. Point it at another binary with --llamay-bin PATH.

Step 2: Get a model#

Pull a small instruct model into llamay's store:

llamay pull qwen2.5:0.5b
llamay models

llamay models lists what the store holds. llamay also reads models that Ollama already has. Other name forms, such as hf:<owner>/<repo>/<file.gguf>, are in Getting models. You can skip the store and pass a GGUF file path instead, for example ~/models/qwen2.5-0.5b-instruct-q4_k_m.gguf.

Step 3: Start a local MAQPNA with llamay#

maqpna dev up --llamay qwen2.5:0.5b

--llamay MODEL takes a store name or a GGUF path. It does five things:

1. It writes a random API key to .maqpna/llamay.key (mode 0600).
2. It starts llamay serve -m MODEL -addr 127.0.0.1:11435 -audit .maqpna/llamay-audit.jsonl -audit-sync, with the key in the LLAMAY_API_KEY environment variable, never on the command line. Change the port with --llamay-port.
3. It waits up to 2 minutes for llamay's /healthz and reads the model name llamay reports.
4. It adds model route llamay to .maqpna/gateway.json with "provider": "llamay", the model name and the key file.
5. It adds the audit anchor dev (checked every 10 seconds, failClosed: true) and a checkpoint directory, .maqpna/checkpoints/llamay.

Check it worked.

maqpna dev status
jq '.models.llamay, .llamay' .maqpna/gateway.json

The route shows "provider": "llamay" and "upstream": "http://127.0.0.1:11435/v1". The llamay block shows auditAnchors and checkpointDir. llamay's log is .maqpna/logs/llamay.log.

Use your own gateway config instead#

If you run the gateway yourself, or you already run llamay, add the route and the llamay block to gateway.json by hand. Start llamay with -audit so there is a chain to anchor:

export LLAMAY_API_KEY="$(cat llamay.key)"
llamay serve -m qwen2.5:0.5b -addr 127.0.0.1:11435 -audit llamay-audit.jsonl -audit-sync

{
  "modelPolicy": "scope-only",
  "models": {
    "llamay": {
      "upstream": "http://127.0.0.1:11435/v1",
      "provider": "llamay",
      "model": "qwen2.5:0.5b",
      "apiKeyFile": "llamay.key",
      "jurisdiction": "EU-DE"
    }
  },
  "llamay": {
    "contextIdleSeconds": 1800,
    "checkpointDir": "checkpoints/llamay",
    "auditAnchors": [
      {"name": "local", "file": "llamay-audit.jsonl", "intervalSeconds": 30, "failClosed": true}
    ]
  }
}

Set model to the name llamay reports (curl -s 127.0.0.1:11435/healthz). Every field is in the llamay reference.

Step 4: Declare the agent#

On a laptop, maqpna dev run is the agent declaration: it mints a session token with models:* and exports the route. In a cluster, you declare an Agent with spec.model.provider: llamay:

apiVersion: maqpna.com/v1alpha1
kind: Agent
metadata:
  name: local-coder
  namespace: team-a
spec:
  image: registry.acme.eu/agents/coder:1.4.0
  tier: tier-1-microvm
  model:
    provider: llamay
    endpoint: http://maqpna-llamay.maqpna-system.svc:8081   # "/v1" is appended
    name: qwen2.5-coder-7b-instruct-q4_k_m
    jurisdiction: EU-DE
    tokensPerMinute: 60000
    maxTokensPerRequest: 4096

The operator renders route team-a.local-coder with "provider": "llamay" into the gateway's models.json. The value is matched case-insensitively. adapter must be unset or openai; the API server rejects anything else. The sandbox gets MAQPNA_MODEL_ENDPOINT={gateway}/llm/team-a.local-coder/v1 and never sees llamay's address or key. Run llamay in a cluster below sets up the endpoint.

Step 5: Make a governed model call#

Under maqpna dev run, the agent gets OPENAI_BASE_URL and MAQPNA_MODEL_ENDPOINT ({gateway}/llm/llamay/v1), OPENAI_API_KEY (its session token) and MAQPNA_MODEL_NAME (the model llamay serves). The agent uses its session token as the API key; llamay's key stays with the gateway.

pip install kosa8-maqpna openai

# agent.py
from openai import OpenAI
from maqpna import MaqpnaClient

client = MaqpnaClient.from_env()
llm = OpenAI(base_url=client.model_endpoint(), api_key=client.token())

messages = [{"role": "system", "content": "You are a careful planner. Answer in one short paragraph."},
            {"role": "user", "content": "Plan a database migration from MySQL to PostgreSQL."}]
for turn in range(2):
    reply = llm.chat.completions.create(
        model=client.model_name() or "default",   # the gateway forces the route's model
        messages=messages, max_tokens=120,
        extra_headers=client.openai_headers(),     # fresh token on every call
    )
    msg = reply.choices[0].message
    details = reply.usage.prompt_tokens_details
    print(f"turn {turn}: prompt={reply.usage.prompt_tokens} cached={details.cached_tokens if details else 0}")
    messages += [{"role": "assistant", "content": msg.content},
                 {"role": "user", "content": "Now list the riskiest step."}]

maqpna dev run --session plan-1 -- python agent.py

npm install @kosa8/maqpna openai

// agent.ts
import OpenAI from "openai";
import { MaqpnaClient } from "@kosa8/maqpna";

const client = MaqpnaClient.fromEnv();
const llm = new OpenAI({ baseURL: client.modelEndpoint(), apiKey: await client.token() });

const messages: OpenAI.Chat.ChatCompletionMessageParam[] = [
  { role: "system", content: "You are a careful planner. Answer in one short paragraph." },
  { role: "user", content: "Plan a database migration from MySQL to PostgreSQL." },
];
for (let turn = 0; turn < 2; turn++) {
  const reply = await llm.chat.completions.create(
    { model: client.modelName() ?? "default", messages, max_tokens: 120 },
    { headers: await client.openaiHeaders() },
  );
  const usage = reply.usage;
  console.log(`turn ${turn}: prompt=${usage?.prompt_tokens} cached=${usage?.prompt_tokens_details?.cached_tokens ?? 0}`);
  messages.push({ role: "assistant", content: reply.choices[0].message.content ?? "" },
                { role: "user", content: "Now list the riskiest step." });
}

maqpna dev run --session plan-1 -- npx tsx agent.ts

maqpna dev run --session plan-1 -- sh -c '
  curl -sS "$OPENAI_BASE_URL/chat/completions" \
    -H "Authorization: Bearer $OPENAI_API_KEY" -H "Content-Type: application/json" \
    -d "{\"model\":\"$MAQPNA_MODEL_NAME\",\"max_tokens\":60,
         \"messages\":[{\"role\":\"user\",\"content\":\"Plan a database migration.\"}]}"'

The second turn resends the whole conversation, and llamay reuses the prefix it kept in the session's context. On that turn cached_tokens is above zero. The exact numbers depend on the model.

Check it worked.

maqpna dev timeline --session plan-1
jq -c 'select(.session=="plan-1" and .ext.llamayContext) | {seq, tool, decision, ctx: .ext.llamayContext}' .maqpna/audit.jsonl

The timeline shows one model_call row per call on llm/llamay/chat.completions. In the ledger, both calls carry the same ext.llamayContext, an id of the form ctx_<32 hex>. The same id is in the context field of llamay's own records in .maqpna/llamay-audit.jsonl.

Step 6: Fork a session and see the context fork#

A forked session is a new session whose token names its parent in maqpna_parent_session. The first time it calls the llamay route, the gateway forks the parent's context (POST /v1/contexts/{parent}/fork?n=1) and binds the child to the copy.

flowchart LR
    subgraph parent["Session plan-1"]
        P1["turns 1-2"] --> PC["context ctx_A"]
    end
    PC -->|"child's first model call:<br/>POST /v1/contexts/ctx_A/fork"| CC["context ctx_B<br/>(copy-on-write pages)"]
    subgraph child["Session plan-1-alt (parent plan-1)"]
        CC --> C1["continues from<br/>ctx_A's exact state"]
    end
    PC --> P2["parent continues<br/>in ctx_A"]

Choose how you fork:

maqpna dev down                              # if the stack from Step 3 runs
maqpna dev up --kosa8 --llamay qwen2.5:0.5b
maqpna dev run --session plan-1 --kosa8-image python:3.12-alpine --kosa8-keep -- python3 -u -c "$(cat agent.py)"
# in a second terminal, while the agent runs
maqpna dev fork --session plan-1 --name plan-1-alt

eval "$(maqpna dev env)"     # MAQPNA_GATEWAY_URL, MAQPNA_BROKER_URL, MAQPNA_BROKER_TOKEN, MAQPNA_ADMIN_TOKEN
CHILD=$(curl -sS "$MAQPNA_BROKER_URL/v1/token" \
  -H "Authorization: Bearer $MAQPNA_BROKER_TOKEN" -H 'Content-Type: application/json' \
  -d '{"namespace":"dev","agent":"coder","user":"you@localhost","session":"plan-1-alt",
       "parentSession":"plan-1","scopes":["models:*"],"tier":"dev","ttlSeconds":900}' | jq -r .token)

curl -sS "$MAQPNA_GATEWAY_URL/llm/llamay/v1/chat/completions" \
  -H "Authorization: Bearer $CHILD" -H 'Content-Type: application/json' \
  -d '{"model":"llamay","max_tokens":60,"messages":[
        {"role":"system","content":"You are a careful planner. Answer in one short paragraph."},
        {"role":"user","content":"Plan a database migration from MySQL to PostgreSQL."},
        {"role":"user","content":"Instead, plan it as a blue-green cut-over."}]}' | jq .usage

Check it worked.

jq -c 'select(.ext.llamayContextParent) | {session, ctx: .ext.llamayContext, parent: .ext.llamayContextParent}' .maqpna/audit.jsonl
curl -s "$MAQPNA_GATEWAY_URL/metrics" | grep maqpna_gateway_llamay_context_ops_total

The child's first model call carries ext.llamayContextParent, which equals the parent's ext.llamayContext. The counter op="fork" went up by one.

If the parent has no context on this route (it made no model call, it reported its result, it was revoked, or its context idled out), the child gets a fresh context instead. The call still works; it only costs a full prefill.

Step 7: Checkpoint and restore a session's model state#

A fork copies the parent's state as it is now. A checkpoint saves it as it is now, to restore later into a new session. maqpna dev snapshot and maqpna dev restore do this together with a kosa8 sandbox snapshot. Without a sandbox, call the two admin routes yourself. They need llamay.checkpointDir, which dev up --llamay sets.

eval "$(maqpna dev env)"
curl -sS -X POST "$MAQPNA_GATEWAY_URL/v1/sessions/plan-1/llamay:checkpoint" \
  -H "Authorization: Bearer $MAQPNA_ADMIN_TOKEN" -H 'Content-Type: application/json' \
  -d '{"namespace":"dev"}' | tee ckpt.json | jq '.checkpoints[] | {context, file, sha256}'

Each saved context is a file in .maqpna/checkpoints/llamay (mode 0600) with its SHA-256, and a llamay/context-checkpoint record in the ledger. Restore it into a new session, plan-1-restored:

jq '{namespace, parentSession: .session, checkpoints}' ckpt.json |
curl -sS -X POST "$MAQPNA_GATEWAY_URL/v1/sessions/plan-1-restored/llamay:restore" \
  -H "Authorization: Bearer $MAQPNA_ADMIN_TOKEN" -H 'Content-Type: application/json' -d @- | jq .

The gateway re-checks each file's SHA-256, loads it into llamay (POST /v1/contexts/restore) and binds plan-1-restored to the restored context. The first model call of a plan-1-restored token (mint it as in Step 6) runs in that context. The restore is refused with 403 when either session is revoked, and with 400 into the checkpointed session itself.

Step 8: Verify llamay's request audit#

llamay writes one hash-chained record per HTTP request: counts, model, context and status, never prompts or completions. The gateway verifies the chain incrementally on every interval, re-reads the whole file every 30 checks, and writes a ledger record when the head moves.

Verify the chain yourself, offline:

maqpna llamay verify-audit .maqpna/llamay-audit.jsonl

llamay audit chain verified: <records> records, head seq <seq> <head hash>

The command exits with status 3 when the chain is broken. Give rotated files oldest first, with --continuation when the first file continues an earlier one (llamay -audit-continue).

See the anchors in the MAQPNA ledger:

jq -c 'select(.rule=="llamay/audit-anchor" or .rule=="llamay/audit-unverified") | {seq, rule, reason, ext}' .maqpna/audit.jsonl

Each llamay/audit-anchor record carries llamayAuditAnchor (the anchor name), llamayAuditSeq and llamayAuditHead. These records fall under MAQPNA's signed checkpoints and appear in evidence packs, so you can show that llamay's chain had head H at seq N when the MAQPNA ledger said so.

Run llamay in a cluster#

The Helm chart can run one llamay pod next to the gateway and make the gateway its only client. Add these values to your release:

llamay:
  enabled: true
  profile: cpu          # cuda for NVIDIA GPU nodes
  model: /models/qwen2.5-coder-7b-instruct-q4_k_m.gguf
  models:
    pull: hf:Qwen/Qwen2.5-Coder-7B-Instruct-GGUF/qwen2.5-coder-7b-instruct-q4_k_m.gguf
  audit:
    persistence: {accessModes: [ReadWriteMany]}
    anchor: {enabled: true, failClosed: true}

helm upgrade --install maqpna oci://$REGISTRY/charts/maqpna -n maqpna-system --reuse-values -f llamay-values.yaml

REGISTRY is the release registry listed on maqpna.com/download.

The chart renders a Deployment (one replica, Recreate), a Service on port 8081, a models PVC, an audit PVC that outlives the release, a generated API key Secret and a NetworkPolicy that admits only the gateway pods. llamay gets no egress unless models.pull is set. The gateway gets llamay.upstream, llamay.apiKeyFile and, with anchor.enabled, auditAnchors in its gateway.json. Air-gapped, set llamay.models.bundle.image instead of models.pull. All values are in the llamay reference.

Then apply the Agent from Step 4. Because its endpoint is on the chart's llamay origin, the gateway presents the chart's key itself: the Agent needs no apiKeySecretRef. For a llamay outside the cluster, set provider: llamay and apiKeySecretRef as for any endpoint.

Check it worked.

kubectl -n maqpna-system get deploy,svc,networkpolicy -l app.kubernetes.io/name=maqpna-llamay
kubectl -n maqpna-system port-forward svc/maqpna-gateway 8080:8080 &
curl -s localhost:8080/metrics | grep -E 'maqpna_gateway_llamay_(contexts|audit_verified)'

maqpna_gateway_llamay_audit_verified{anchor="in-cluster"} is 1 while the chain verifies. The full cluster path is in The stack on Kubernetes.

What can go wrong#

More cases are in Troubleshooting the stack.

Next steps#

• llamay reference: every config field, route, record and metric.
• Tutorial: build a fork-and-explore agent.
• Run MAQPNA sessions in kosa8 sandboxes.
• Call models through the gateway.
• Command reference: maqpna dev up, maqpna llamay verify-audit.