MAQPNADocs

Serve models with llamay

Run the llamay inference engine as a governed model provider, with a forkable inference context per session and its request audit anchored in the ledger.

llamay is a local-first inference engine: GGUF models, pure Go, CPU or CUDA. Its contexts (the model's KV and inference state) can be forked, snapshotted and restored. With llamay as a provider, MAQPNA does three things:

  • gives every agent session its own llamay context. A long agent conversation is not prefilled again on every turn.
  • forks the context when you fork the session. A forked agent continues from its parent's exact inference state, just as its sandbox continues from the parent's state.
  • anchors llamay's hash-chained request audit in the MAQPNA ledger. It can also take llamay out of routing while that chain does not verify.

Everything else on the model path applies unchanged: scopes, policies, budgets, token limits, DLP, guards and residency.

Before you begin#

  • On a laptop: the llamay binary from llamay.com/download, and a GGUF model or a llamay store name.
  • In a cluster: MAQPNA installed with Helm, and the llamay container image (mirrored into your registry if you are air-gapped).

On your laptop#

maqpna dev up --llamay ~/models/qwen2.5-0.5b-instruct-q4_k_m.gguf
maqpna dev run -- python agent.py     # OPENAI_BASE_URL points at model route "llamay"
maqpna dev timeline --last            # each model call shows its llamay context

dev up --llamay starts llamay serve next to the gateway. The API key is passed in the environment, and the request audit goes to .maqpna/llamay-audit.jsonl. dev up also adds model route llamay, and the gateway re-verifies the audit chain every 10 seconds.

To verify the chain yourself, offline:

maqpna llamay verify-audit .maqpna/llamay-audit.jsonl

In a cluster#

Enable the chart's llamay:

llamay:
  enabled: true
  profile: cpu        # cuda for GPU nodes
  model: /models/qwen2.5-coder-7b-instruct-q4_k_m.gguf
  models:
    pull: hf:Qwen/Qwen2.5-Coder-7B-Instruct-GGUF/qwen2.5-coder-7b-instruct-q4_k_m.gguf
  audit:
    persistence: {accessModes: [ReadWriteMany]}
    anchor: {enabled: true}

The chart runs one llamay pod and makes the gateway its only client:

  • A NetworkPolicy admits the gateway pods alone, and the pod has no egress unless it pulls the model.
  • The API key Secret is mounted into the gateway only.
  • llamay's request audit is required to serve: a record that cannot be written stops llamay rather than going missing.

When air-gapped, ship the model as llamay.models.bundle.image instead of models.pull.

Point an agent at it:

apiVersion: maqpna.com/v1alpha1
kind: Agent
metadata: {name: local-coder, namespace: team-a}
spec:
  image: registry.acme.eu/agents/coder:1.4.0
  tier: tier-1-microvm
  model:
    provider: llamay
    endpoint: http://maqpna-llamay.maqpna-system.svc:8081
    name: qwen2.5-coder-7b-instruct-q4_k_m
    jurisdiction: EU-DE

provider: llamay turns on per-session contexts and context fork. The gateway presents the chart's key itself, so the Agent needs no apiKeySecretRef.

What you see in the audit ledger#

Record Meaning
A model call with ext.llamayContext The llamay context the session ran in. The same id appears in llamay's own audit record.
ext.llamayContextParent A forked session's context was forked from this one.
llamay/audit-anchor llamay's chain verified through llamayAuditSeq, head llamayAuditHead.
llamay/audit-unverified llamay's chain is broken, missing or rewritten. With failClosed, llamay is out of routing (503 llamay_audit_unverified) until it verifies.

Good to know#

  • The agent cannot pick a context. The gateway removes any llamay_session an agent sends and binds the session's own context. Every request reaches llamay with the gateway's key, so this is what keeps sessions apart.
  • A lost or expired context costs only a prefill. The conversation is always sent whole.
  • llamay's audit records counts, never prompts or completions.

The design, security model, failure behaviour and the changes still needed in llamay are in the integration design, docs/integrations/llamay.md in the MAQPNA source.

Next steps#