Day-to-day development with the stack

Lay out an agent project for MAQPNA, llamay and kosa8, configure each product, call llamay models and tools from the SDKs, test locally with policy tests and forks, run CI, and promote to a cluster.

This page is the daily loop for a developer who builds agents on the Kosa8 Labs stack. You write the agent once. On your laptop it runs in a kosa8 microVM and calls a llamay model through a local MAQPNA gateway. In a cluster the same code and image run in a MAQPNA sandbox and call the in-cluster llamay through the same gateway route.

Read the quickstart first if the three products are not installed yet. The general MAQPNA developer guide starts at Developer overview. This page covers what changes when llamay and kosa8 are part of the loop.

flowchart LR
    E["Edit agent<br/>and policies"] --> P["maqpna policy test"]
    P --> R["maqpna dev run<br/>--kosa8-image"]
    R --> T["maqpna dev timeline"]
    T --> F["maqpna dev fork /<br/>snapshot / restore"]
    F --> E
    T --> C["CI: validate, policy<br/>tests, agent tests"]
    C --> K["Build image,<br/>maqpna apply to a cluster"]

Project layout#

Use the layout from Anatomy of an agent project, plus three stack-specific files:

coder-agent/
  agent.py                  # or agent.mjs, main.go
  requirements.txt          # kosa8-maqpna, openai
  Dockerfile                # one image for kosa8 on the laptop and the cluster
  manifests/
    agent.yaml              # Agent with model.provider: llamay
  policies/
    coder-policy.yaml       # ToolPolicy for the cluster
  policy-tests/
    maqpna-test.yaml        # PolicyTest suite
  local/
    policies.json           # policy bundle for maqpna dev up --policy
    gateway.json            # extra gateway settings for maqpna dev up --gateway-config
    dev-up.sh               # the one dev up command your team uses
  tests/
    test_agent.py
  .gitignore                # include .maqpna/

local/dev-up.sh keeps everyone on the same local setup:

#!/bin/sh
# Start the local stack: llamay model, kosa8 sandboxes, the team's policy and gateway settings.
exec maqpna dev up --llamay qwen2.5:0.5b --kosa8 \
  --policy local/policies.json --gateway-config local/gateway.json

Configuration for each product#

A local/gateway.json that tunes the llamay integration:

{
  "llamay": {
    "contextIdleSeconds": 600,
    "maxContexts": 256
  }
}

dev up still adds its own audit anchor and checkpointDir to the llamay block when you leave them out. Every key of the llamay block is listed in the llamay reference.

Call a llamay model and a tool from code#

The agent does not know it talks to llamay. It uses the gateway's OpenAI-compatible model route, with the session token as the API key. maqpna dev run sets MAQPNA_MODEL_ENDPOINT (and OPENAI_BASE_URL) to {gateway}/llm/llamay/v1. In a cluster the operator sets the same variable to the agent's own route.

Two rules matter on this stack:

• Read the token on every request. After maqpna dev fork or dev restore, MAQPNA replaces the token file with the new session's token. An agent that reads the token once keeps the parent's identity. The SDKs re-read MAQPNA_TOKEN_FILE on every call. For the OpenAI client, pass fresh headers per request.
• Never send llamay_session. The gateway strips it and binds the session's own llamay context. You get context reuse and context fork without doing anything.

# pip install kosa8-maqpna openai
from openai import OpenAI
from maqpna import MaqpnaClient, PolicyDenied

client = MaqpnaClient.from_env()
llm = OpenAI(base_url=client.model_endpoint(), api_key=client.token())

def ask(prompt: str) -> str:
    reply = llm.chat.completions.create(
        model=client.model_name() or "default",   # the gateway forces the route's model
        messages=[{"role": "user", "content": prompt}],
        max_tokens=256,
        extra_headers=client.openai_headers(),    # current token, per request
    )
    return reply.choices[0].message.content

print(ask("Name one risk of running untested migrations."))

echo = client.tools("echo")
try:
    print(echo.call("echo", text="hello").text)
except PolicyDenied as err:
    print("denied:", err.policy, err.rule, err.reason)

// npm install @kosa8/maqpna openai
import OpenAI from "openai";
import { MaqpnaClient, PolicyDenied } from "@kosa8/maqpna";

const client = MaqpnaClient.fromEnv();
const llm = new OpenAI({ baseURL: client.modelEndpoint(), apiKey: await client.token() });

async function ask(prompt: string): Promise<string | null> {
  const reply = await llm.chat.completions.create(
    { model: client.modelName() ?? "default", messages: [{ role: "user", content: prompt }], max_tokens: 256 },
    { headers: await client.openaiHeaders() }, // current token, per request
  );
  return reply.choices[0].message.content;
}

console.log(await ask("Name one risk of running untested migrations."));

try {
  const res = await client.tools("echo").call("echo", { text: "hello" });
  console.log(res.text);
} catch (err) {
  if (err instanceof PolicyDenied) console.log("denied:", err.policy, err.rule, err.reason);
  else throw err;
}

// The Go SDK is not published yet; see /sdk/go/ for the replace directive.
package main

import (
    "context"
    "errors"
    "fmt"
    "io"
    "log"
    "net/http"
    "strings"

    maqpna "github.com/maqpna/maqpna/sdk/go"
)

func main() {
    ctx := context.Background()
    client, err := maqpna.NewFromEnv(ctx)
    if err != nil {
        log.Fatal(err)
    }

    // Model call: OpenAI wire format to the gateway's route, current token per request.
    hdr, err := client.OpenAIHeaders(ctx)
    if err != nil {
        log.Fatal(err)
    }
    body := `{"model":"default","max_tokens":256,"messages":[{"role":"user","content":"Name one risk of running untested migrations."}]}`
    req, _ := http.NewRequestWithContext(ctx, http.MethodPost,
        client.ModelBaseURL("")+"/chat/completions", strings.NewReader(body))
    req.Header = hdr
    req.Header.Set("Content-Type", "application/json")
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        log.Fatal(err)
    }
    out, _ := io.ReadAll(resp.Body)
    resp.Body.Close()
    fmt.Println(string(out))

    // Tool call.
    res, err := client.CallTool(ctx, "echo", "echo", map[string]any{"text": "hello"})
    var denied *maqpna.PolicyDenied
    switch {
    case errors.As(err, &denied):
        fmt.Println("denied:", denied.Policy, denied.Rule, denied.Reason)
    case err != nil:
        log.Fatal(err)
    default:
        fmt.Println(res.Text())
    }
}

How to react to each denial, budget and approval in code is in Handle decisions.

Build an image that runs in kosa8#

A kosa8 sandbox has no internet access: its only destination is the gateway. Everything the agent needs must be in the image, and the image needs /bin/sh, because MAQPNA starts the agent through a small shell that writes the token file.

FROM python:3.12-alpine
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY agent.py .
CMD ["python", "/app/agent.py"]

The kosa8 host pulls the image, so push it to a registry the host can reach. Pin the digest so every run uses the same bytes:

maqpna dev run --kosa8-image registry.example.eu/agents/coder@sha256:<digest> -- python /app/agent.py

maqpna dev run passes the command after -- to the container, so it replaces the image's CMD.

The local test loop#

1. Test the policy#

Policies are code. Test the cluster ToolPolicy and the local bundle with the same command:

maqpna policy test --policy policies/ --ns team-a --agent coder --server echo --tool echo --expect allow
maqpna policy test --policy local/policies.json --ns dev --agent coder --server echo --tool echo --expect allow
maqpna policy test policy-tests/ --require-tests

A mismatch exits with code 3. Suites and --explain are in Test agents and policies.

2. Run the agent and its tests under the local gateway#

./local/dev-up.sh
maqpna dev run -- python -m pytest -q tests/                          # fast: a local process
maqpna dev run --kosa8-image registry.example.eu/agents/coder@sha256:<digest> \
  -- python /app/agent.py                                             # real boundary: a microVM
maqpna dev timeline --last

Run unit tests as a local process for speed. Run the agent in kosa8 before you push, because that is where missing files, a missing /bin/sh or hidden internet calls show up.

3. Fork to compare two continuations#

A fork copies a running agent: the microVM's memory and disks, and on llamay the session's inference context. Use it to try two continuations from exactly the same state.

# Terminal 1: keep the sandbox, and give the agent a pause point (here: it sleeps 30 s mid-task)
maqpna dev run --kosa8-image registry.example.eu/agents/coder@sha256:<digest> --kosa8-keep \
  --session base -- python /app/agent.py

# Terminal 2, while it runs
maqpna dev fork --session base --name try-b --attach=false
maqpna dev timeline --session base -o json > base.json
maqpna dev timeline --session try-b-1 -o json > try-b.json

The child session is try-b-1. It has its own token, and the parent's namespace, agent, user, scopes, taint and budget. Its first llamay call runs in a fork of the parent's context. Diff the two timelines to compare what each run did.

4. Snapshot before a risky step, restore to retry#

maqpna dev snapshot --session base --id before-migration
maqpna dev restore --checkpoint before-migration --name retry --attach=false
maqpna dev timeline --session retry-1

dev snapshot saves the session's llamay contexts first, then the sandbox, so the model state is never ahead of the agent. dev restore boots the snapshot as a new session with the checkpointed session as its parent, and resumes the saved contexts. A restore of a revoked session is refused. More in kosa8 with MAQPNA and the fork and explore tutorial.

5. Check the evidence#

maqpna audit verify .maqpna/audit.jsonl
maqpna llamay verify-audit .maqpna/llamay-audit.jsonl
maqpna dev attest --session base -o json     # kernel, rootfs and image digests; exit 3 if egress is not the gateway only

CI with GitHub Actions#

This workflow uses only published install scripts and real commands. It runs the offline checks, then the agent tests against a local MAQPNA with a real llamay model. It needs no cluster and no secrets.

# .github/workflows/agent.yml
name: agent
on: [pull_request]

jobs:
  policies:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install the maqpna CLI
        run: |
          curl -fsSL https://maqpna.com/install.sh | sh -s -- --version v2.1.0 --bin maqpna --dir "$HOME/.local/bin"
          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
      - run: maqpna validate -f manifests/ -f policies/ --strict
      - run: maqpna policy lint policies/ --strict
      - run: maqpna policy test policy-tests/ --require-tests -o junit > policy-tests.xml

  agent-with-llamay:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install maqpna and llamay
        run: |
          curl -fsSL https://maqpna.com/install.sh | sh -s -- --version v2.1.0 --bin maqpna --dir "$HOME/.local/bin"
          curl -fsSL https://llamay.com/install.sh | LLAMAY_INSTALL_DIR="$HOME/.local/bin" sh
          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
      - run: pip install -r requirements.txt pytest
      - run: llamay pull qwen2.5:0.5b
      - name: Start a local MAQPNA with llamay
        run: maqpna dev up --llamay qwen2.5:0.5b --policy local/policies.json --gateway-config local/gateway.json
      - name: Agent tests through the gateway
        run: maqpna dev run -- python -m pytest -q tests/
      - name: Verify the ledger and llamay's audit
        run: |
          maqpna audit verify .maqpna/audit.jsonl
          maqpna llamay verify-audit .maqpna/llamay-audit.jsonl
      - if: always()
        run: maqpna dev down

The model download takes a minute or two on each run. A small model keeps the job fast; test the behaviour of your production model separately.

Package and promote to a cluster#

The same image runs in the cluster. Only the manifest says where the model is:

# manifests/agent.yaml
apiVersion: maqpna.com/v1alpha1
kind: Agent
metadata: {name: coder, namespace: team-a}
spec:
  image: registry.example.eu/agents/coder@sha256:<digest>
  tier: tier-1
  model:
    provider: llamay
    endpoint: http://maqpna-llamay.maqpna-system.svc:8081
    name: qwen2.5-coder-7b-instruct-q4_k_m
    jurisdiction: EU-DE
    maxTokensPerRequest: 4096
  tools:
  - name: echo        # the chart's example MCP server (examples.mcpEcho); use serverRef for your own

maqpna validate -f manifests/ -f policies/ --strict
maqpna apply -f policies/ -n team-a
maqpna apply -f manifests/ -R -n team-a
maqpna rollout status ag/coder -n team-a
maqpna session start --agent coder -n team-a --task "Review the open migration" --wait

Apply the policy before the agent, so no session runs without one. What changes between laptop and cluster:

The cluster side, step by step, is in Run the stack on Kubernetes. Promotion between environments and GitOps are in Package and deploy.

Guidelines#

Do:

• Read the token from MAQPNA_TOKEN_FILE on every call, or use the SDKs. It is what lets a fork or a restore switch to its own identity.
• Send the whole conversation every turn. The gateway's llamay context is an accelerator. A lost context costs one prefill, never correctness.
• Pin images by digest in --kosa8-image and in Agent.spec.image. A changed image changes the digests dev attest reports.
• Keep one policy truth. Test both policies/ and local/policies.json in CI, so the laptop and the cluster cannot drift apart.
• Run in kosa8 before you push. It is the closest thing to the cluster's default-deny sandbox on a laptop.
• Verify both chains (maqpna audit verify, maqpna llamay verify-audit) in CI. They are cheap.
• Use --session and --name with meaningful ids when you fork or restore, so the timelines are easy to compare.

Don't:

• Don't call llamay directly (127.0.0.1:11435) from the agent. Those calls are not governed, not metered and not in the ledger. In a kosa8 sandbox and in the cluster they are blocked anyway.
• Don't send llamay_session or try to pick a context. The gateway strips it.
• Don't put the token in an environment variable you export once, such as OPENAI_API_KEY at start. A fork would keep the parent's identity until the process re-reads it.
• Don't commit .maqpna/. It holds keys, the admin token and the ledger.
• Don't expose the dev gateway on an untrusted network. dev up --kosa8 listens on your LAN address; firewall port 8080.
• Don't rely on Anthropic /v1/messages or Ollama routes of llamay. The gateway governs llamay through the OpenAI routes only.

Next steps#

• Run the stack on Kubernetes
• llamay with MAQPNA
• kosa8 with MAQPNA
• Test agents and policies
• Stack troubleshooting