Test agents and policies

Check manifests and policies offline, test agent code against a local gateway, and run the same checks in a GitHub Actions pipeline.

Test two things separately. Policies and manifests are data: check them offline, with no cluster and no gateway, in seconds. Agent code is behaviour: run it against a real local gateway, so a denial in a test is the same denial it gets in production. This page shows the commands for both and a CI pipeline that runs them on every pull request.

flowchart LR
    subgraph Offline["Offline, no cluster"]
        V["maqpna validate<br/>schemas, CEL, references"]
        LI["maqpna policy lint"]
        PT["maqpna policy test<br/>PolicyTest suites"]
        EV["maqpna eval<br/>multi-step tasks"]
        RP["maqpna replay<br/>against the ledger"]
    end
    subgraph Local["Against a local gateway"]
        UP["maqpna dev up"] --> UT["maqpna dev run --<br/>pytest / npm test / go test"]
    end
    Offline --> CI["CI: exit 3 fails the job"]
    Local --> CI

Every check exits 0 when it passes and 3 when a check fails, so CI needs no output parsing.

The offline checks#

Step 1: Test one call#

Ask "would this agent be allowed to call this tool?" with --explain to see every rule's verdict:

maqpna policy test --policy policies/triage-policy.yaml --ns team-a --agent triage \
  --server github --tool add_issue_comment --explain

{
  "decision": {
    "action": "require_approval",
    "policy": "triage-policy",
    "rule": "comment-needs-approval",
    "reason": "Public comments need a human."
  },
  "trace": {
    ...
        "rules": [
          { "rule": "github-read", "matched": false, "why": "tool \"add_issue_comment\" not in tools" },
          { "rule": "comment-needs-approval", "matched": true }
        ]
}

Add --expect ACTION to turn it into an assertion. A mismatch prints expected deny, got allow and exits 3. Other inputs you can set: --args JSON, --user, --group, --scope, --taint, --sink, --at RFC3339 (for schedules).

Step 2: Write a test suite#

A suite is a file named maqpna-test.yaml. maqpna policy test DIR runs every suite under DIR.

# policy-tests/maqpna-test.yaml
apiVersion: maqpna.com/v1alpha1
kind: PolicyTest
name: triage
policies: [../policies/]
defaults: {namespace: team-a, agent: triage, user: [email protected]}
cases:
  - name: reads are allowed
    server: github
    tool: get_issue
    args: {number: 4821}
    expect: allow
    expectRule: github-read
  - name: comments wait for a human
    server: github
    tool: add_issue_comment
    expect: require_approval
  - name: anything else is denied
    server: github
    tool: delete_repository
    expect: deny

maqpna policy test policy-tests/

SUITE   CASE                       EXPECT               GOT               RULE                                  RESULT
triage  reads are allowed          allow (github-read)  allow             triage-policy/github-read             PASS
triage  comments wait for a human  require_approval     require_approval  triage-policy/comment-needs-approval  PASS
triage  anything else is denied    deny                 deny              triage-policy                         PASS

3 passed, 0 failed, 0 suite error(s)

--require-tests also fails when a policy file is not covered by any suite, and -o junit writes a JUnit report for CI. For multi-step tasks where taint matters (read an untrusted page, then try to send mail), use maqpna eval --suite suite.yaml --policy policies/; examples/eval/suite.yaml in the source tree is a worked example.

Step 3: Test the agent code against a local gateway#

Run your normal test runner under maqpna dev run. Every call the tests make goes through the local gateway, with the local policy, DLP and approvals.

# tests/test_governance.py
import pytest
from maqpna import MaqpnaClient, PolicyDenied, ApprovalPending


@pytest.fixture(scope="module")
def echo():
    return MaqpnaClient.from_env().tools("echo")


def test_read_is_allowed(echo):
    assert echo.call("echo", text="hello").text == "hello"


def test_system_namespace_is_denied(echo):
    with pytest.raises(PolicyDenied) as e:
        echo.call("delete_resource", id="db-1", namespace="kube-system")
    assert e.value.rule == "never-touch-system-namespaces"
    assert e.value.reason == "policy_denied"


def test_delete_needs_approval(echo):
    with pytest.raises(ApprovalPending) as p:
        echo.call_tool("delete_resource", {"id": "db-1", "namespace": "dev"}, async_mode=True)
    assert p.value.approval_id.startswith("apr_")

maqpna dev up
maqpna dev run -- python -m pytest -q tests/

maqpna dev: session dev-3c7a14ec (agent coder, namespace dev, on behalf of you@localhost)
...                                                                      [100%]
3 passed in 0.33s
maqpna dev: dev-3c7a14ec: 1 allow, 1 deny, cost $0.0000 (maqpna dev timeline --last)

// test/governance.test.mjs (node --test)
import { test } from "node:test";
import assert from "node:assert/strict";
import { MaqpnaClient, PolicyDenied } from "@kosa8/maqpna";

const echo = MaqpnaClient.fromEnv().tools("echo");

test("system namespace is denied", async () => {
  await assert.rejects(
    echo.call("delete_resource", { id: "db-1", namespace: "kube-system" }),
    (e) => e instanceof PolicyDenied && e.reason === "policy_denied",
  );
});

maqpna dev run -- node --test test/

func TestSystemNamespaceDenied(t *testing.T) {
    ctx := context.Background()
    client, err := maqpna.NewFromEnv(ctx)
    if err != nil {
        t.Skip("not under maqpna dev run:", err)
    }
    _, err = client.CallTool(ctx, "echo", "delete_resource",
        map[string]any{"id": "db-1", "namespace": "kube-system"})
    var denied *maqpna.PolicyDenied
    if !errors.As(err, &denied) || denied.Reason != "policy_denied" {
        t.Fatalf("want policy_denied, got %v", err)
    }
}

maqpna dev run -- go test ./...

Tips:

• Use --agent, --namespace, --user and --scope on maqpna dev run to test as the identity your policy targets.
• Load your own rules with maqpna dev up --policy local/policies.json (bundle JSON; see Anatomy of an agent project).
• Point tests at a fake MCP server with --upstream NAME=URL, and at the scripted model with --stub-llm (--stub-llm-script FILE changes its answers).
• For fast unit tests with no gateway at all, the SDK test suites show how to inject a fake transport: sdk/python/tests/, sdk/typescript/test/ (a custom fetch) and sdk/go (an httptest server with WithHTTPClient).

Step 4: Check a change against real traffic#

Before you tighten a policy that is in use, replay the audit ledger against the candidate. It prints only the decisions that would change, and exits 3 if any do:

maqpna replay --ledger .maqpna/audit.jsonl --policy local/policies.json --namespace dev

The ledger stores a hash of the arguments, not the arguments, so calls with arguments are replayed without them and flagged argsMissing; their changes count as unknown, not changed. See Write policies.

Step 5: Run it in CI#

This GitHub Actions workflow runs the offline checks and the agent tests against a local MAQPNA. It needs no cluster and no secrets.

# .github/workflows/maqpna.yml
name: maqpna
on: [pull_request]

jobs:
  policies:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install the maqpna CLI
        run: |
          curl -fsSL https://maqpna.com/install.sh | sh -s -- --version v2.1.0 --bin maqpna --dir "$HOME/.local/bin"
          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
      - name: Validate manifests
        run: maqpna validate -f manifests/ -f policies/ --strict -o junit > validate.xml
      - name: Lint policies
        run: maqpna policy lint policies/ --strict
      - name: Policy test suites
        run: maqpna policy test policy-tests/ --require-tests -o junit > policy-tests.xml
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: maqpna-reports
          path: "*.xml"

  agent:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install the maqpna CLI
        run: |
          curl -fsSL https://maqpna.com/install.sh | sh -s -- --version v2.1.0 --bin maqpna --dir "$HOME/.local/bin"
          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
      - name: Install the agent and the SDK
        run: pip install -r requirements.txt pytest   # requirements.txt lists kosa8-maqpna
      - name: Start a local MAQPNA
        run: maqpna dev up --stub-llm --policy local/policies.json
      - name: Agent tests through the gateway
        run: maqpna dev run --agent triage --namespace team-a -- python -m pytest -q tests/
      - name: Verify the audit ledger
        if: always()
        run: maqpna audit verify .maqpna/audit.jsonl
      - name: Show the gateway logs on failure
        if: failure()
        run: tail -n 100 .maqpna/logs/*
      - name: Stop
        if: always()
        run: maqpna dev down

maqpna dev up downloads maqpna-gateway and maqpna-identity of the CLI's version on the first run and checks them against the release checksums. A step fails on exit status 3 (a failed check) as well as 1 (an error).

What can go wrong#

Next steps#

• Policy as code in CI, a complete tutorial
• Write policies
• Package and deploy
• Debug and observe