An AI SRE agent that asks before it changes anything

Build an SRE agent that reads a kind cluster through the gateway's maqpna-k8s tools, proposes a fix, applies it only after a human approves, and checks Terraform plans by blast-radius tier.

In this tutorial you give an AI site reliability engineering (SRE) agent access to a Kubernetes cluster without giving it the cluster. The agent diagnoses a failing workload with read-only tools, asks to contain it, and waits. A person approves. Only then does the change reach the cluster. Every step lands in the audit ledger, and the kill switch stops the agent at once. At the end you check two Terraform plans against the same policy.

Everything runs on your laptop: a kind cluster and a local MAQPNA started with maqpna dev up.

What you'll build#

flowchart LR
  A["sre_agent.py<br/>agent sre"] -->|"tools:maqpna-k8s"| G["Gateway<br/>policy sre-policy"]
  G -->|read: allow| K["maqpna-k8s<br/>typed tools"]
  G -->|write: require approval| H["Approver<br/>oncall@localhost"]
  H -->|approve| K
  K -->|"Impersonate-User<br/>maqpna-k8s-ops"| API["kind API server<br/>RBAC decides"]
  G --> L[("Audit ledger<br/>.maqpna/audit.jsonl")]
  P["plan.json"] -->|"maqpna infra plan check"| T["Tier 0-4<br/>same policy"]


Time	About 40 minutes
Level	Intermediate
You learn	The built-in `maqpna-k8s` server, impersonated read-only RBAC, proposal and apply write modes, approval floors, taint from container output, the kill switch, blast-radius tiers for Terraform plans

Prerequisites#

The maqpna CLI (curl -fsSL https://maqpna.com/install.sh | sh, or see Install).
Docker, kind and kubectl.
Python 3.10 or later, and the Python SDK: pip install kosa8-maqpna.
jq, to read the audit ledger.
Ports 8080 and 8081 free.

You should have run Your first governed agent, so maqpna dev up, maqpna dev run and maqpna approvals are familiar.

Step 1: Create a cluster with a broken workload#

Make a working folder and a kind cluster:

mkdir sre-lab && cd sre-lab
kind create cluster --name sre-lab

Create shop.yaml. Its worker deployment exits at start, so it ends in CrashLoopBackOff:

apiVersion: v1
kind: Namespace
metadata: {name: shop}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: worker, namespace: shop}
spec:
  replicas: 1
  selector: {matchLabels: {app: worker}}
  template:
    metadata: {labels: {app: worker}}
    spec:
      containers:
      - name: worker
        image: busybox:1.36
        command: ["sh", "-c", "echo 'config /etc/worker/config.yaml not found'; exit 1"]

kubectl apply -f shop.yaml

Check it works#

After a minute, kubectl get pods -n shop shows the worker pod in CrashLoopBackOff (or Error).

Step 2: Give the gateway a read-only identity#

The gateway never uses your kubeconfig. It authenticates as one ServiceAccount, maqpna-gateway, and impersonates a second one, maqpna-k8s-ops, on every API call. Only maqpna-k8s-ops has permissions, and they are get and list. A prompt cannot widen them.

Create rbac-read.yaml. The rules are the ones the Helm chart renders in templates/k8sops-rbac.yaml, without the metrics API (kind has no metrics-server):

apiVersion: v1
kind: Namespace
metadata: {name: maqpna-system}
---
apiVersion: v1
kind: ServiceAccount
metadata: {name: maqpna-gateway, namespace: maqpna-system}
---
apiVersion: v1
kind: ServiceAccount
metadata: {name: maqpna-k8s-ops, namespace: maqpna-system}
automountServiceAccountToken: false   # used only by impersonation
---
# The gateway may impersonate this one ServiceAccount, nothing else.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: {name: maqpna-gateway-impersonate-k8s-ops, namespace: maqpna-system}
rules:
- apiGroups: [""]
  resources: ["serviceaccounts"]
  verbs: ["impersonate"]
  resourceNames: ["maqpna-k8s-ops"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: {name: maqpna-gateway-impersonate-k8s-ops, namespace: maqpna-system}
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: Role, name: maqpna-gateway-impersonate-k8s-ops}
subjects:
- {kind: ServiceAccount, name: maqpna-gateway, namespace: maqpna-system}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: {name: maqpna-k8s-ops-read}
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "services", "endpoints", "configmaps", "persistentvolumeclaims", "serviceaccounts", "events"]
  verbs: ["get", "list"]
- apiGroups: ["apps"]
  resources: ["deployments", "statefulsets", "daemonsets", "replicasets"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["jobs", "cronjobs"]
  verbs: ["get", "list"]
---
# Read access in namespace shop only.
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: {name: maqpna-k8s-ops-read, namespace: shop}
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: maqpna-k8s-ops-read}
subjects:
- {kind: ServiceAccount, name: maqpna-k8s-ops, namespace: maqpna-system}

Apply it, then write the three files the gateway needs to reach the cluster: a token for maqpna-gateway, the cluster CA and the API server address.

kubectl apply -f rbac-read.yaml
mkdir -p k8s
kubectl create token maqpna-gateway -n maqpna-system --duration 2h > k8s/gateway.token
kubectl config view --raw --minify -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > k8s/ca.crt
APISERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
echo "$APISERVER"

Check it works#

kubectl auth can-i list pods -n shop --as system:serviceaccount:maqpna-system:maqpna-k8s-ops
kubectl auth can-i patch deployments -n shop --as system:serviceaccount:maqpna-system:maqpna-k8s-ops
kubectl auth can-i list pods -n kube-system --as system:serviceaccount:maqpna-system:maqpna-k8s-ops

The answers are yes, no and no.

Step 3: Write the policy and start a local MAQPNA#

Create policies/sre.json. It is a policy bundle, the format the gateway loads. Rules are checked in order and the first match wins; anything unmatched is denied.

{
  "policies": [
    {
      "name": "sre-policy",
      "namespace": "sre",
      "agents": ["sre"],
      "defaultAction": "deny",
      "rules": [
        {"name": "no-system-writes", "servers": ["maqpna-k8s"],
         "tools": ["scale_workload", "rollout_restart", "delete_pod"],
         "argMatch": {"namespace": "kube-*"}, "action": "deny",
         "reason": "Agents never change system namespaces."},
        {"name": "diagnose", "servers": ["maqpna-k8s"],
         "tools": ["list_resources", "get_resource", "describe_resource", "list_events", "explain_failure", "get_pod_logs"],
         "action": "allow", "maxCallsPerMinute": 120},
        {"name": "remediate-with-approval", "servers": ["maqpna-k8s"],
         "tools": ["scale_workload", "rollout_restart", "delete_pod", "cordon_node", "uncordon_node"],
         "action": "require_approval", "maxCallsPerMinute": 10,
         "reason": "Every cluster change needs an on-call engineer's approval."},
        {"name": "no-database-deletes", "servers": ["maqpna-infra"], "tools": ["infra__apply"],
         "argMatch": {"deleteTypes": "aws_db_*"}, "action": "deny",
         "reason": "Database deletes go through the DBA team."},
        {"name": "infra-tier-2-needs-human", "servers": ["maqpna-infra"], "tools": ["infra__apply"],
         "when": [{"arg": "tier", "op": "gte", "value": "2"}], "action": "require_approval"},
        {"name": "infra-small-changes", "servers": ["maqpna-infra"], "tools": ["infra__apply", "infra__plan"],
         "action": "allow"}
      ]
    }
  ]
}

The last three rules are for Terraform plans in Step 8.

Now the gateway configuration. k8sOps turns on the built-in maqpna-k8s server. namespaces: ["shop"] is the gateway's own scope, checked before the policy. writes.mode: proposal (the default) means a write returns a YAML patch and changes nothing. --gateway-config merges this object over the generated .maqpna/gateway.json; --no-echo skips the test MCP server.

cat > k8s-ops.json <<EOF
{
  "k8sOps": {
    "enabled": true,
    "apiServer": "$APISERVER",
    "tokenFile": "$PWD/k8s/gateway.token",
    "caFile": "$PWD/k8s/ca.crt",
    "serviceAccount": "maqpna-system/maqpna-k8s-ops",
    "namespaces": ["shop"],
    "writes": {"mode": "proposal"}
  }
}
EOF
maqpna dev up --no-echo --policy policies/sre.json --gateway-config k8s-ops.json

Check it works#

maqpna-k8s is not an upstream, so maqpna dev run does not grant its scope by default. Pass it with --scope, and run as agent sre in namespace sre so the policy applies:

maqpna dev run --agent sre --namespace sre --scope tools:maqpna-k8s -- \
  maqpna call --server maqpna-k8s --list

The list has the read tools (list_resources, get_resource, explain_failure and the others) and the five write tools (scale_workload, rollout_restart, cordon_node, uncordon_node, delete_pod). In proposal mode, each write tool's description ends with "Returns the patch for a GitOps commit; nothing is applied."

Step 4: Diagnose the failure#

Set up Python with python3 -m venv .venv && . .venv/bin/activate && pip install kosa8-maqpna.

Create sre_agent.py. A real SRE agent would let a model choose the next tool. This one is scripted, so you see each decision clearly: it explains the failure, then asks to scale worker to zero to stop the crash loop while a person fixes the configuration.

from maqpna import MaqpnaClient, PolicyDenied, wait_for_approval

client = MaqpnaClient.from_env()
k8s = client.tools("maqpna-k8s")

# 1. Diagnose. Read tools are allowed by rule "diagnose".
diag = k8s.call("explain_failure", kind="deployment", namespace="shop", name="worker")
print(diag.text)

# 2. Contain. A write is held for a human by rule "remediate-with-approval".
try:
    res = wait_for_approval(
        k8s, "scale_workload",
        {"kind": "deployment", "namespace": "shop", "name": "worker", "replicas": 0},
        poll_interval=2, timeout=300,
        on_pending=lambda p: print(f"waiting for approval {p.approval_id}", flush=True),
    )
    print("applied:", (res.structured or {}).get("applied"))
    print(res.text)
except PolicyDenied as e:
    print("denied:", e.reason, e.data.get("detail"))

Run it:

maqpna dev run --agent sre --namespace sre --scope tools:maqpna-k8s -- python sre_agent.py

explain_failure reads the deployment, its pods' container states, warning events and the failing container's last log lines. The output names deployment/shop/worker, the CrashLoopBackOff state and the log line config /etc/worker/config.yaml not found. Then the agent prints waiting for approval apr_... and waits. Leave it waiting.

Check it works#

The log line came from the container, so the gateway treats it as untrusted input and taints the session. In a second terminal in sre-lab:

eval "$(maqpna dev env)"
maqpna taint list -n sre

The session is listed with the label untrusted-input. A rule can now deny or hold tools that could leak data from this session (see Taint and prompt-injection containment).

Step 5: Approve a proposal#

Still in the second terminal:

maqpna approvals list
maqpna approvals approve <approval-id> --approver oncall@localhost --note "contain crash loop"

Approvers cannot approve calls made on their own behalf. The agent acts for you@localhost, so the approver is oncall@localhost.

Back in the first terminal, the agent prints applied: False, then Proposed change (NOT applied; commit it to the GitOps repository that manages deployment/shop/worker): and the strategic-merge patch with replicas: 0.

Check it works#

Run kubectl get deployment worker -n shop: worker still wants one replica. In proposal mode, the agent's change is a patch for your normal GitOps review, never a cluster write. The ledger holds an observe record with reason k8s_write_proposed: deployment/shop/worker, the patch and its SHA-256.

Step 6: Let approved writes apply#

To let an approved change reach the cluster, two things must change, and both are yours, not the agent's: RBAC and the gateway's write mode.

Create rbac-write.yaml. It is the chart's opt-in write role (writes.rbac: true), bound in shop only:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: {name: maqpna-k8s-ops-write}
rules:
- apiGroups: ["apps"]
  resources: ["deployments", "statefulsets", "daemonsets", "deployments/scale", "statefulsets/scale"]
  verbs: ["patch"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: {name: maqpna-k8s-ops-write, namespace: shop}
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: maqpna-k8s-ops-write}
subjects:
- {kind: ServiceAccount, name: maqpna-k8s-ops, namespace: maqpna-system}

Apply it, switch the write mode to apply and restart the local MAQPNA. The state and the ledger stay in .maqpna/.

kubectl apply -f rbac-write.yaml
sed -i.bak 's/"mode": "proposal"/"mode": "apply"/' k8s-ops.json
maqpna dev down
maqpna dev up --no-echo --policy policies/sre.json --gateway-config k8s-ops.json
maqpna dev run --agent sre --namespace sre --scope tools:maqpna-k8s -- python sre_agent.py

Approve the new approval from the second terminal as in Step 5.

Check it works#

The agent prints applied: True and a result that starts Applied scale_workload to deployment/shop/worker. kubectl get deployment worker -n shop now shows 0/0 replicas. The ledger has an observe record k8s_write_applying: deployment/shop/worker, written before the request to the API server, and k8s_write_applied: deployment/shop/worker after it.

Step 7: Watch the guardrails hold#

Ask for something outside the gateway's scope:

maqpna dev run --agent sre --namespace sre --scope tools:maqpna-k8s -- \
  maqpna call --server maqpna-k8s --tool explain_failure \
  --arg kind=deployment --arg namespace=kube-system --arg name=coredns
echo "exit status $?"

The call is denied with JSON-RPC error -32001 by policy maqpna-k8s, rule namespace, with the detail k8s_namespace_denied: namespace kube-system is outside the gateway's k8sOps scope. The exit status is 3. The gateway refused it before the policy ran and before any API request. Even without that check, the impersonated ServiceAccount has no RBAC in kube-system.

Then use the kill switch. Revoke every session of agent sre:

maqpna kill -n sre --agent sre --reason "INC-7: test the SRE kill switch" --ttl 1h --yes
maqpna dev run --agent sre --namespace sre --scope tools:maqpna-k8s -- \
  maqpna call --server maqpna-k8s --tool list_resources --arg kind=pod --arg namespace=shop

The second command is denied with reason revoked. Lift it with maqpna revocations list and maqpna revocations delete <revocation-id> --yes.

Check it works#

jq -c 'select(.agent=="sre") | {seq, tool, decision, rule, reason, resource: .ext.k8sResource, approver}' .maqpna/audit.jsonl
maqpna audit verify .maqpna/audit.jsonl

You see every read with decision allow, the held writes and their approvals by oncall@localhost, the observe records with the patch, the kube-system denial and the revoked call. maqpna audit verify prints OK and the record count. maqpna audit tail --agent sre --since 0 shows the same records from the gateway's admin API.

Step 8: Check Terraform plans with the same policy#

Infrastructure changes through Terraform or OpenTofu follow the same idea. The agent proposes a saved plan. The gateway reads it as a call of tool infra__apply on the built-in server maqpna-infra, places it in a blast-radius tier from 0 (no change) to 4 (deletes of data-bearing resources, or more than 50 changes), and the policy decides. You can run the same evaluation offline, with no cloud account.

Create plan-db-delete.json. It is the JSON shape of terraform show -json tfplan, trimmed to two resource changes:

{
  "format_version": "1.2",
  "terraform_version": "1.10.0",
  "resource_changes": [
    {"address": "aws_db_instance.orders", "mode": "managed", "type": "aws_db_instance", "name": "orders",
     "provider_name": "registry.terraform.io/hashicorp/aws",
     "change": {"actions": ["delete"], "before": {"identifier": "orders"}, "after": null,
                "after_unknown": {}, "before_sensitive": {}, "after_sensitive": false}},
    {"address": "aws_s3_bucket.tfstate", "mode": "managed", "type": "aws_s3_bucket", "name": "tfstate",
     "provider_name": "registry.terraform.io/hashicorp/aws",
     "change": {"actions": ["update"], "before": {"versioning": true}, "after": {"versioning": false},
                "after_unknown": {}, "before_sensitive": {}, "after_sensitive": {}}}
  ],
  "complete": true,
  "applyable": true,
  "errored": false
}

maqpna infra plan check plan-db-delete.json --values --policy policies/sre.json --namespace sre --agent sre
echo "exit status $?"

The output lists each change with the attributes it touches, then the summary and the decision:

Tier 4 plan: 1 to change, 1 to destroy. Tier driven by data-bearing-delete: aws_db_instance.orders.

The tier reasons follow (data-bearing-delete, delete, protected-address for the state bucket, sensitive-type, update). The last line says the policy decision is deny by policy sre-policy, rule no-database-deletes. The exit status is 3, so a CI job can fail on it.

Drop the database change and check again:

jq 'del(.resource_changes[0])' plan-db-delete.json > plan-bucket.json
maqpna infra plan check plan-bucket.json --policy policies/sre.json --namespace sre --agent sre

The bucket name contains tfstate, a protected address, so the plan is tier 3 and rule infra-tier-2-needs-human returns require_approval. The exit status is 0; add --fail-on approval to fail on it too.

Check it works#

You saw one plan denied and one held for approval, from the same policy file the gateway enforces. To apply plans under governance, the gateway needs "infra": {"enabled": true} in its configuration and the session the scope tools:maqpna-infra. Then maqpna infra apply -- terraform apply tfplan asks the gateway first. It applies only the exact plan that was approved, bound to its SHA-256, and stops the apply when the session is revoked.

What you built#

A Kubernetes identity for the gateway that can only impersonate one read-only ServiceAccount, scoped to namespace shop.
The gateway's maqpna-k8s server with typed tools: no kubectl, no free-form API paths.
A policy that allows diagnosis, holds every change for a person, and denies system namespaces.
Two write modes: proposals that return a GitOps patch, and approved writes that reach the cluster. Write RBAC is opt-in, and the approval floor holds even if a policy says allow.
Taint from container output, a kill switch that stops the agent within about a second, and a verified audit ledger with every resource and patch.
Blast-radius tiers and policy decisions for Terraform plans, offline.

Clean up#

maqpna dev down
kind delete cluster --name sre-lab
rm -f k8s/gateway.token

maqpna dev down keeps .maqpna/ and its ledger. Delete the sre-lab folder when you no longer need them.

Next steps#

Run it in a cluster with Helm values gateway.k8sOps: Governed cluster operations.
Route approvals to on-call people, with quorum for big changes: Human approvals.
Quarantine an agent during an incident: Kill switch and revocations.
Test this policy in CI before it ships: Policy as code in CI.
Export the session for a review: Evidence packs.