Runbook: kill switch

Stop an agent, session, user or token across the installation now with maqpna kill, make the revocation durable, confirm it is enforced on every replica and lift it afterwards.

Use this runbook to stop an agent, a session, a user's sessions or a token now: suspected misuse, a runaway loop, a leaked token, or a policy gap that is being exploited. A kill creates a revocation at the gateway. The next call from the revoked principal is denied with reason revoked (JSON-RPC error -32001), within about a second.

The task-by-task guide with example output is Kill switch and revocations. This page is the on-call version.

Symptoms#

• An agent calls tools in a loop, or spends faster than expected (MaqpnaBudgetExceeded, cost spikes in maqpna top agents).
• An agent does something harmful or unexpected, or you suspect a compromise.
• A session token or a user's account is known to be leaked.

Impact#

While the incident runs, the agent can make every call its policy allows. After the kill, the matching principal can make no tool or model call. Other agents are not affected unless you widen the scope (--all).

Detection#

GW is the gateway URL (with adminListen, the admin port). You need the killswitch or admin role: sign in with maqpna login, or pass --oidc-token-file (the break-glass --token also works where it is enabled).

export GW=https://gateway-admin.example.internal NS=team-a AGENT=coder INC=INC-123

Decide#

Pick the narrowest scope that stops the harm:

And the action:

flowchart TD
  A[Stop it now] --> B{"Gateway reachable?"}
  B -->|yes| C["maqpna kill --gateway<br/>(effective in about 1 s)"]
  B -->|no| D["maqpna kill --emit-yaml<br/>| kubectl apply"]
  C --> E{"More than one gateway replica,<br/>or need a durable record?"}
  E -->|yes| D
  E -->|no| F[Verify]
  D --> F
  F --> G{"Compromise suspected?"}
  G -->|yes| H["Compromised-agent runbook"]
  G -->|no| I["Fix cause, then lift"]

Respond#

Step 1: Revoke at the gateway#

maqpna kill --gateway "$GW" -n "$NS" --agent "$AGENT" --reason "$INC data exfiltration" --terminate --yes

The reason is required; it is written to the audit ledger and the session events. Add --ttl 2h for a revocation that expires by itself. The replica that answers enforces it on the next call.

Step 2: Make it durable and cluster-wide#

An AgentRevocation reaches every replica through configSync and survives restarts and database restores. It is required with several replicas on the file state backend, where each replica has its own break-glass list. With state.backend: postgres, step 1 already reaches every replica; apply it anyway as the record that survives a restore.

maqpna kill -n "$NS" --agent "$AGENT" --reason "$INC" --terminate --emit-yaml | kubectl apply -f -

--emit-yaml creates nothing at the gateway; it only prints the manifest. Use it alone when the gateway is unreachable (for example during a PostgreSQL outage): it goes through Kubernetes.

Step 3: Revoke delegated access (user incidents)#

--terminate with --user revokes the user's connected accounts. To revoke one account explicitly:

maqpna accounts list --gateway "$GW" --user <user>
maqpna accounts revoke <user> <provider> --gateway "$GW" --yes

Step 4: Hand over#

If the cause may be a compromise, continue with the compromised agent runbook before you lift anything. Pending approvals of the revoked agent are moot; approvers can deny them:

maqpna approvals list --gateway "$GW" -n "$NS"

Step 5: Lift the revocation#

When the cause is fixed, lift the revocation from both places:

maqpna revocations list --gateway "$GW"
maqpna revocations delete --gateway "$GW" breakglass/<id> --yes
kubectl -n maqpna-system delete agentrevocation <name>

maqpna kill --gateway "$GW" -n "$NS" --agent "$AGENT" --release --yes lifts the break-glass revocations an earlier kill of the same target created (and their mirrored AgentRevocations); for a quarantine add --quarantine. Suspended sessions stay suspended until you resume them with maqpna session resume.

Verify#

maqpna revocations list --gateway "$GW"                                  # the entry is listed, with its expiry
maqpna audit tail --gateway "$GW" -n "$NS" --agent "$AGENT" --decision deny --since 10m

• New calls from the principal are denied with reason revoked.
• The audit ledger has a revocation.create record with the reason and who made it, and a revocation.delete record when you lift it.
• MaqpnaRevocationLag is not firing: every replica has the current list. With Prometheus, maqpna_gateway_revocation_propagation_max_seconds shows how long enforcement took per replica.

Troubleshooting#

Follow-up#

• Record the scope, the reason, the time of the kill and the time of the lift in the incident record; the audit ledger has the same facts with who made them.
• If the cause was a policy gap, add a rule and a policy test.
• Rehearse the kill switch regularly with a test session and --ttl 5m (Production checklist).

Next steps#

• Kill switch and revocations
• Kill switch flow
• Runbook: compromised agent
• Command reference: maqpna kill, maqpna revocations list