Runbook: PostgreSQL outage

What happens when the PostgreSQL state backend is down, how to keep the kill switch working meanwhile, and how to recover and verify the shared state afterwards.

Use this runbook when the PostgreSQL state backend is unreachable. It applies to installations with state.backend: postgres, where every gateway replica shares one schema for approvals, revocations, session taint, the token vault, memory, the shared audit chain and rate-limit counters. With state.backend: file there is no database to lose.

Background: State backend and Scaling and high availability.

Symptoms#

• MaqpnaGatewayNotReady fires. GET /readyz on a gateway pod says state backend unreachable: ..., and the pod leaves the Service.
• MaqpnaAuditLedgerUnavailable fires: maqpna_gateway_audit_errors_total rising.
• Approvals cannot be created or decided; break-glass changes (maqpna kill --gateway) return HTTP 500.
• Attestation is refused with reason store (renewals return 503).
• maqpna_gateway_leader drops to 0 on every replica: SIEM and WORM shipping pause. They resume from their cursors.

Impact#

Decide between open and closed before an incident: closed keeps evidence over availability.

Detection#

GW_POD=$(kubectl -n maqpna-system get pods -l app.kubernetes.io/name=maqpna-gateway -o jsonpath='{.items[0].metadata.name}')
kubectl -n maqpna-system port-forward "$GW_POD" 8080:8080 &
curl -s localhost:8080/readyz                       # lists the problems
kubectl -n maqpna-system logs "$GW_POD" --since 10m | grep -i 'state backend'

Alerts: MaqpnaGatewayNotReady, MaqpnaAuditLedgerUnavailable, and your database's own alerts.

Decide#

flowchart TD
  A["readyz: state backend unreachable"] --> B{"Database alive?"}
  B -->|"network or credentials"| N["Fix DSN Secret,<br/>NetworkPolicy, TLS"]
  B -->|"primary down"| F["Fail over<br/>(Patroni / CloudNativePG)"]
  B -->|"data lost"| R["Restore from backup or PITR"]
  A --> K{"Need to stop an agent<br/>meanwhile?"}
  K -->|yes| Y["kill --emit-yaml<br/>| kubectl apply"]
  N --> V[Verify]
  F --> V
  R --> C["Compare heads:<br/>ledger-divergence runbook"]
  C --> V

Respond#

Step 1: Do not restart all gateways at once#

Ready pods serve reads from their last folded state. Restarting every replica makes them all unready at the same time. Leave them running while you fix the database.

Step 2: Keep the kill switch working#

The gateway's break-glass API needs the database, but an AgentRevocation goes through Kubernetes: the operator renders it into the revocations ConfigMap, which every gateway reads through configSync.

maqpna kill -n team-a --agent coder --reason "INC-123" --terminate --emit-yaml | kubectl apply -f -

See the kill switch runbook.

Step 3: Restore database service#

1. Find why the gateways cannot connect: the database itself, the network path, or the DSN Secret (maqpna-state, key dsn, unless you changed state.postgres.dsnSecret).
2. Fail over with your operator (Patroni, CloudNativePG) or restart the database.
3. The gateways' pools reconnect by themselves. Replicas catch up through the poller (state.postgres.pollMillis). In-flight transactions fail and are retried by their callers. The schema is migrated again at start under an advisory lock; migrations are additive.

Step 4: If the database was lost, restore it#

Restore from your point-in-time recovery, or from the last MAQPNA backup:

maqpna restore /backups/<latest> --components postgres --postgres-dsn-file dsn.txt --dry-run
maqpna restore /backups/<latest> --components postgres --postgres-dsn-file dsn.txt

It refuses to roll back a database whose audit_chain is ahead of the backup unless you pass --force. Then compare the restored head with the ledger mirror on each gateway volume and with the WORM and SIEM copies (ledger divergence runbook). A gateway whose local ledger is not a prefix of the restored chain refuses to start; that runbook covers it.

Revocations made through AgentRevocation objects (step 2) survive the restore. Break-glass revocations made through the gateway after the backup do not; re-apply them.

Verify#

kubectl -n maqpna-system get pods -l app.kubernetes.io/name=maqpna-gateway     # all ready
maqpna audit verify --postgres dsn.txt --schema maqpna
maqpna audit verify --gateway "$GW"
maqpna audit sinks --gateway "$GW" --max-lag 1000     # shipping caught up (exit 3 if not)
maqpna approvals list --gateway "$GW" -A              # pending approvals are back
maqpna revocations list --gateway "$GW"               # every revocation you expect is listed

GW is the gateway admin URL; you need the auditor or admin role.

Troubleshooting#

Common problems while you run this runbook:

Follow-up#

• Run a restore drill to confirm backups still restore:

maqpna dr drill --gateway "$GW" --postgres-scratch-dsn-file scratch-dsn.txt

• Record how long the gateways were unready and how many calls were refused (MaqpnaGatewayNotReady duration, denials in the ledger after recovery).
• Review whether auditFailurePolicy and the database's HA setup match what the business expects.
• If shipping lagged, check that MaqpnaAuditSinkLag cleared.

Next steps#

• Scaling and high availability
• HA and failure behaviour
• Backup, restore and DR drill
• Runbook: ledger divergence