Runbook: ledger divergence or tampering

What to do when the audit ledger fails verification, replicas disagree on its head, or a SIEM copy differs, from establishing what is intact to restoring without rewriting the chain.

Use this runbook when the audit ledger may be damaged or changed. The ledger is append-only and hash-chained: every record holds the hash of the one before it, and signed checkpoints sign the head at points in time. A break in the chain, or a record that no longer matches a signed checkpoint, means a crash, a storage fault, a misconfiguration or tampering. The rule: never rewrite the chain. Restore into a new ledger and keep the damaged copy as evidence.

Background: Audit ledger and Ledger format.

Symptoms#

• maqpna audit verify prints TAMPERED: chain breaks at seq N after M valid records and exits 3.
• Gateway replicas report different ledger heads.
• A gateway pod refuses to start because its local ledger is not a prefix of the shared chain (state.backend: postgres).
• MaqpnaAuditLedgerUnavailable fires: audit appends fail (maqpna_gateway_audit_errors_total rising).
• A SIEM or WORM copy has records the ledger does not have, or the other way round.

Impact#

• With gateway.auditFailurePolicy: closed, calls are refused while audit appends fail: agents stop, evidence is kept.
• With open, calls continue without an audit record; you lose evidence for that time.
• A chain break does not stop new appends by itself, but every evidence export over the broken range will fail verification until you restore.

Detection#

GW is the gateway admin URL; you need the auditor or admin role. dsn.txt holds the state database DSN; never put it on the command line.

Decide#

flowchart TD
  A["verify fails or heads differ"] --> B["Step 1: what is intact?<br/>first bad seq, last good head"]
  B --> C["Step 2: compare with backup,<br/>WORM, SIEM, checkpoints"]
  C --> D{"Finding"}
  D -->|"torn final line"| E["Crash during append:<br/>restart, verify again"]
  D -->|"break mid-chain,<br/>checkpoints fail"| F["Security incident:<br/>restore, keep copy"]
  D -->|"replicas differ,<br/>file backend"| G["Run 1 replica or<br/>move to postgres"]
  D -->|"audit_chain columns<br/>disagree"| H["Restore schema,<br/>restrict DB role"]
  D -->|"appends failing"| I["Fix storage or DB"]

Respond#

Step 1: Establish what is intact#

maqpna audit verify --gateway "$GW"                       # exit 3 if broken
maqpna audit verify --postgres dsn.txt --schema maqpna    # state.backend=postgres only
maqpna audit fetch ledger --gateway "$GW" --out now.jsonl # also writes now.jsonl.checkpoints.jws and .jwks.json
maqpna audit verify now.jsonl --jwks now.jsonl.jwks.json

Write down the first bad sequence number and the last good head (sequence number and hash). Keep now.jsonl and its companion files unchanged as evidence.

Step 2: Compare with independent copies#

• The last backup. maqpna backup verify /backups/<latest> prints its head; every record up to it must be identical.
• WORM segments (audit.sink: worm) and SIEM sinks hold copies with their own cursors (maqpna audit sinks --gateway "$GW").
• Signed checkpoints. A checkpoint signs a sequence number and hash. A record before the last checkpoint that no longer hashes to it was changed after signing; a missing tail after it was truncated.

Step 3: Act on the finding#

For a suspected tampering, also open a security incident and follow the compromised agent runbook steps for preserving evidence.

Step 4: Restore#

Restore into a new ledger file or database, never on top of the damaged one. Always dry-run first:

maqpna restore /backups/<latest> --components ledger --ledger-out /restore/audit.jsonl --dry-run
maqpna restore /backups/<latest> --components ledger --ledger-out /restore/audit.jsonl
maqpna restore /backups/<latest> --components postgres --postgres-dsn-file dsn.txt

• --components ledger writes the restored ledger to --ledger-out; copy it onto the gateway's audit volume. It refuses to replace a longer live ledger.
• --components postgres runs pg_restore of the state schema. It refuses to roll back a database whose audit_chain is ahead of the backup.
• --force overrides both checks. Use it only when you have decided that the newer records are the damaged ones, and you have kept a copy.
• Add --verify-key backup-signer.pub to refuse a backup whose signature does not verify.

Records written after the backup and lost with the corruption cannot be recreated. The signed checkpoints and the WORM and SIEM copies document them.

Verify#

maqpna audit verify --gateway "$GW"
maqpna audit verify --postgres dsn.txt --schema maqpna
maqpna audit sinks --gateway "$GW" --max-lag 1000      # exit 3 if a sink lags or has errors
maqpna doctor -n maqpna-system --gateway "$GW" --component audit --strict

Troubleshooting#

Common problems while you run this runbook:

Follow-up#

• File an incident record with the heads from steps 1 and 2, the first bad sequence number and the cause.
• If records were lost, attach the WORM or SIEM copies and the checkpoints that cover the gap.
• If the cause was storage, add alerts on volume fill and fsync latency; MaqpnaGatewayAuditCommitSlow warns early.
• If it was tampering, review who can write to the ledger volume and the state schema.
• Run maqpna dr drill to confirm backups still restore.

Next steps#

• Audit ledger guide
• Backup, restore and DR drill
• Audit ledger flow
• Command reference: maqpna audit verify, maqpna restore