Runbook: key rotation

Rotate the identity signing key, the audit checkpoint key, the token-vault key and the attest response key without downtime, on a schedule or after a suspected leak.

Use this runbook to rotate a customer-held key, on a schedule or because it may have leaked. maqpna keys rotate rotates each key without downtime: every key lives in a Kubernetes Secret mounted as a directory, and the servers also load rotation keys from files next to the active one. A rotation edits only the Secret and restarts the Deployment that reads it. Progress is recorded on the Secret (annotation maqpna.com/key-rotation), so you can finish an interrupted rotation later.

Keys held in a KMS or HSM (kms:// or pkcs11: URIs) rotate in the KMS instead; this runbook does not apply to them.

Symptoms (unscheduled rotation)#

• A Secret holding a signing key was read by someone who should not have it, or a node that mounted it is compromised.
• A backup holding unsealed keys was exposed.
• Your key policy says the key is due.

Impact#

No call fails during a rotation: a token is only ever signed with a key the gateways already trust.

Detection#

GW is the gateway admin URL (the adminListen port, for example through kubectl -n maqpna-system port-forward svc/maqpna-gateway 9090:9090). Sign in with maqpna login, or pass --oidc-token-file to the commands that take --gateway.

export GW=http://127.0.0.1:9090
maqpna keys inventory --resolve      # what keys exist, where they live, their algorithms
kubectl -n maqpna-system get secrets -o custom-columns='NAME:.metadata.name,ROTATION:.metadata.annotations.maqpna\.com/key-rotation'

Decide#

flowchart TD
  A[Rotate a key] --> B{"Which key?"}
  B -->|identity| I["keys rotate identity<br/>--grace >= longest token TTL"]
  B -->|audit checkpoint| C["keys rotate audit-checkpoint<br/>old key kept as retired"]
  B -->|token vault| V["keys rotate vault<br/>later --drop-old"]
  B -->|attest response| T["keys rotate attest<br/>--grace >= renewal window"]
  B -->|"kms:// or pkcs11:"| K["Rotate in the KMS"]
  I --> L{"Leaked?"}
  L -->|yes| F["--finish --force after<br/>gateways refresh the JWKS"]
  L -->|no| G["Let the grace period run"]
  C --> BK[Take a new backup]
  V --> BK
  T --> BK
  F --> BK
  G --> BK

Common flags: -n NS (default: the control-plane namespace, maqpna-system), --dry-run, --timeout (per rollout, default 5m), --secret and --deployment to override the chart's names, and --gateway URL to confirm results through the admin API. Start every rotation with --dry-run.

Respond#

Identity signing key#

The identity broker signs session tokens; gateways verify them with the broker's JWKS, refreshed every identity.jwksRefreshSeconds (default 300 s). Requires identity.key.mode helm or existingSecret.

maqpna keys rotate identity --dry-run
maqpna keys rotate identity --grace 1h       # grace >= the longest token TTL

1. The new public key is added as previous-<kid>.pem and the broker restarts. The JWKS lists old and new keys while tokens are still signed with the old one.
2. The CLI waits --jwks-wait (default 5m30s) so every gateway refreshes the JWKS. --restart-gateway restarts the gateway instead of waiting.
3. The broker switches to the new key (key.pem). The old public key stays as previous-<oldkid>.pem, so tokens issued before the switch keep verifying for --grace.
4. After the grace period the old key is removed.

With --no-finish the CLI stops after step 3. Finish later:

maqpna keys rotate identity --finish          # refuses before the grace period ends
maqpna keys rotate identity --finish --force  # leaked key: remove it now

After the finish, tokens signed with the old key are rejected once the gateways refresh the JWKS. The chart keeps the previous-*.pem entries across helm upgrade.

Audit checkpoint key#

maqpna keys rotate audit-checkpoint --dry-run
maqpna keys rotate audit-checkpoint
maqpna audit fetch jwks --gateway "$GW" --out audit-jwks.json     # refresh offline copies

The Ed25519 key that signs ledger checkpoints and memory-erasure certificates is replaced. The old public key is kept as retired-<kid>.pub.pem, so /v1/audit/jwks lists both and earlier checkpoints stay verifiable. Retired keys are never removed automatically: keep them as long as the ledger is retained.

Token-vault key#

maqpna keys rotate vault --gateway "$GW" --dry-run
maqpna keys rotate vault --gateway "$GW"

1. The new key is added as retired-<id> and the gateways restart, so every replica can open records sealed with it before any replica seals with it.
2. kek becomes the new key and the old one retired-<id>. The gateways restart and re-seal every account at start. POST /v1/vault:rekey re-seals anything left, and GET /v1/vault/keys reports the accounts per key ID.
3. Later, remove the old key. The CLI refuses until the gateway reports no account sealed under it (notUnderPrimary: 0):

maqpna keys rotate vault --gateway "$GW" --drop-old

This works for the file vault and the shared PostgreSQL vault alike.

Attest response key#

maqpna keys rotate attest --secret maqpna-attest-response-signing --dry-run
maqpna keys rotate attest --secret maqpna-attest-response-signing --grace 24h

1. pub.pem lists old and new public keys and the operator restarts: new sessions trust both.
2. After --grace (at least the longest session renewal window, so sessions that trust only the old key have ended), the attestation service signs with the new key.
3. pub.pem is reduced to the new key and the operator restarts.

--no-finish stops after step 2; finish with maqpna keys rotate attest --secret maqpna-attest-response-signing --finish. The Secret name is your attestation.responseSigning.secretName.

Other keys#

Verify#

kubectl -n maqpna-system get secret <secret> -o jsonpath='{.metadata.annotations.maqpna\.com/key-rotation}'
helm test maqpna -n maqpna-system              # tokens still verify end to end
maqpna audit verify --gateway "$GW"             # the ledger and its checkpoints verify
maqpna doctor -n maqpna-system --gateway "$GW" --strict

Follow-up#

• Take a new backup right away: the sealed Secrets in older backups hold the old keys.

maqpna backup create --out /backups/$(date +%F)-post-rotation --gateway "$GW" --seal-to recipient.pub

• Record the old and new key IDs, the time of the switch and the finish, and who ran it.
• After a leak, look for tokens or checkpoints signed with the old key after the suspected leak time in the audit ledger, and follow the compromised agent runbook.

Next steps#

• Backup, restore and DR drill
• Key management
• Cloud identity & keys: keys in a KMS.
• Command reference: maqpna keys rotate