Runbook: attestation failure

What to do when tier-2 sessions fail attestation, from reading the failure reason to fixing reference values, clocks or the verifier and restarting the affected sessions.

Use this runbook when sessions of an attestation-gated trust tier (tier-2, confidential VM) do not start because attestation fails. Before the identity broker issues a tier-2 session its identity, the MAQPNA attestation service (maqpna-attest) checks hardware evidence from the sandbox against a verifier (Trustee) and your reference values. If any check fails, the session gets no identity and ends Failed with reason AttestationFailed.

How the flow works: Attestation flow.

Symptoms#

• The MaqpnaAttestationFailureRate alert fires: more than 10% of attestations fail.
• Sessions of a tier with attestationRequired: true end Failed with reason AttestationFailed.
• The sandbox's attest init container (maqpna-attest) or renewal sidecar (maqpna-attest-renewer) exits with attest response: or renew response: errors.

Impact#

• New tier-2 sessions cannot start. Sessions of tier-0 and tier-1 are not affected.
• Running tier-2 sessions keep working until their token needs renewal; then the renewal fails and they lose their identity.
• Nothing is released without passing attestation: the failure is closed, by design.

Detection#

Set these once for the commands below:

export NS=team-a SESSION=<session-name>
POD=$(kubectl -n "$NS" get pods -l maqpna.com/session="$SESSION" -o jsonpath='{.items[0].metadata.name}')

Decide#

flowchart TD
  A[Attestation fails] --> B{"reason in<br/>maqpna_attest_failure_total"}
  B -->|"measurement, svn, debug, tee"| C["Guest image, firmware or<br/>TEE changed: review it"]
  C -->|"change is expected"| C2["Publish new reference values"]
  C -->|"change is unexpected"| SEC["Security incident:<br/>compromised-agent runbook"]
  B -->|"nonce, replay, expired"| D["Check node clocks and retries"]
  B -->|"verifier-unavailable"| E["Check Trustee, URL, network"]
  B -->|"store"| F["Check release store /<br/>PostgreSQL"]
  B -->|"broker"| G["Check identity broker"]
  B -->|"signature, binding, malformed"| H{"agent log says<br/>response: signature?"}
  H -->|"during attest key rotation"| R["Wait for the grace period"]
  H -->|"no rotation running"| SEC
  C2 --> Z[Restart failed sessions]
  D --> Z
  E --> Z
  F --> Z
  G --> Z
  R --> Z

Respond#

Step 1: Read the reason#

kubectl -n maqpna-system port-forward deploy/maqpna-attest 8082:8082 &
curl -s localhost:8082/metrics | grep -E 'maqpna_attest_(failure|success)_total'
kubectl -n "$NS" logs "$POD" -c maqpna-attest
kubectl -n "$NS" logs "$POD" -c maqpna-attest-renewer      # renewals, when the sidecar is present
kubectl -n maqpna-system logs deploy/maqpna-attest --since 30m | grep '"decision":"deny"'

The attestation service logs every denial with its reason, the release ID, namespace, agent, session and tier.

Step 2: Act on the reason#

Renewals faster than -renew-min-interval are refused with rate-limited; that points to a sidecar setting (--renew-at), not to an attack.

Step 3: Publish new reference values (measurement changes)#

Reference values come from the chart (attestation.referenceValues) and are rendered into a ConfigMap the service reads. Change them through your values, never by editing the ConfigMap:

attestation:
  referenceValues:
    allowedMeasurements:
      - "<old measurement>"      # keep until no session uses the old image
      - "<new measurement>"

maqpna upgrade -f my-values.yaml --wait

An empty allowedMeasurements list rejects every report.

Step 4: Response-signature errors#

attest response: signature or renew response: signature in the sandbox means the response was not signed by a key the agent trusts.

• During an attest response key rotation, sessions created before maqpna keys rotate attest published the new key trust only the old one. The rotation waits --grace before it switches. Let it finish (Key rotation).
• Otherwise, something on the path between the sandbox and maqpna-attest substitutes responses. Treat it as a security incident: keep the pod logs, check the network path, and follow the compromised agent runbook.

Step 5: Restart the failed sessions#

Failed sessions do not retry by themselves. After the fix, delete them and start them again through your workflow:

maqpna session list -n "$NS" --phase Failed
maqpna session delete "$SESSION" -n "$NS" --wait

Verify#

curl -s localhost:8082/metrics | grep maqpna_attest_success_total     # rising again
maqpna session list -n "$NS"                                           # new sessions reach Running
maqpna doctor -n maqpna-system --component attest --strict             # attest-verifier and attest-response-signing pass

Follow-up#

• Record which change caused the failure and how the measurement was reviewed.
• Add measurement updates to your image release process, so new guest images ship with their reference values.
• If clocks drifted, add node clock monitoring.
• If the cause was a response-signature mismatch outside a rotation, finish the incident record from the compromised agent runbook.

Next steps#

• Sovereignty and confidential tiers
• Attestation
• Runbook: key rotation
• Monitoring and alerts