MAQPNADocs

Workload identity with SPIFFE and SPIRE

Give each sandbox an X.509-SVID whose SPIFFE ID equals its session token subject, and make the gateway reject calls whose client certificate does not match.

Every session gets a signed session token from the identity broker. The token's subject is already a workload identity (SPIFFE) ID: spiffe://<trust domain>/ns/<namespace>/agent/<agent>/session/<session>. A token is still a bearer credential, so anyone who copies it can use it until it expires.

With SPIRE, the sandbox also receives an X.509-SVID, a short-lived client certificate for the same SPIFFE ID. The gateway serves HTTPS, checks the client certificate on every agent call and requires it to equal the token subject. A stolen token alone is then not enough. The MAQPNA interop suite runs this nightly against the SPIRE Helm chart.

flowchart LR
    O["MAQPNA operator"] -->|"labels and annotates<br/>sandbox pod"| P["Sandbox pod<br/>maqpna.com/spiffe=enabled"]
    CS["ClusterSPIFFEID<br/>(rendered by the chart)"] --> SC["spire-controller-manager"]
    SC -->|"registration entry"| SS["SPIRE server"]
    SS --> SA["SPIRE agent<br/>on the node"]
    SA -->|"X.509-SVID"| P
    P -->|"mTLS + session token"| G["MAQPNA gateway<br/>svidMode"]
    B["spire-bundle ConfigMap"] -->|"trust bundle"| G
    G -->|"SVID = token subject?"| D{"allow / 401"}

When to use it#

  • You already run SPIRE, or want a second factor next to the session token.
  • Your threat model includes token theft from logs, files or memory dumps.
  • You need the ledger to show which workload certificate made a call (ext.svid).

If you cannot deliver SVIDs into the sandbox, look at DPoP sender-constrained tokens instead.

Prerequisites#

  • SPIRE server, SPIRE agents and spire-controller-manager, installed separately (the hardened SPIRE Helm charts work).
  • A TLS serving certificate for the gateway: a kubernetes.io/tls Secret with DNS SAN maqpna-gateway.<namespace>.svc.
  • The gateway reached directly over TLS. Do not put a TLS-terminating ingress or service-mesh sidecar in front of it: the gateway does not trust X-Forwarded-Client-Cert.

Step 1: Install SPIRE with MAQPNA's trust domain#

The SPIRE trust domain must equal MAQPNA's identity.trustDomain (default maqpna.local), because the SVID must match the token subject byte for byte.

REPO=https://spiffe.github.io/helm-charts-hardened
helm upgrade --install spire-crds spire-crds --repo $REPO -n spire-mgmt --create-namespace
helm upgrade --install spire spire --repo $REPO -n spire-mgmt \
  --set global.spire.trustDomain=maqpna.local \
  --set global.spire.clusterName=prod-eu-1 \
  --set spire-server.controllerManager.identities.clusterSPIFFEIDs.default.enabled=false

Disable the chart's default ClusterSPIFFEID (/ns/<ns>/sa/<sa>), or exclude the agent namespaces from it. Otherwise sandbox pods get two SVIDs and may present the wrong one (svid_mismatch).

Step 2: Copy the trust bundle#

SPIRE's bundle notifier writes ConfigMap spire-bundle (key bundle.spiffe, SPIFFE bundle format) in the SPIRE namespace. Copy it into the MAQPNA namespace, or configure the notifier to write there:

kubectl get configmap spire-bundle -n spire-mgmt -o json \
  | jq '{apiVersion, kind, data, metadata: {name: .metadata.name, namespace: "maqpna-system"}}' \
  | kubectl apply -f -

The gateway accepts the SPIFFE bundle format (a JWKS with x509-svid keys) and PEM.

Step 3: Set the Helm values#

identity:
  trustDomain: maqpna.local              # = SPIRE's trust domain

spire:
  enabled: true                          # operator labels sandbox pods; the chart renders a ClusterSPIFFEID
  clusterSPIFFEID:
    create: true
    className: spire-mgmt-spire          # spire-controller-manager class: "<namespace>-<release>"
    ttl: "1h"                            # optional upper bound of the SVID lifetime
    namespaceSelector: {}

gateway:
  tls:
    enabled: true
    certSecret: maqpna-gateway-tls       # kubernetes.io/tls
    clientCAConfigMap: spire-bundle
    clientCAKey: bundle.spiffe           # default bundle.crt
    svidMode: optional                   # off | optional | required
    svidTrustDomains: [maqpna.local]     # empty = any trust domain in the bundle

With spire.enabled, the operator runs with --spire-workload-registration and gives each sandbox pod:

Pod metadata Value
Label maqpna.com/spiffe enabled (the ClusterSPIFFEID selects on it)
Annotation maqpna.com/spiffe-id The session's SPIFFE ID, equal to the token subject
Annotation maqpna.com/spiffe-trust-domain identity.trustDomain

The rendered ClusterSPIFFEID builds the ID from the trust-domain annotation, the pod's own namespace and the operator's maqpna.com/agent and maqpna.com/session labels. It never uses a free-form annotation, so a pod creator cannot claim another namespace's session.

The gateway re-reads its certificate and the trust bundle every 30 seconds, so SPIRE CA rotations are picked up without a restart.

Step 4: Choose the SVID mode#

For /mcp, /llm, /a2a, egress and exec calls, after it verifies the session token, the gateway checks:

Situation optional required
No client certificate Allowed 401 svid_required
SVID equals the token subject Allowed, ext.svid in the ledger Allowed, ext.svid
SVID of another session or workload 401 svid_mismatch 401 svid_mismatch
SVID outside svidTrustDomains 401 svid_trust_domain 401 svid_trust_domain
Certificate that does not chain to the bundle 401 svid_invalid 401 svid_invalid

With off, client certificates are ignored. Health probes and the admin API never need a client certificate: the TLS layer asks for a certificate but does not require it at the handshake.

Start with optional. Move to required when every caller presents an SVID.

Verify#

  1. Check that spire-controller-manager created an entry for a running session:

    kubectl exec -n spire-mgmt spire-server-0 -c spire-server -- \
      /opt/spire/bin/spire-server entry show -spiffeID spiffe://maqpna.local/ns/team-a/agent/coder/session/<session>
    
  2. Scrape the gateway's /metrics and read maqpna_gateway_svid_checks_total{result=...}. The results are bound, absent, missing, mismatch, trust_domain and invalid.

  3. Find a record with ext.svid in the ledger: maqpna audit tail --session <session>.

Troubleshooting#

Symptom Cause Fix
401 svid_mismatch The pod has a second SVID from SPIRE's default ClusterSPIFFEID Disable the default ClusterSPIFFEID or exclude agent namespaces
401 svid_mismatch for every call SPIRE trust domain differs from identity.trustDomain Use the same trust domain on both sides
401 svid_required from agents in tier-0 The sandbox cannot reach the SPIRE agent socket Use svidMode: optional (see the note above)
401 svid_invalid The bundle ConfigMap is missing, stale or uses the wrong key Copy spire-bundle again; set clientCAKey: bundle.spiffe
No registration entries appear className does not match spire-controller-manager's class Set spire.clusterSPIFFEID.className to <namespace>-<release> of the SPIRE chart
Client certificates never arrive A TLS-terminating proxy or mesh sits in front of the gateway Let the gateway terminate TLS itself

Next steps#