Workload identity with SPIFFE and SPIRE
Give each sandbox an X.509-SVID whose SPIFFE ID equals its session token subject, and make the gateway reject calls whose client certificate does not match.
Every session gets a signed session token from the identity broker. The token's subject is already a workload identity (SPIFFE) ID: spiffe://<trust domain>/ns/<namespace>/agent/<agent>/session/<session>. A token is still a bearer credential, so anyone who copies it can use it until it expires.
With SPIRE, the sandbox also receives an X.509-SVID, a short-lived client certificate for the same SPIFFE ID. The gateway serves HTTPS, checks the client certificate on every agent call and requires it to equal the token subject. A stolen token alone is then not enough. The MAQPNA interop suite runs this nightly against the SPIRE Helm chart.
flowchart LR
O["MAQPNA operator"] -->|"labels and annotates<br/>sandbox pod"| P["Sandbox pod<br/>maqpna.com/spiffe=enabled"]
CS["ClusterSPIFFEID<br/>(rendered by the chart)"] --> SC["spire-controller-manager"]
SC -->|"registration entry"| SS["SPIRE server"]
SS --> SA["SPIRE agent<br/>on the node"]
SA -->|"X.509-SVID"| P
P -->|"mTLS + session token"| G["MAQPNA gateway<br/>svidMode"]
B["spire-bundle ConfigMap"] -->|"trust bundle"| G
G -->|"SVID = token subject?"| D{"allow / 401"}
When to use it#
- You already run SPIRE, or want a second factor next to the session token.
- Your threat model includes token theft from logs, files or memory dumps.
- You need the ledger to show which workload certificate made a call (
ext.svid).
If you cannot deliver SVIDs into the sandbox, look at DPoP sender-constrained tokens instead.
Prerequisites#
- SPIRE server, SPIRE agents and spire-controller-manager, installed separately (the hardened SPIRE Helm charts work).
- A TLS serving certificate for the gateway: a
kubernetes.io/tlsSecret with DNS SANmaqpna-gateway.<namespace>.svc. - The gateway reached directly over TLS. Do not put a TLS-terminating ingress or service-mesh sidecar in front of it: the gateway does not trust
X-Forwarded-Client-Cert.
Step 1: Install SPIRE with MAQPNA's trust domain#
The SPIRE trust domain must equal MAQPNA's identity.trustDomain (default maqpna.local), because the SVID must match the token subject byte for byte.
REPO=https://spiffe.github.io/helm-charts-hardened
helm upgrade --install spire-crds spire-crds --repo $REPO -n spire-mgmt --create-namespace
helm upgrade --install spire spire --repo $REPO -n spire-mgmt \
--set global.spire.trustDomain=maqpna.local \
--set global.spire.clusterName=prod-eu-1 \
--set spire-server.controllerManager.identities.clusterSPIFFEIDs.default.enabled=false
Disable the chart's default ClusterSPIFFEID (/ns/<ns>/sa/<sa>), or exclude the agent namespaces from it. Otherwise sandbox pods get two SVIDs and may present the wrong one (svid_mismatch).
Step 2: Copy the trust bundle#
SPIRE's bundle notifier writes ConfigMap spire-bundle (key bundle.spiffe, SPIFFE bundle format) in the SPIRE namespace. Copy it into the MAQPNA namespace, or configure the notifier to write there:
kubectl get configmap spire-bundle -n spire-mgmt -o json \
| jq '{apiVersion, kind, data, metadata: {name: .metadata.name, namespace: "maqpna-system"}}' \
| kubectl apply -f -
The gateway accepts the SPIFFE bundle format (a JWKS with x509-svid keys) and PEM.
Step 3: Set the Helm values#
identity:
trustDomain: maqpna.local # = SPIRE's trust domain
spire:
enabled: true # operator labels sandbox pods; the chart renders a ClusterSPIFFEID
clusterSPIFFEID:
create: true
className: spire-mgmt-spire # spire-controller-manager class: "<namespace>-<release>"
ttl: "1h" # optional upper bound of the SVID lifetime
namespaceSelector: {}
gateway:
tls:
enabled: true
certSecret: maqpna-gateway-tls # kubernetes.io/tls
clientCAConfigMap: spire-bundle
clientCAKey: bundle.spiffe # default bundle.crt
svidMode: optional # off | optional | required
svidTrustDomains: [maqpna.local] # empty = any trust domain in the bundle
With spire.enabled, the operator runs with --spire-workload-registration and gives each sandbox pod:
| Pod metadata | Value |
|---|---|
Label maqpna.com/spiffe |
enabled (the ClusterSPIFFEID selects on it) |
Annotation maqpna.com/spiffe-id |
The session's SPIFFE ID, equal to the token subject |
Annotation maqpna.com/spiffe-trust-domain |
identity.trustDomain |
The rendered ClusterSPIFFEID builds the ID from the trust-domain annotation, the pod's own namespace and the operator's maqpna.com/agent and maqpna.com/session labels. It never uses a free-form annotation, so a pod creator cannot claim another namespace's session.
The gateway re-reads its certificate and the trust bundle every 30 seconds, so SPIRE CA rotations are picked up without a restart.
Step 4: Choose the SVID mode#
For /mcp, /llm, /a2a, egress and exec calls, after it verifies the session token, the gateway checks:
| Situation | optional |
required |
|---|---|---|
| No client certificate | Allowed | 401 svid_required |
| SVID equals the token subject | Allowed, ext.svid in the ledger |
Allowed, ext.svid |
| SVID of another session or workload | 401 svid_mismatch |
401 svid_mismatch |
SVID outside svidTrustDomains |
401 svid_trust_domain |
401 svid_trust_domain |
| Certificate that does not chain to the bundle | 401 svid_invalid |
401 svid_invalid |
With off, client certificates are ignored. Health probes and the admin API never need a client certificate: the TLS layer asks for a certificate but does not require it at the handshake.
Start with optional. Move to required when every caller presents an SVID.
Verify#
-
Check that spire-controller-manager created an entry for a running session:
kubectl exec -n spire-mgmt spire-server-0 -c spire-server -- \ /opt/spire/bin/spire-server entry show -spiffeID spiffe://maqpna.local/ns/team-a/agent/coder/session/<session> -
Scrape the gateway's
/metricsand readmaqpna_gateway_svid_checks_total{result=...}. The results arebound,absent,missing,mismatch,trust_domainandinvalid. - Find a record with
ext.svidin the ledger:maqpna audit tail --session <session>.
Troubleshooting#
| Symptom | Cause | Fix |
|---|---|---|
401 svid_mismatch |
The pod has a second SVID from SPIRE's default ClusterSPIFFEID |
Disable the default ClusterSPIFFEID or exclude agent namespaces |
401 svid_mismatch for every call |
SPIRE trust domain differs from identity.trustDomain |
Use the same trust domain on both sides |
401 svid_required from agents in tier-0 |
The sandbox cannot reach the SPIRE agent socket | Use svidMode: optional (see the note above) |
401 svid_invalid |
The bundle ConfigMap is missing, stale or uses the wrong key | Copy spire-bundle again; set clientCAKey: bundle.spiffe |
| No registration entries appear | className does not match spire-controller-manager's class |
Set spire.clusterSPIFFEID.className to <namespace>-<release> of the SPIRE chart |
| Client certificates never arrive | A TLS-terminating proxy or mesh sits in front of the gateway | Let the gateway terminate TLS itself |