Troubleshooting the stack

Error messages you can see when you connect the MAQPNA gateway to llamay and kosa8, what causes them and how to fix them, plus diagnostic commands and the support bundle.

This page lists the messages MAQPNA prints when its connection to llamay or kosa8 goes wrong, what causes each one, and how to fix it. The messages are quoted as the code writes them, so you can search this page for the text you see. For problems with MAQPNA itself, read the general troubleshooting guide.

Where to look first#

On a laptop, maqpna dev up keeps its state in .maqpna/ in the directory you started it from.

What	Where
Gateway log	`.maqpna/logs/maqpna-gateway.log`
Identity broker log	`.maqpna/logs/maqpna-identity.log`
llamay log	`.maqpna/logs/llamay.log`
Generated gateway config	`.maqpna/gateway.json`
MAQPNA audit ledger	`.maqpna/audit.jsonl`
llamay request audit	`.maqpna/llamay-audit.jsonl`
llamay checkpoints	`.maqpna/checkpoints/llamay/`

In a cluster, read the gateway pod's log and run maqpna doctor.

Diagnostic commands#

maqpna dev status                     # which local components run, and their PIDs
eval "$(maqpna dev env)"              # gateway URL and admin token in this shell
maqpna dev timeline --last            # what the last session did, call by call
maqpna dev attest --session <session> # kosa8 measurements and egress (exit 3: egress not the gateway)
maqpna llamay verify-audit .maqpna/llamay-audit.jsonl   # exit 3: chain broken
maqpna-gateway -check-config -config .maqpna/gateway.json  # validate the config, start nothing
curl -s -H "Authorization: Bearer $MAQPNA_ADMIN_TOKEN" "$MAQPNA_GATEWAY_URL/metrics" | grep -E 'llamay|kosa8'

On the product side:

curl -s http://127.0.0.1:11435/healthz    # llamay, the only route without a key
kosa8 doctor                              # daemon, guest images, virtualization
kosa8 audit verify                        # kosa8's own audit chain

The metrics to watch:

Metric	Healthy value
`maqpna_gateway_llamay_audit_verified{anchor}`	`1`
`maqpna_gateway_llamay_audit_head_seq{anchor}`	Rising while llamay serves calls
`maqpna_gateway_llamay_context_ops_total{op}`	`error` and `stale` low compared with `create`
`maqpna_gateway_llamay_anchor_append_failures_total{anchor}`	`0`
`maqpna_gateway_kosa8_audit_verified`	`1`
`maqpna_gateway_kosa8_mirror_total{outcome}`	`error` and `dropped` at `0`
`maqpna_gateway_kosa8_anchor_append_failures_total`	`0`

Starting the local stack#

Symptom	Cause	Fix
`--llamay needs the llamay binary on PATH or --llamay-bin (…)`	`llamay` is not installed, or not on `PATH`.	Install llamay from llamay.com/download, or pass `--llamay-bin /path/to/llamay`.
`llamay: … (see .maqpna/logs/llamay.log)`	`llamay serve` did not answer `/healthz` within 2 minutes. Often the model path or store name is wrong, or the model is still loading.	Read `.maqpna/logs/llamay.log`. Check the GGUF path or store name you gave `--llamay`.
`port 11435 is in use (…)`	Another llamay, or another process, uses the port.	Stop it, or pass `--llamay-port`.
`a local MAQPNA is already running from .maqpna (maqpna dev down first)`	`dev up` was already run in this directory.	Run `maqpna dev down`, then `dev up` again with the new flags.
--kosa8: kosa8d unreachable (is `kosa8 daemon start` running?): …	kosa8d is not running, or its socket is not at `~/.kosa8/run/kosa8.sock`.	Run `kosa8 daemon start` (the kosa8 Mac app starts it for you), then `kosa8 doctor`.
`cannot find this host's LAN address (use --kosa8-host-ip): …` or `this host has no LAN IPv4 address (use --kosa8-host-ip)`	kosa8 guests cannot reach your loopback, so the gateway must listen on a LAN IPv4 address, and none was found.	Pass `--kosa8-host-ip <your IPv4>`.
`--kosa8-host-ip must be a non-loopback IPv4 address`	You passed `127.0.0.1`, an IPv6 address or a name.	Pass the host's LAN IPv4 address.

Connecting the gateway to llamay#

Symptom	Cause	Fix
Model calls get `503` with code `llamay_audit_unverified`	A `failClosed` audit anchor does not verify, so every llamay endpoint is out of routing. The ledger has a `llamay/audit-unverified` record with the reason.	Find the reason (next rows) in the gateway log line `llamay request audit does not verify`, or in the record. Fix the file; routing resumes on the next check that verifies. Add a non-llamay fallback endpoint if calls must continue meanwhile.
`audit file unreadable: …`	The gateway cannot open the anchor's `file`. In a cluster, the audit volume is not mounted into the gateway.	Check the path in `llamay.auditAnchors[].file`. In Helm, the volume must be `ReadWriteMany` with `llamay.audit.anchor.enabled: true`.
`llamay audit chain broken: record seq N was altered (hash mismatch)`	A record was changed after it was written.	Treat this as an incident: keep the file, and follow the ledger divergence runbook.
`llamay audit chain broken: record seq N does not link to seq M` or `… follows seq M`	Records were removed, reordered or copied in from another chain.	As above.
`llamay audit chain broken: first record seq N does not start from genesis`	The file continues an earlier file, but the anchor does not allow that.	If llamay runs with `-audit-continue`, set `"continuation": true` on the anchor.
`rotated audit file continues …, not the verified head …`	The file was rotated, and the new file continues a chain other than the one the gateway verified.	Restore the right previous file, or investigate the rotation.
`audit file was rewritten: it no longer holds the verified chain`	The periodic full re-check found that the file no longer contains the head the gateway pinned.	Treat as an incident, as above.
`llamay/audit-anchor` record with `ext.llamayAuditRestartedFrom`	llamay restarted without `-audit-continue`, so a new chain started from genesis.	Expected after a plain restart. Use `-audit-continue` to keep one chain across restarts.
Gateway log: `llamay context create failed; call runs unbound` or `llamay context unavailable; call runs unbound`	llamay refused or failed `POST /v1/contexts`. The call still runs, without a context.	Check llamay's log. Calls are correct; they cost a full prefill.
Gateway log: `llamay context fork failed; creating a fresh context`	The parent's context is gone (expired, dropped or llamay restarted).	None needed: the child gets a fresh context.
Model calls fail; gateway log shows `upstream credential unavailable: llamay endpoint … key: …`	The gateway cannot read the llamay key file.	Check `llamay.apiKeyFile`, or the endpoint's own `apiKeyFile`, and its permissions.
`401` from llamay	Wrong or rotated key.	Put the current key in the key file. The gateway re-reads it on every call.
`503` from llamay	llamay with `-audit-strict` could not write an audit record.	Check the audit volume's free space and permissions. The gateway fails over or returns the error.
`llamay.upstream: invalid URL "…"` at start or in `-check-config`	`upstream` is not an `http://` or `https://` URL with a host.	Fix `llamay.upstream`, for example `http://maqpna-llamay.maqpna-system.svc:8081/v1`.
`llamay.auditAnchors[N]: name "…" is invalid or duplicate` / `file is required` / `intervalSeconds must be >= 0`	Invalid anchor config.	Give each anchor a unique, simple name, a file, and a non-negative interval.
`llamay: contextIdleSeconds and maxContexts must be >= 0`	Negative value in config.	Remove the field (defaults: 1800 and 4096) or set a positive value.
Agent pins a context with `llamay_session`, but each session still gets its own	Expected: the gateway strips `llamay_session` from every agent request.	None. Agents cannot choose a context.

Checkpoint and restore#

Symptom	Cause	Fix
`409`: `llamay.checkpointDir is not set: session checkpoints are off`	The gateway has no `llamay.checkpointDir`.	`dev up --llamay` sets it. In other configs, set `llamay.checkpointDir`.
`502`: `llamay snapshot of context …: …`	llamay refused the snapshot. A llamay that enforces classification markings refuses bare snapshots with `403`.	Context export and import are not wired into MAQPNA yet. Run without markings for checkpoints.
`403`: `session … is revoked (…): its llamay state is not restored`	The checkpointed session, or the new one, is revoked.	Expected. A revoked session's state is evidence, not something to resume.
`400`: `restore into a new session, not the checkpointed one`	The restore named the checkpointed session itself.	Restore into a new session (`maqpna dev restore` does).
`checkpoint …: sha256 … does not match the recorded …`	The checkpoint file changed after it was saved.	Do not restore it. Take a new checkpoint.
`no llamay endpoint … on route "…" any more: the checkpoint cannot be restored here`	The model route that served the context is gone from the config.	Restore on a gateway with the same llamay route.
`502`: `llamay restore of …: …`	llamay could not adopt the snapshot. Encrypted snapshots (llamay `-context-key`) restore only on a llamay with the same key.	Use the same llamay, with the same key.

Connecting the gateway to kosa8#

Symptom	Cause	Fix
`sandbox: egress is not pinned to the gateway: kosa8 sandboxes reach the gateway by a non-loopback IPv4 literal, not "127.0.0.1" (…)`	`dev run --kosa8-image` against a local MAQPNA started without `--kosa8`, so the gateway listens on loopback.	`maqpna dev down`, then `maqpna dev up --kosa8`.
`mint: HTTP … identity: proof refused: sandbox … egress "…" is not enforced to the gateway only`	The broker asked kosa8d, and the sandbox's egress is not exactly the gateway.	Run `maqpna dev attest --session <session>`. Do not change a sandbox's egress by hand.
`… proof refused: sandbox … measurement … sha256:… is not allowed`	A kernel, rootfs or image digest is not in the broker's `-sandbox-check-allowed` file. Common after a kosa8 upgrade.	Add the new digest, from `maqpna dev attest`, to the file.
`… proof refused: sandbox … runs … for "…", not …`	The sandbox's labels name another namespace, agent, user or session.	Start a new session with `maqpna dev run`; do not reuse sandboxes by name.
identity broker (is `maqpna dev up` running?): …	The broker is not running.	`maqpna dev status`, then `maqpna dev up`.
`kosa8sandbox: deliver the token: …`	The token could not be written to the container's stdin.	Check that the image has `/bin/sh`, then `kosa8 doctor`.
`session … has no kosa8 sandbox (start it with dev run --kosa8-image ... --kosa8-keep): …`	`dev fork`, `snapshot` or `attest` on a session whose sandbox was removed when its command exited.	Run the agent again with `--kosa8-keep`.
`sandbox: copy could not be re-keyed: … (removed): …`	After a fork or restore, MAQPNA could not write the child token into the clone (no running container, no `/bin/sh`, or a non-zero exit). The clone was removed.	Keep the agent running while you fork, and use an image with `/bin/sh`.
`kosa8sandbox: sandbox … runs no container`	The sandbox's agent already exited.	Fork while the agent runs, for example with `HOLD=30` in the example agent.
`kosa8: fork returned N clones, want 1`, or a kosa8 error such as `restore machine state: … permission denied`	kosa8 failed the fork. With little free disk, kosa8 0.5.0 failed forks this way.	Free disk space and retry. `dev fork` leaves no clone or child session behind.
`checkpoint … was taken with driver …, not kosa8 (use --sandbox …)`	The checkpoint came from another sandbox driver.	Pass the `--sandbox` value the message names.
`maqpna dev attest` exits with code 3 and prints `egress NOT PINNED`	The sandbox's egress is not the gateway only.	Remove the sandbox and start a new session.
Gateway log: `kosa8 audit anchor unavailable`	kosa8d did not answer the audit routes.	`kosa8 daemon start`, `kosa8 doctor`. The anchor retries on the next tick.
`kosa8/audit-unverified`: `kosa8 audit chain does not verify (broken at seq N): …`	kosa8d reports its own chain broken.	Run `kosa8 audit verify`, keep the log, and treat it as an incident.
`kosa8/audit-unverified`: `kosa8 audit head went back from seq … to seq …: the log was replaced or truncated`	The kosa8 log is shorter than, or different from, what the ledger anchored.	Treat as an incident. kosa8d verifies with its own key, so only this record shows the replacement.
Gateway log: `kosa8 decision mirror failed`, with kosa8 saying `refusing to append unauthenticated entries`	kosa8d's own audit key is missing, so it refuses AI records.	Fix kosa8's audit key setup (see the kosa8 docs). Mirror errors are counted in `maqpna_gateway_kosa8_mirror_total{outcome="error"}`.
`maqpna_gateway_kosa8_mirror_total{outcome="dropped"}` rising	More decisions than `kosa8.queueSize` (default 1024) were waiting.	Raise `queueSize`, or check that kosa8d keeps up. The MAQPNA ledger still has every decision.
`kosa8.url: "…" is plain HTTP to a non-loopback host; put kosa8d behind TLS (https://)`	`kosa8.url` would send the kosa8d token in clear text.	Put kosa8d behind TLS and use `https://`.
`kosa8.tokenFile is required with kosa8.url (kosa8d refuses TCP without a token)`	`url` without `tokenFile`.	Add `tokenFile`.
`kosa8: set socket or url, not both`	Both are set.	Keep one.
`kosa8: a TCP daemon needs a token (kosa8d --token / KOSA8_TOKEN)`	`KOSA8_HOST` is set in your shell without `KOSA8_TOKEN`.	Set `KOSA8_TOKEN`, or unset `KOSA8_HOST` to use the local socket.

An agent in a fork acts as its parent#

The clone resumes before MAQPNA re-keys it. An agent that read its token once at start, for example into OPENAI_API_KEY, keeps the parent's identity after the fork. MAQPNA_SESSION in the clone's environment also still names the parent. Read MAQPNA_TOKEN_FILE on every call (the MAQPNA SDKs do), and use the token's claims, not MAQPNA_SESSION, to know which session you are. See the security model.

Collect a support bundle#

In a cluster, maqpna support-bundle writes a redacted .tar.gz with what support needs:

maqpna support-bundle --since 2h --gateway "$GW" --token "$MAQPNA_ADMIN_TOKEN" --out bundle.tar.gz

It never collects Secret values, only their names and keys. Logs, values, ConfigMaps and custom resources are redacted: bearer credentials, JWTs, passwords, private keys and values of keys named like token, secret, password or apiKey. Review the archive before you send it.

support-bundle reads a Kubernetes installation. For a local stack it does not apply. Instead, send:

maqpna dev status -o json and maqpna dev timeline --session <session> -o json;
the log lines around the error from .maqpna/logs/;
maqpna llamay verify-audit .maqpna/llamay-audit.jsonl -o json and maqpna dev attest -o json;
the output of kosa8 doctor and llamay version.

Do not send .maqpna/dev.env, .maqpna/llamay.key or other key files: they hold tokens and keys.