Prometheus metrics

Every Prometheus metric the MAQPNA gateway, operator, identity broker and attestation service export, with the alert rules, dashboards and PromQL queries that ship with the Helm chart.

This page lists every Prometheus metric that MAQPNA components export, grouped by component. Use it when you write your own alerts or dashboards, or when you need to know what a series means. To turn on scraping, alerts and dashboards step by step, read Monitoring and alerts. For how metrics fit with traces and the audit ledger, read Observability.

Where metrics come from#

flowchart LR
    P["Prometheus"] -->|"scrape :8080/metrics<br/>or adminListen port"| GW["Gateway"]
    P -->|"scrape :8080/metrics"| OP["Operator"]
    P -->|"scrape :8082/metrics"| AT["Attestation service"]
    P -.->|"build info only"| ID["Identity broker :8081"]
    P --> R["PrometheusRule<br/>alerts"]
    G["Grafana"] --> P

The gateway has no client library. It writes the text format by hand, so some series only appear once the feature that owns them is configured (for example DPoP, the audit collector or a transparency log). A counter with labels has no series until the first event.

Every component#

Gateway#

Tool calls and MCP servers#

Policy, approvals and taint#

Data loss prevention (DLP)#

Kill switch and revocations#

Budgets and cost#

Model calls#

Agent-to-agent calls#

Sandbox tools#

Audit ledger#

Transparency log anchors#

Present when transparencyLog is configured.

Offline edge and audit collector#

The audit_forward and offline series come from an edge gateway with offline configured. The audit_collector series come from the central gateway with auditCollector configured.

Configuration, state and high availability#

OpenTelemetry#

Integrations#

Licence#

The gateway and the operator export the same licence series. The licence never affects agent traffic.

Operator#

Registered on the controller-runtime registry and served on --metrics-bind-address (default :8080).

Attestation service#

Alert rules in the chart#

Set monitoring.prometheusRule.enabled: true to render a PrometheusRule (monitoring.coreos.com/v1, needs the Prometheus Operator CRDs). Each alert links to <runbookBaseURL>#<alert name in lower case>. The default base URL is https://docs.maqpna.com/operations/monitoring-and-alerts/. Turn off single alerts with monitoring.prometheusRule.disabled, for example [MaqpnaBudgetExceeded].

All thresholds live under monitoring.prometheusRule.thresholds in the Helm values. Add labels to every alert with monitoring.prometheusRule.additionalAlertLabels, and labels on the rule object (for your Prometheus ruleSelector) with monitoring.prometheusRule.labels. What each alert means and what to do is in Monitoring and alerts.

Dashboards in the chart#

Set monitoring.dashboards.enabled: true to render one ConfigMap per file in deploy/helm/maqpna/dashboards/, labelled grafana_dashboard: "1" for the Grafana sidecar (kube-prometheus-stack). Each dashboard has a datasource variable, so it works with any Prometheus data source.

Other values: monitoring.dashboards.namespace (default: the release namespace), labels and annotations (for example grafana_folder: MAQPNA).

Useful PromQL#

Tool calls per second by decision:

sum by (action) (rate(maqpna_gateway_decisions_total[5m]))

Share of tool calls denied, per MCP server:

sum by (server) (rate(maqpna_gateway_decisions_total{action="deny"}[15m]))
  / sum by (server) (rate(maqpna_gateway_decisions_total[15m]))

p99 latency of each MCP server:

histogram_quantile(0.99, sum by (le, server) (rate(maqpna_gateway_upstream_latency_seconds_bucket[5m])))

Model tokens per hour, by route and kind:

sum by (route, kind) (increase(maqpna_gateway_model_tokens_total[1h]))

Model call error rate, by route:

sum by (route) (rate(maqpna_gateway_model_requests_total{status=~"5..|429"}[10m]))
  / sum by (route) (rate(maqpna_gateway_model_requests_total[10m]))

Spend so far (USD), highest replica:

max(maqpna_gateway_metered_cost_usd)

How far a replica's revocation list is behind (seconds):

max by (pod) (time() - maqpna_gateway_configsync_last_success_timestamp_seconds{configmap="maqpna-revocations"})

Average revocation propagation time per replica:

rate(maqpna_gateway_revocation_propagation_seconds_sum[15m])
  / rate(maqpna_gateway_revocation_propagation_seconds_count[15m])

Audit records written but not yet signed in a checkpoint:

max(maqpna_gateway_audit_head_seq) - max(maqpna_gateway_audit_signed_checkpoint_seq)

Warm-pool session start p90 by trust tier:

histogram_quantile(0.9, sum by (le, tier) (rate(maqpna_session_ready_seconds_bucket{mode="claim"}[15m])))

Running sessions per namespace:

sum by (namespace) (maqpna_sessions{phase="Running"})

Components running a different version from the gateway:

count by (version) (maqpna_build_info)

Check it worked#

Port-forward to the gateway and read its metrics. Use the admin port instead of 8080 when adminListen is set.

kubectl -n maqpna-system port-forward svc/maqpna-gateway 8080:8080
curl -s http://127.0.0.1:8080/metrics | grep '^maqpna_build_info'

In Prometheus, up{job=~".*maqpna.*"} should be 1 for the operator, the gateway and (when enabled) the attestation service.

What can go wrong#

Next steps#

• Monitoring and alerts: turn on scraping, alerts and dashboards, and the runbook for each alert.
• Observability: how metrics, traces and the audit ledger fit together.
• Ports and endpoints: which port serves /metrics on each component.
• Gateway configuration: adminListen, otel and the other keys that change what is exported.
• Helm values: every monitoring.* and serviceMonitor.* value.