OpenTelemetry traces and metrics

Export gateway spans over OTLP/HTTP to an OpenTelemetry Collector, Jaeger or any GenAI-aware backend, link traces to audit records, and scrape the gateway's Prometheus metrics.

The MAQPNA gateway sees every tool call, model call and agent-to-agent (A2A) call. With tracing on, it exports a span for each of them over OTLP/HTTP, using the OpenTelemetry GenAI and MCP semantic conventions. Each audit record carries the same trace and span ID, so you can jump from a trace to the ledger and back.

By default spans never contain prompts, completions or tool arguments. Content capture is an opt-in, and captured content always passes DLP (data loss prevention) redaction first. The interop suite sends gateway spans through the OpenTelemetry Collector to Jaeger every night.

flowchart LR
    A["Agent<br/>traceparent"] --> G["MAQPNA gateway"]
    G -->|"tools/call"| T["MCP server"]
    G -->|"chat {model}"| M["Model upstream"]
    G -->|"invoke_agent"| P["Called agent"]
    G -->|"OTLP/HTTP JSON<br/>/v1/traces"| C["OTel Collector"]
    C --> J["Jaeger, Tempo,<br/>Datadog, Langfuse ..."]
    PR["Prometheus"] -->|"scrape /metrics"| G
    G --> L[("audit ledger<br/>ext.traceId")]

When to use it#

• You want one trace per agent task across the agent, the gateway, MCP servers and models.
• Your GenAI observability tool expects gen_ai.* attributes.
• You need to find the audit record behind a slow or failed span.

This page covers setup. For the full span and attribute list, metrics and content capture, see Trace agents with OpenTelemetry GenAI.

Prerequisites#

• An OpenTelemetry Collector (or any backend) with an OTLP/HTTP receiver, usually on port 4318. The gateway sends only http/json, not gRPC.
• MAQPNA installed with the Helm chart.
• With sovereignty.enabled, the collector URL must be in sovereignty.telemetryEndpoints.

Step 1: Run a collector#

A minimal pipeline that receives OTLP/HTTP and sends traces to Jaeger:

receivers:
  otlp:
    protocols:
      http: {endpoint: 0.0.0.0:4318}
exporters:
  otlp/jaeger:
    endpoint: jaeger.observability.svc:4317
    tls: {insecure: true}       # use TLS outside a lab
service:
  pipelines:
    traces: {receivers: [otlp], exporters: [otlp/jaeger]}

Step 2: Turn tracing on#

gateway:
  tracing:
    enabled: true
    endpoint: http://otel-collector.observability.svc:4318   # /v1/traces is appended
    serviceName: maqpna-gateway
    sampleRatio: 1.0         # root spans only; a caller's sampled flag wins
    recordArgs: false        # true: maqpna.args_sha256 and maqpna.args_bytes (never contents)
    recordUser: false        # true: enduser.id (the delegating user)
    resourceAttributes: {}   # extra resource attributes, e.g. deployment.environment
    headersSecret: ""        # Secret with key "headers" ("Name: value" per line) for collector auth
    genAI:
      captureContent: false  # true: DLP-redacted prompts, completions and tool arguments
      emitLegacySystem: false

sovereignty:
  telemetryEndpoints:        # required when sovereignty.enabled
    - http://otel-collector.observability.svc:4318

The chart refuses to render:

• gateway.tracing.enabled without gateway.tracing.endpoint;
• with sovereignty.enabled, an endpoint that is not listed in sovereignty.telemetryEndpoints.

If your collector needs authentication, create the headers Secret:

kubectl create secret generic maqpna-otel-headers -n maqpna-system \
  --from-literal=headers='Authorization: Bearer <your-collector-token>'

Then set gateway.tracing.headersSecret: maqpna-otel-headers. It is mounted at /var/run/maqpna-otel/headers.

Step 3: Send trace context from the agent#

The gateway continues the agent's trace when the request carries a W3C traceparent header, or params._meta.traceparent on MCP calls. The MAQPNA SDKs send it; the Go SDK also reads TRACEPARENT and TRACESTATE from the environment. Without a caller context, the gateway starts a root span, sampled by sampleRatio.

The gateway injects its own span as the parent into the upstream's traceparent (and MCP params._meta.traceparent), so MCP servers, model servers and called agents can continue the trace.

Step 4: Read the trace#

For one tool call tools/call echo on server everything, the trace contains:

gen_ai.conversation.id is the MAQPNA session. gen_ai.agent.id is <namespace>/<agent>.

Step 5: Scrape the metrics#

The gateway serves Prometheus text on /metrics (on the admin listener when one is set). With serviceMonitor.enabled: true the chart creates a ServiceMonitor for the Prometheus Operator. Metrics are Prometheus text only, not OTLP metrics.

See Monitoring, alerts and runbooks for the alerts and dashboards the chart ships.

Verify#

1. Make a governed call through the gateway.
2. In your trace backend, search for service maqpna-gateway and gen_ai.operation.name = execute_tool.

3. Copy the trace ID and find the matching audit records:

   maqpna audit tail --trace <trace-id> --since 1h
   

4. Check that maqpna_gateway_otel_spans_exported_total grows and ..._failed_total stays at 0.

Troubleshooting#

Next steps#

• Trace agents with OpenTelemetry GenAI
• Observability
• Monitoring, alerts and runbooks
• SIEM and audit forwarding
• Metrics reference