Trace agents with OpenTelemetry GenAI

Export gateway spans and metrics that follow the OpenTelemetry GenAI semantic conventions (chat, execute_tool, invoke_agent), propagate W3C trace context to models, MCP servers and agents, and opt in to DLP-redacted prompt capture.

flowchart LR
  A[Agent<br/>traceparent] --> G[Gateway]
  G -- "chat {model}<br/>gen_ai.*" --> M[Model upstream]
  G -- "tools/call {tool}<br/>execute_tool" --> T[MCP server]
  G -- "invoke_agent {agent}" --> P[Agent or A2A peer]
  G -- OTLP/HTTP JSON --> C[(Collector)]
  G -- "/metrics<br/>gen_ai_client_*" --> PR[(Prometheus)]

Goal#

See every model call, tool call and agent-to-agent call of an agent as one trace, with the attribute names that Grafana, Datadog, Langfuse, Arize Phoenix and other GenAI backends recognise, without exporting prompts unless you opt in.

The GenAI semantic conventions are still in Development status. MAQPNA tracks the latest experimental names (gen_ai.provider.name, not gen_ai.system), which is what OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental selects in the OpenTelemetry SDKs.

Prerequisites#

• A collector with an OTLP/HTTP receiver (see Observability).
• gateway.tracing.enabled: true and gateway.tracing.endpoint in the chart values.

Steps#

1. Turn tracing on#

gateway:
  tracing:
    enabled: true
    endpoint: http://otel-collector.observability.svc:4318
    genAI:
      captureContent: false     # the default: no prompts, completions or arguments
      emitLegacySystem: false   # true: also set gen_ai.system for older backends

With tracing off, the gateway still forwards the caller's W3C traceparent, tracestate and baggage to every upstream; it creates no spans of its own.

2. Read the spans#

gen_ai.agent.id is <namespace>/<agent>; gen_ai.conversation.id is the MAQPNA session. gen_ai.provider.name comes from the route's upstream host (openai, anthropic, azure.ai.openai, gcp.vertex_ai, aws.bedrock, mistral_ai, cohere); self-hosted OpenAI-compatible servers report openai. Governance attributes (maqpna.decision, maqpna.policy, maqpna.dlp.hits, ...) stay on the same spans.

The gateway injects its own span as the parent into the upstream's traceparent header (and MCP params._meta.traceparent), so model servers, MCP servers and called agents continue the trace.

3. Read the metrics#

The gateway's /metrics exposes the GenAI client metrics in Prometheus form:

gen_ai_client_token_usage_bucket{gen_ai_operation_name="chat",gen_ai_provider_name="openai",maqpna_model_route="coder-llm",gen_ai_token_type="input",le="1024"} 12
gen_ai_client_operation_duration_seconds_count{gen_ai_operation_name="chat",gen_ai_provider_name="openai",gen_ai_request_model="qwen3-coder",gen_ai_response_model="qwen3-coder",error_type=""} 12
gen_ai_client_operation_duration_seconds_count{gen_ai_operation_name="execute_tool",gen_ai_provider_name="",gen_ai_request_model="",gen_ai_response_model="",error_type=""} 40

Bucket boundaries are the conventions' recommended ones. gen_ai.client.operation.duration covers model calls, execute_tool and invoke_agent; failed calls carry error_type.

4. Opt in to content capture (optional)#

gateway:
  tracing:
    genAI:
      captureContent: true
      contentDlpProfile: ""     # "" = every built-in DLP detector
      maxContentBytes: 16384

This adds the opt-in attributes gen_ai.input.messages and gen_ai.output.messages (chat and text completions, including streamed completions), gen_ai.tool.call.arguments (MCP tools/call) and gen_ai.input.messages (A2A messages), as JSON in the conventions' message shape (role, parts, finish_reason).

Captured content always passes DLP first:

• The telemetry scanner uses contentDlpProfile (a dlpProfiles entry) or, when empty, every built-in detector including email, phone_e164, ipv4_private and high_entropy_secret.
• Its actions are forced to redact, so every finding becomes [REDACTED:<detector>] even if the profile only audits traffic.
• Content that is larger than the profile's maxScanBytes, or larger than maxContentBytes once encoded, is dropped, not truncated. The span then carries maqpna.genai.content_dropped (dlp_denied, oversize or unparseable).
• Prompts are captured after the gateway's own DLP and guards, so telemetry never sees more than the model received. Redaction for telemetry never changes the traffic itself.
• Images, audio and files are recorded by part type only.

Verify#

Scrape the gateway's /metrics (on the admin listener when adminListen is set) and look for gen_ai_client_operation_duration_seconds_count.

In your trace backend, search for gen_ai.operation.name = execute_tool or invoke_agent.

Limits#

• No create_agent spans: agents are created by the operator from Agent resources, not through the gateway.
• gen_ai.tool.call.result is not captured; MCP tool results are DLP-scanned in flight but not recorded.
• The newer gen_ai.client.inference.* counters are not yet exported; gen_ai.client.token.usage is.
• Metrics are Prometheus text only, not OTLP metrics.

Next steps#

• Observability
• Protect data with DLP profiles