Connect model providers#

An agent always talks to the gateway in one of two wire formats: the OpenAI API on {gateway}/llm/<route>/v1 (chat completions, completions, embeddings, models and the Responses API), or the native Anthropic Messages API on {gateway}/llm/<route>/anthropic. Each endpoint of a model route names a provider adapter. The adapter turns the governed request into the provider's own request, signs it with the gateway's credential, and turns the answer back into the agent's wire format.

flowchart LR
    A["Agent<br/>OpenAI or Anthropic SDK"] -->|session token| G["Gateway /llm<br/>scope, budgets, DLP, guards"]
    G --> AD{"endpoint adapter"}
    AD -->|SigV4| B["Bedrock Converse"]
    AD -->|OAuth bearer| V["Vertex AI generateContent"]
    AD -->|Entra or api-key| Z["Azure OpenAI v1"]
    AD -->|x-api-key| C["Anthropic Messages"]
    AD -->|Bearer| O["OpenAI-compatible<br/>vLLM, OCI, STACKIT, IONOS, ..."]

Governance runs before the adapter and after it, on the agent's format. DLP, injection guards, budgets and the audit record therefore see the same text and the same token usage whatever the provider. That includes streamed deltas and the provider's prompt-cache counts. The agent never sees a provider URL, key or account detail.

Choose an adapter#

converse, native (Vertex AI and Gemini) and anthropic translate OpenAI chat completions: messages, system prompts, images sent as base64 data: URLs (or https URLs where the provider fetches them; other schemes such as gs://, which a provider would read with the gateway's cloud identity, are refused), tools and tool results, tool choice, stop sequences, streaming, finish reasons and usage. They answer GET /models locally with the route's model and aliases. Other OpenAI operations (completions, embeddings, Responses) need an OpenAI-compatible mode and are refused with 400 unsupported_by_provider before any upstream call. The OpenAI-compatible adapters (openai, azure-openai, oci-genai, and the openai modes of bedrock and vertex) forward every operation unchanged apart from path and authentication.

Configure a route#

On Kubernetes, set the adapter on the Agent. spec.model.provider stays an informational label; spec.model.adapter selects the wire format:

apiVersion: maqpna.com/v1alpha1
kind: Agent
metadata: {name: reviewer, namespace: team-a}
spec:
  model:
    adapter: bedrock
    endpoint: https://bedrock-runtime.eu-central-1.amazonaws.com
    region: eu-central-1
    name: smart                      # what the agent asks for
    aliases:
      smart: anthropic.claude-sonnet-5-5-v1:0
      fast: anthropic.claude-haiku-5-5-v1:0
    fallbacks:
      - adapter: vertex
        endpoint: https://europe-west3-aiplatform.googleapis.com
        region: europe-west3
        project: acme-eu
        name: gemini-2.5-pro

The operator renders the route into models.json. Without Kubernetes, write the same route into the gateway's models (or modelsFile):

{"models": {"reviewer": {
  "provider": "bedrock", "region": "eu-central-1", "model": "smart",
  "aliases": {"smart": "anthropic.claude-sonnet-5-5-v1:0"},
  "namespaces": ["team-a"],
  "fallbacks": [{"provider": "vertex", "region": "europe-west3", "project": "acme-eu",
                 "model": "gemini-2.5-pro", "upstream": "https://europe-west3-aiplatform.googleapis.com"}]
}}}

Route and fallback fields: provider, mode, region, project, apiVersion, auth, aliases, reasoning, plus the existing upstream, model, jurisdiction, requestsPerMinute and apiKeyFile. Fallbacks inherit only model, never the provider settings or the key. In a models file, a route with an invalid provider block is skipped and logged while the others keep serving; in the gateway configuration it is a start-up error.

Amazon Bedrock#

The converse mode calls POST /model/<modelId>/converse (or /converse-stream, whose binary event stream the gateway decodes) and signs it with SigV4 for service bedrock. Model IDs can be in-region IDs, cross-region inference profiles (eu., us., apac., global.) or profile ARNs:

{"provider": "bedrock", "region": "eu-central-1", "model": "eu.anthropic.claude-sonnet-5-5-v1:0",
 "auth": {"accessKeyIdFile": "/var/run/aws/id", "secretAccessKeyFile": "/var/run/aws/secret"}}

Without auth the gateway uses its AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN. With apiKeyFile it sends a Bedrock API key as Bearer instead of signing. Converse usage, including cacheReadInputTokens and cacheWriteInputTokens, is metered exactly. A guardrail_intervened stop becomes finish_reason: content_filter.

"mode": "openai" uses Bedrock's OpenAI-compatible endpoint (/openai/v1 on bedrock-runtime, or a Mantle URL you set as upstream, such as https://bedrock-mantle.eu-central-1.api.aws/v1).

Bedrock Guardrails can also screen prompts, tool arguments and tool results for any route, as a guard:

{"guards": [{"name": "bedrock", "type": "bedrock-guardrail", "on": ["llmPrompt", "toolResult"], "action": "block",
  "url": "https://bedrock-runtime.eu-central-1.amazonaws.com/guardrail/gr-abc123/version/1/apply"}]}

Google Vertex AI and the Gemini API#

The vertex adapter calls generateContent / streamGenerateContent?alt=sse of projects/<project>/locations/<region>/publishers/google/models/<model> with a Google OAuth access token. Tool calls round-trip, including Gemini's thought signatures, which travel inside the tool-call ID. Usage maps promptTokenCount, cachedContentTokenCount, candidatesTokenCount and thoughtsTokenCount (billed as output). Safety blocks become finish_reason: content_filter.

{"provider": "vertex", "project": "acme-eu", "region": "europe-west3", "model": "gemini-2.5-flash",
 "auth": {"tokenFile": "/var/run/gcp/access-token"}}

auth.tokenFile is re-read on every call, so a sidecar can keep it fresh. Without it the gateway reads GOOGLE_OAUTH_ACCESS_TOKEN. "mode": "openai" uses Vertex AI's OpenAI-compatible endpoints/openapi path, with model names such as google/gemini-2.5-flash. For the Gemini API (AI Studio), use "provider": "gemini" with apiKeyFile.

Azure OpenAI and Microsoft Foundry#

The default v1 mode uses https://<resource>.openai.azure.com/openai/v1/ (or a Foundry services.ai.azure.com resource), which needs no api-version:

{"provider": "azure-openai", "upstream": "https://acme-eu.openai.azure.com", "model": "gpt-5-mini",
 "auth": {"tokenFile": "/var/run/azure/access-token"}}

The Entra token needs the scope https://cognitiveservices.azure.com/.default and the caller role Cognitive Services OpenAI User. With apiKeyFile the key goes in the api-key header. "mode": "deployments" uses the legacy /openai/deployments/<model>/...?api-version= paths (apiVersion, default 2024-10-21); model is then the deployment name.

Anthropic#

"provider": "anthropic" translates OpenAI chat completions into the Messages API (x-api-key, anthropic-version: 2023-06-01). Its modes reach Claude directly, on Vertex AI ("mode": "vertex", :rawPredict / :streamRawPredict, Google token) or on Bedrock ("mode": "bedrock", InvokeModel, SigV4). Requests without max_tokens get 4096, or the agent's maxTokensPerRequest.

{"provider": "anthropic", "model": "claude-sonnet-5-5", "apiKeyFile": "/var/run/maqpna-model-credentials/team-a.claude.key"}

Native Anthropic SDK and Claude Agent SDK clients#

Routes with provider anthropic or bedrock also accept native Messages requests on {gateway}/llm/<route>/anthropic/v1/messages (and /v1/messages/count_tokens). Anthropic SDK and Claude Agent SDK clients work unmodified:

export ANTHROPIC_BASE_URL="$MAQPNA_ANTHROPIC_BASE_URL"     # {gateway}/llm/<route>/anthropic
export ANTHROPIC_AUTH_TOKEN="$(cat "$MAQPNA_TOKEN_FILE")"   # or ANTHROPIC_API_KEY: x-api-key works too

The operator sets MAQPNA_ANTHROPIC_BASE_URL in sandboxes of such agents. These calls are governed on the native shape, with the same scope, budgets, token limits, DLP (prompt blocks, streamed text_delta, input_json_delta and thinking_delta with the carry-over window, and every other event, citations_delta included, scanned whole; only signature_delta passes as opaque), guards on user, tool_result and document content, and audit as the OpenAI path. A stream that ends before its final message_delta usage is metered on the text it delivered. Requests that ask the provider to call MCP servers itself (Anthropic mcp_servers or mcp_toolset tools, OpenAI Responses mcp tools) are refused with 403 provider_hosted_mcp on every wire: those tool calls would never pass the gateway's MCP governance. Register the server with MAQPNA and call it through /mcp instead. Provider-hosted server tools are refused the same way (403 provider_hosted_tool) unless the route allows them: Anthropic web_search, web_fetch and code_execution, OpenAI Responses web_search, file_search, code_interpreter, computer_use and image_generation (and chat web_search_options), Gemini google_search, url_context and code_execution, and Bedrock system tools (system_tool) search, fetch and run code at the provider, where the gateway's egress policy, DLP and audit never see each call. List the ones a route may use in providerTools (route JSON) or spec.model.providerTools (Agent), by canonical name or versioned type such as web_search_20250305. On Kubernetes the platform caps them: the Agent's TrustTier lists what its Agents may enable in spec.allowedProviderTools (empty or absent: none), and the operator renders the intersection of the two lists. A canonical tier entry (web_search) admits its versioned types; a versioned entry admits only itself. Entries the tier does not allow are dropped from the route, and the Agent reports them in its ProviderToolsAllowed=False condition and a ProviderToolsDropped Warning event.

apiVersion: maqpna.com/v1alpha1
kind: TrustTier
metadata: {name: research}
spec:
  # ...
  allowedProviderTools: [web_search]
---
# Agent in that tier
spec:
  tier: research
  model:
    adapter: anthropic
    providerTools: [web_search_20250305, web_fetch]   # web_fetch is dropped
``` The policy "tool" is `messages`. The route's model is forced, `anthropic-version` and `anthropic-beta` are forwarded, and errors use Anthropic's `{"type":"error",...}` shape. `count_tokens` is governed but not metered. OpenAI-compatible routes refuse native requests with `400`.

## OCI Generative AI

```json
{"provider": "oci-genai", "region": "eu-frankfurt-1", "model": "meta.llama-4-maverick",
 "apiKeyFile": "/var/run/maqpna-model-credentials/team-a.oci.key"}

The OCI OpenAI-compatible endpoint takes an OCI Generative AI API key as Bearer. Without apiKeyFile (or with "auth": {"type": "oci"}) the gateway signs each request with OCI IAM request signing, using OKE workload identity or a resource principal (MAQPNA_OCI_AUTH).

Sovereign OpenAI-compatible providers and model aliases#

STACKIT AI Model Serving, IONOS AI Model Hub, OVHcloud AI Endpoints, Scaleway Generative APIs, Mistral, and Sarvam models served by vLLM on an Indian GPU cloud are OpenAI-compatible. They need no adapter: provider stays openai. Use aliases so that agents ask for a role (fast, smart) and a provider retiring a model ID becomes a one-line route change. IONOS, for example, retires Llama 3.1 8B on 15 October 2026 and Mistral Nemo on 22 October 2026. The base URLs below were current when this was written; take the exact URL and model IDs from your provider console.

{"models": {
  "team-a.ionos": {"upstream": "https://openai.inference.de-txl.ionos.com/v1", "model": "fast", "jurisdiction": "EU-DE",
    "aliases": {"fast": "meta-llama/Llama-3.3-70B-Instruct"}, "apiKeyFile": "/var/run/keys/ionos"},
  "team-a.ovh": {"upstream": "https://oai.endpoints.kepler.ai.cloud.ovh.net/v1", "model": "fast", "jurisdiction": "EU-FR",
    "aliases": {"fast": "Mistral-Small-3.2-24B-Instruct-2506"}, "apiKeyFile": "/var/run/keys/ovh"},
  "team-a.scaleway": {"upstream": "https://api.scaleway.ai/v1", "model": "smart", "jurisdiction": "EU-FR",
    "aliases": {"smart": "mistral-small-3.2-24b-instruct-2506"}, "apiKeyFile": "/var/run/keys/scaleway"},
  "team-a.mistral": {"upstream": "https://api.mistral.ai/v1", "model": "smart", "jurisdiction": "EU-FR",
    "aliases": {"smart": "mistral-medium-latest"}, "apiKeyFile": "/var/run/keys/mistral"},
  "team-a.sarvam": {"upstream": "http://vllm.models.svc:8000/v1", "model": "indic", "jurisdiction": "IN",
    "aliases": {"indic": "sarvamai/sarvam-105b"}}
}}

For STACKIT, use the OpenAI-compatible URL and token shown for your project in the STACKIT portal. The audit record notes every resolution (model_alias:fast->...), and metering prices the resolved model ID.

OpenAI Responses API and reasoning models#

POST /llm/<route>/v1/responses is proxied and governed on OpenAI-compatible routes. DLP scans the whole request, including input and instructions, and guards classify the user and tool input. The gateway meters usage.input_tokens and output_tokens (with input_tokens_details.cached_tokens), and redacts streamed response.*.delta events with the carry-over window. The .done events and response.completed, which repeat the full text, are scanned whole. maxTokensPerRequest clamps max_output_tokens.

OpenAI reasoning models reject max_tokens. For models detected as reasoning models (OpenAI o1, o3 and o4 IDs and gpt-5*) or routes with "reasoning": true, the gateway sends max_completion_tokens. It renames a client's max_tokens and uses max_completion_tokens when it clamps or sets the limit. "reasoning": false turns detection off.

Credentials#

Every credential belongs to the gateway, never to the agent:

A missing or empty credential fails the attempt closed, and the next endpoint is tried. Without an explicit credential, aws-sigv4, gcp, azure-entra and oci use the gateway's own cloud workload identity: EKS Pod Identity or IRSA, GKE Workload Identity Federation, AKS Entra Workload ID, or OKE workload identity. These are the same keyless credentials the WORM sink and KMS signers use. Bind the gateway's ServiceAccount with gateway.serviceAccount.annotations and tune the providers with gateway.cloudCredentials (see Cloud identity & keys). Static bearer tokens still need tokenFile or tokenEnv.

The gateway's own cloud credentials (workload identity, GOOGLE_OAUTH_ACCESS_TOKEN, or the gateway's AWS_* variables) are only used for the provider's API domain or a loopback address: *.googleapis.com for gcp; *.openai.azure.com, *.cognitiveservices.azure.com or *.services.ai.azure.com (and their .us/.cn sovereign forms) for azure-entra; the partition DNS suffixes (amazonaws.com, amazonaws.com.cn, amazonaws.eu, the ISO partitions, gateway.cloudCredentials.aws.partitionDNSSuffix) and api.aws for aws-sigv4; and oraclecloud.com plus the realm domains in MAQPNA_OCI_REALM_DOMAINS (comma-separated, e.g. oraclegovcloud.com) for oci. Agent endpoints are chosen by namespace users, so a vertex, azure-openai, bedrock or oci-genai Agent pointing anywhere else fails closed instead of handing them the gateway's token or a signature carrying its AWS session token or OCI security token. For a private endpoint under another name (for example a Private Service Connect DNS name), an operator-managed route can set auth.type explicitly (gcp, azure-entra, aws-sigv4 or oci) or give explicit AWS credential files.

Residency#

Each endpoint's region gives the jurisdiction it processes data in: eu-central-1 and europe-west3 map to EU-DE, eu-west-3 to EU-FR, uk-london-1 to GB, ap-south-1 and asia-south1 to IN. Bedrock inference-profile prefixes widen it (eu. is EU, us. is US). The Vertex AI global location and global. profiles are GLOBAL. With a SovereigntyPolicy that constrains jurisdictions, the gateway skips endpoints outside them, as it skips hosts outside the egress allow-list. The call fails with 403 maqpna_egress_denied when no endpoint is left, and in enforce mode static routes fail at start-up. Endpoints without a known region use their jurisdiction label.

Prompt-cache pricing#

Provider usage includes prompt-cache reads and writes. Price them in the pricing table; they default to inputPer1kUSD:

{"models": {"anthropic.claude-sonnet-5-5-v1:0": {"inputPer1kUSD": 0.003, "outputPer1kUSD": 0.015,
  "cacheReadPer1kUSD": 0.0003, "cacheWritePer1kUSD": 0.00375}}}

The audit reason carries usage:prompt=N,completion=M,cache_read=R,cache_write=W.

Errors#

Provider errors keep the OpenAI shape (Anthropic's on the native path). The message is the provider's, with ARNs, account IDs, project paths, URLs and keys redacted, and the code is upstream_<provider error>, for example upstream_validation_exception. A provider refusing the gateway's credential (401 or 403) answers 502 upstream_auth_failed and fails over. Throttling (429) and 5xx answers fail over as for any endpoint. Other 4xx answers are returned without retry. A request the adapter cannot express answers 400 unsupported_by_provider.

Check a provider for real#

The provider tests run against recorded API shapes. To exercise real accounts, run the opt-in live tests with credentials in the environment. Each provider is skipped unless its variables are set (see cmd/maqpna-gateway/llm_live_test.go):

ANTHROPIC_API_KEY=... MAQPNA_LIVE_BEDROCK_REGION=eu-central-1 MAQPNA_LIVE_BEDROCK_MODEL=eu.anthropic.claude-haiku-5-5-v1:0 \
  go test -tags liveproviders -run TestLive ./cmd/maqpna-gateway/

Related#

• Model call governance
• Budgets and cost limits
• Protect data with DLP profiles
• Sovereignty, confidential tiers and customer-held keys
• Use a framework template