Call models through the gateway

Send model calls to the gateway's /llm route with the OpenAI or Anthropic wire format, point existing SDKs at it, and know which budgets, token limits and DLP checks apply.

A model call is one request from a session to a model through the gateway's model route, /llm/{route}. Your agent never learns the real model URL or API key. It uses the session token as its API key and the gateway's route as its base URL, so the official OpenAI and Anthropic SDKs work with two settings changed. The gateway forces the route's model, meters tokens against budgets, runs data loss prevention (DLP) on the prompt and the completion, and writes one audit record per call. Prompts are not stored in the ledger, only a hash of the request body.

flowchart LR
    A["Agent<br/>OpenAI or Anthropic SDK"] -->|"base URL {gateway}/llm/route/...<br/>api key = session token"| G["Gateway /llm"]
    G --> C{"scope models:route,<br/>budgets, token limits,<br/>DLP on prompt"}
    C -->|denied| E["OpenAI-style error<br/>401 / 402 / 403 / 429"]
    C -->|allowed| P["Provider adapter<br/>openai, azure-openai, bedrock,<br/>vertex, gemini, anthropic, oci-genai"]
    P --> M["Model endpoint<br/>(primary, then fallbacks)"]
    M --> D["DLP on the completion,<br/>meter tokens, audit"]
    D --> A

Which wire formats the gateway accepts#

Your agent speaks one of two formats to the gateway. The route's provider adapter translates to whatever the upstream needs (internal/llmwire), so the agent's code does not change when the platform team moves the route from vLLM to Bedrock.

The Anthropic paths work only on routes whose adapter is anthropic or bedrock (Agent.spec.model.adapter). On another route the gateway answers 400 invalid_request_error with unsupported_by_provider. Upstream adapters and their settings are in Connect model providers.

Step 1: Get a model route#

maqpna dev run exports the model variables for route stub when it exists, otherwise for the only --model route. The token's models:* scope (the local default) or models:<route> (in a cluster, granted automatically) lets the session use it.

Step 2: Make a call#

from openai import OpenAI
from maqpna import MaqpnaClient

client = MaqpnaClient.from_env()
llm = OpenAI(base_url=client.model_endpoint(), api_key=client.token())
reply = llm.chat.completions.create(
    model=client.model_name() or "default",      # the gateway forces the route's model anyway
    messages=[{"role": "user", "content": "Summarise issue 4821"}],
    extra_headers=client.openai_headers(),        # fresh token per request; it rotates in place
)
print(reply.choices[0].message)

import OpenAI from "openai";
import { MaqpnaClient } from "@kosa8/maqpna";

const client = MaqpnaClient.fromEnv();
const llm = new OpenAI({ baseURL: client.modelEndpoint(), apiKey: await client.token() });
const reply = await llm.chat.completions.create(
  { model: client.modelName() ?? "default", messages: [{ role: "user", content: "Summarise issue 4821" }] },
  { headers: await client.openaiHeaders() },
);
console.log(reply.choices[0].message);

hdr, err := client.OpenAIHeaders(ctx) // Authorization: Bearer <current token> + traceparent
if err != nil {
    return err
}
body := `{"model":"default","messages":[{"role":"user","content":"Summarise issue 4821"}]}`
req, _ := http.NewRequestWithContext(ctx, http.MethodPost,
    client.ModelBaseURL("")+"/chat/completions", strings.NewReader(body))
req.Header = hdr
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)

curl -sS "$MAQPNA_MODEL_ENDPOINT/chat/completions" \
  -H "Authorization: Bearer $MAQPNA_TOKEN" -H 'Content-Type: application/json' \
  -d '{"model":"stub","messages":[{"role":"user","content":"hi"}]}'

{"choices":[{"finish_reason":"tool_calls","index":0,"message":{"content":null,"role":"assistant","tool_calls":[{"function":{"arguments":"{\"text\":\"hello from stub-llm\"}","name":"echo"},"id":"call_chatcmpl-stub-1_0","type":"function"}]}}],"created":1792359118,"id":"chatcmpl-stub-1","model":"stub","object":"chat.completion","usage":{"completion_tokens":44,"prompt_tokens":8,"total_tokens":52}}

Without a route, model_endpoint() / modelEndpoint() raise ConfigError. They never fall back to a public API. Locally you can also pass the route name: client.model_endpoint("stub").

Point an existing SDK at the gateway#

Agents that already use an SDK need two settings, no code change:

# OpenAI SDK: reads OPENAI_BASE_URL and OPENAI_API_KEY (maqpna dev run sets both with --stub-llm)
export OPENAI_BASE_URL="$MAQPNA_MODEL_ENDPOINT"
export OPENAI_API_KEY="$(cat "$MAQPNA_TOKEN_FILE")"

# Anthropic SDK and Claude Agent SDK, on a route with adapter anthropic or bedrock
export ANTHROPIC_BASE_URL="$MAQPNA_ANTHROPIC_BASE_URL"
export ANTHROPIC_API_KEY="$(cat "$MAQPNA_TOKEN_FILE")"

import Anthropic from "@anthropic-ai/sdk";

// On a route with adapter anthropic or bedrock; the SDK sends the key as x-api-key,
// which the gateway accepts as the session token.
const anthropic = new Anthropic({ baseURL: process.env.MAQPNA_ANTHROPIC_BASE_URL, apiKey: await client.token() });

An API key exported once from the file goes stale when the operator refreshes the token (at most every 15 minutes in a cluster). For long-running agents, build the client per task, or pass the SDK's per-request headers (client.openai_headers(), await client.openaiHeaders(), client.OpenAIHeaders(ctx)).

Streaming (stream=True, stream: true) works on chat completions, completions, responses and Anthropic messages. The gateway relays the server-sent events, runs DLP on the stream and always asks the upstream for usage, so streamed calls are metered.

What applies to model calls#

Errors are OpenAI-shaped (Anthropic-shaped on /anthropic/ paths), with MAQPNA's reason and domain, so the OpenAI SDKs raise their normal APIStatusError subclasses:

{"error":{"code":"budget_exceeded","domain":"maqpna.com","message":"namespace budget (day) exhausted for team-a: spent 200.000000 of 200.000000 USD","reason":"budget_exceeded","type":"maqpna_budget_exceeded"}}

How to react to each in code is in Handle decisions. The full list is in Error and reason codes.

Check it worked#

maqpna dev timeline --last

TIME      KIND        SERVER/TOOL                   DECISION  DETAIL
17:31:58  model_call  llm/stub/chat.completions     allow     scope:models:stub; model:stub; usage:prompt=8,completion=44

Each model call is one row with its token usage. maqpna costs summary --scope agent --window day totals spend and tokens per agent; maqpna session cost SESSION shows one session.

What can go wrong#

Next steps#

• Handle decisions in code
• Budgets and cost limits
• Model call governance, the full pipeline
• Trace agents with OpenTelemetry GenAI