Use Model Gateway Tools with the Responses API

The Model Gateway's Responses API lets a model use gateway-hosted RAG and MCP tools during a response. The gateway executes those tools, returns their results to the model, and repeats until the model produces a final answer. Your application receives the complete tool-calling trace and answer from one request to /v1/responses.

This guide uses a non-streaming request so the complete trace is easy to inspect.

For ordinary generation, streaming, and conversation continuation, start with the Responses API guide.

Prerequisites

Set your API base URL and API key:

export API_BASE_URL="https://api.<your-domain>"
export API_KEY="cm_api_…"

The API key needs invoke access to a chat model that supports function calling and access to the RAG or MCP tools you want to use. The raw HTTP examples use curl.

The examples verify TLS certificates. Add -k to curl only when testing an environment with a self-signed certificate.

This guide covers RAG and MCP tools that do not need an end-user sign-in. To call a tool from a service each user signs in to, send the user's access token with the request — see Call tools on the MCP endpoint.

1. Discover available tools

The Model Gateway exposes an MCP endpoint at /mcp/. Call tools/list to see the tools available to your API key:

curl -fsS -X POST "${API_BASE_URL}/mcp/" \
  -H "Authorization: Bearer ${API_KEY}" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "tools/list",
    "params": {}
  }'

The endpoint can return plain JSON or a Server-Sent Event whose data: line contains the JSON-RPC response. In both cases, the available definitions are in result.tools:

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "tools": [
      {
        "name": "YOUR_RAG__search_documents",
        "description": "Search the relevant organizational documents.",
        "inputSchema": {
          "type": "object",
          "properties": {
            "query": {
              "type": "string"
            }
          },
          "required": ["query"]
        }
      }
    ]
  }
}

Each tool includes:

• name — the name published by the MCP endpoint
• description — when the model should use the tool
• inputSchema — the arguments accepted by the tool

To use a published tool as a /v1/responses function, its name must match ^[A-Za-z0-9_-]{1,64}$.

The list is filtered by the caller's permissions. An empty array means no tool definitions were returned. This can be caused by missing access, a tool that is not ready or reachable, or missing credentials for an OAuth-backed MCP tool.

Some MCP servers need their own OAuth authorization from the end user. The tools of such a server are left out of tools until it is authorized. The response then lists the server in result._meta, under the key confidentialmind.com/mcp-auth-required:

{
  "_meta": {
    "confidentialmind.com/mcp-auth-required": [
      {
        "type": "mcp_auth_required",
        "code": "mcp_auth_required",
        "server": "github",
        "auth_type": "oauth_authorization_code",
        "reason": "missing_token"
      }
    ]
  }
}

Each entry describes one server:

• server — the name of the MCP server that needs authorization
• auth_type — the authorization the server expects, oauth_authorization_code
• reason — missing_token when the request carried no token for that server, unsupported_token_type when the token is not a bearer token, and rejected_token when the server refused the token

The key is absent when no server needs authorization. Tools from all other servers are returned as usual, so use this list to show the user which servers are still waiting to be authorized.

2. Choose a model

List the chat models available to the key:

curl -sS "${API_BASE_URL}/v1/models" \
  -H "Authorization: Bearer ${API_KEY}"

Choose an id from the response's data array. Listing a chat model does not guarantee that it supports function calling. Select a model configured for function calling and a compatible tool from the previous steps:

export MODEL_ID="YOUR_MODEL_ID"
export TOOL_NAME="YOUR_RAG__search_documents"

3. Call the Responses API

Declare the gateway-hosted tool as a function in the request. Copy its description from tools/list, and map its inputSchema to the Responses API's parameters field. tool_choice defaults to auto; use none to disable tool calls. Additional choices depend on the selected model — see tool-choice support.

curl -sS --max-time 90 "${API_BASE_URL}/v1/responses" \
  -H "Authorization: Bearer ${API_KEY}" \
  -H "Content-Type: application/json" \
  --data @- <<JSON
{
  "model": "${MODEL_ID}",
  "input": "Answer from the documents: What are the main conclusions?",
  "stream": false,
  "store": false,
  "max_output_tokens": 2048,
  "tools": [
    {
      "type": "function",
      "name": "${TOOL_NAME}",
      "description": "Search the relevant organizational documents.",
      "parameters": {
        "type": "object",
        "properties": {
          "query": {
            "type": "string"
          }
        },
        "required": ["query"]
      }
    }
  ]
}
JSON

For a gateway-hosted tool, the response's output array contains the ordered trace:

1. function_call — the tool selected by the model and its arguments
2. function_call_output — the result returned by the gateway-hosted tool
3. message — the model's final answer after it has used the result

The gateway handles this loop inside the original HTTP request. If the model emits several gateway-hosted tool calls in one turn, the gateway executes them concurrently and preserves their planned order in the trace.

Only gateway-hosted tools are executed this way. A function that is not registered with the gateway is returned as a function_call for the client to handle.

Tool access is decided for each request

Before each model turn, the Model Gateway determines which tools are callable for the current request. For a direct model request, the set starts with the tool definitions supplied in that request. For a preset-agent request, it starts with the tools configured for the agent. Gateway-hosted tools must also pass current discovery and access checks, and tool_choice can narrow the set further. The Model Gateway includes the resulting tool names in the instructions sent to the model.

A tool call or tool result included in input is historical context only. It does not add that tool to the current request or authorize the Model Gateway to execute it. The current callable-tool list gives the model an authoritative view of its access for that turn.

Setting "tool_choice": "none" makes the current callable-tool list empty even when the request contains tool definitions. The model is therefore instructed that it cannot call any tool for that turn.

Complete a client-owned function call

Declare a client-owned function in tools using the same shape as above. If the response ends with its function_call, run the function and send its output back with the prior items. Keep the same tools value in the next request:

import json
import os

prompt = "Look up the weather in Helsinki."
first = client.responses.create(
    model=os.environ["MODEL_ID"],
    input=prompt,
    tools=tools,
    store=False,
)

call = next(item for item in first.output if item.type == "function_call")
result = run_your_function(call.name, json.loads(call.arguments))

input_items = [
    {"role": "user", "content": prompt},
    *(item.model_dump() for item in first.output),
    {
        "type": "function_call_output",
        "call_id": call.call_id,
        "output": json.dumps(result),
    },
]

second = client.responses.create(
    model=os.environ["MODEL_ID"],
    input=input_items,
    tools=tools,
    store=False,
)

print(second.output_text)

Keep store: false because the application resends the complete history in input.

Use the published tool definitions

The example above declares one tool by hand. In an application that should automatically use every available tool, call tools/list at runtime and convert each compatible definition before sending the Responses request:

MCP tools/list field	Responses API field
name	name
description	description
inputSchema	parameters

Add "type": "function" to each converted definition. Include only names matching ^[A-Za-z0-9_-]{1,64}$, then place the resulting array in the Responses request's tools field. Perform this transformation with the JSON facilities in your application's language rather than shell text processing.

If discovery succeeds with an empty list, the request sends "tools": [] and the model cannot call a gateway-hosted tool. Published tools with incompatible names are skipped. They remain available to MCP clients but cannot be copied verbatim into a Responses API function declaration.

Clear tool descriptions improve routing. Ask the model for document evidence when retrieval is required. Otherwise, the model may answer from its own knowledge without calling a tool.

Troubleshooting

Symptom	What to check
tools/list returns an empty array	Check access, readiness, reachability, and OAuth credentials.
A tool from an OAuth-backed MCP server is missing from tools	Read result._meta under confidentialmind.com/mcp-auth-required. It names the server and the reason: no token was sent, the token type is not supported, or the server rejected the token.
/v1/responses returns 403	The API key needs invoke access to the selected model.
/v1/responses fails with a normalized model-service message	See Model backend errors on the Responses API for the public meaning. Ask your administrator to inspect the Model Gateway diagnostics for the exact provider reason.
/v1/responses rejects a tool name	Use a name matching ^[A-Za-z0-9_-]{1,64}$.
The response contains no tool call	Confirm function-calling configuration, then explicitly ask for evidence.
The response ends with function_call	Confirm the function name belongs to a tool published by the gateway.
mcp_auth_required result	Send the user's token; see MCP tool calls.
/mcp/ returns the batch error	Send one JSON-RPC message per request when using mcp_authorizations.