Use tools with the Responses API
The Model Gateway's Responses API lets a model use gateway-hosted RAG and MCP tools during a response.
The gateway executes those tools, returns their results to the model, and repeats until the model produces a final
answer. Your application receives the complete tool-calling trace and answer from one request to /v1/responses.
This guide uses a non-streaming request so the complete trace is easy to inspect.
For ordinary generation, streaming, and conversation continuation, start with the Responses API guide.
Prerequisites
Set your API base URL and API key:
export API_BASE_URL="https://api.<your-domain>"
export API_KEY="cm_api_…"
The API key needs invoke access to a chat model that supports function calling and access to the RAG or MCP tools
you want to use. The raw HTTP examples use curl.
The examples verify TLS certificates. Add
-ktocurlonly when testing an environment with a self-signed certificate.
This guide covers RAG and MCP tools that do not need an end-user sign-in. To call a tool from a service each user signs in to, send the user's access token with the request — see Call tools on the MCP endpoint.
1. Discover available tools
The Model Gateway exposes an MCP endpoint at /mcp/. Call tools/list to see the tools available to your API key:
curl -fsS -X POST "${API_BASE_URL}/mcp/" \
-H "Authorization: Bearer ${API_KEY}" \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-d '{
"jsonrpc": "2.0",
"id": 1,
"method": "tools/list",
"params": {}
}'
The endpoint can return plain JSON or a Server-Sent Event whose data: line contains the JSON-RPC response. In both
cases, the available definitions are in result.tools:
{
"jsonrpc": "2.0",
"id": 1,
"result": {
"tools": [
{
"name": "YOUR_RAG__search_documents",
"description": "Search the relevant organizational documents.",
"inputSchema": {
"type": "object",
"properties": {
"query": {
"type": "string"
}
},
"required": ["query"]
}
}
]
}
}
Each tool includes:
name— the name published by the MCP endpointdescription— when the model should use the toolinputSchema— the arguments accepted by the tool
To use a published tool as a /v1/responses function, its name must match ^[A-Za-z0-9_-]{1,64}$.
The list is filtered by the caller's permissions. An empty array means no tool definitions were returned. This can be caused by missing access, a tool that is not ready or reachable, or missing credentials for an OAuth-backed MCP tool.
Some MCP servers need their own OAuth authorization from the end user. The tools of such a server are left out of
tools until it is authorized. The response then lists the server in result._meta, under the key
confidentialmind.com/mcp-auth-required:
{
"_meta": {
"confidentialmind.com/mcp-auth-required": [
{
"type": "mcp_auth_required",
"code": "mcp_auth_required",
"server": "github",
"auth_type": "oauth_authorization_code",
"reason": "missing_token"
}
]
}
}
Each entry describes one server:
server— the name of the MCP server that needs authorizationauth_type— the authorization the server expects,oauth_authorization_codereason—missing_tokenwhen the request carried no token for that server,unsupported_token_typewhen the token is not a bearer token, andrejected_tokenwhen the server refused the token
The key is absent when no server needs authorization. Tools from all other servers are returned as usual, so use this list to show the user which servers are still waiting to be authorized.
2. Choose a model
List the chat models available to the key:
curl -sS "${API_BASE_URL}/v1/models" \
-H "Authorization: Bearer ${API_KEY}"
Choose an id from the response's data array. Listing a chat model does not guarantee that it supports function
calling. Select a model configured for function calling and a compatible tool from the previous steps:
export MODEL_ID="YOUR_MODEL_ID"
export TOOL_NAME="YOUR_RAG__search_documents"
3. Call the Responses API
Declare the gateway-hosted tool as a function in the request. Copy its description from tools/list, and map its
inputSchema to the Responses API's parameters field. tool_choice defaults to auto; use none to disable tool
calls. Additional choices depend on the selected model — see
tool-choice support.
curl -sS --max-time 90 "${API_BASE_URL}/v1/responses" \
-H "Authorization: Bearer ${API_KEY}" \
-H "Content-Type: application/json" \
--data @- <<JSON
{
"model": "${MODEL_ID}",
"input": "Answer from the documents: What are the main conclusions?",
"stream": false,
"store": false,
"max_output_tokens": 2048,
"tools": [
{
"type": "function",
"name": "${TOOL_NAME}",
"description": "Search the relevant organizational documents.",
"parameters": {
"type": "object",
"properties": {
"query": {
"type": "string"
}
},
"required": ["query"]
}
}
]
}
JSON
For a gateway-hosted tool, the response's output array contains the ordered trace:
function_call— the tool selected by the model and its argumentsfunction_call_output— the result returned by the gateway-hosted toolmessage— the model's final answer after it has used the result
The gateway handles this loop inside the original HTTP request. If the model emits several gateway-hosted tool calls in one turn, the gateway executes them concurrently and preserves their planned order in the trace.
Only gateway-hosted tools are executed this way. A function declared and allowed in the request but not registered with the gateway is returned as a
function_callfor the client to handle.
When the gateway makes more than one model call for a request, usage also lists each call's counts in the order the
calls ran, in input_tokens_details.input_tokens_per_turn and output_tokens_details.output_tokens_per_turn. The
totals stay the sum of every call. Each call is sent the whole conversation so far, so the last input count plus the
last output count is the context the next request carries.
Tool access is decided for each request
Before each model turn, the Model Gateway determines which tools are callable for the current request. For a direct
model request, the set starts with the tool definitions supplied in that request. For a preset-agent request, it starts
with the tools configured for the agent. Gateway-hosted tools must also pass current discovery and access checks, and
tool_choice can narrow the set further. The Model Gateway includes the resulting tool names in the instructions sent
to the model.
A tool call or tool result included in input is historical context only. It does not add that tool to the current
request or authorize the Model Gateway to execute it. The current callable-tool list gives the model an authoritative
view of its access for that turn.
A tool call in input can name a tool that the current request no longer declares. For in-platform and custom
OpenAI-compatible models, the Model Gateway sends such a call, and its result, to the model as plain text instead of as
structured items. Any part of the result that is not text, such as an image, is replaced by a placeholder. Models
provided by OpenAI or Azure OpenAI receive the items unchanged. This changes only what the model reads: the response
you receive and the stored conversation keep the original items.
Setting "tool_choice": "none" makes the current callable-tool list empty even when the request contains tool
definitions. The model is therefore instructed that it cannot call any tool for that turn.
If the model calls a function that is undeclared or disallowed for the request, the gateway returns a
function_call_output error to the model without executing the function, then lets the model continue. This also
applies when no tools are declared or tool_choice is none. For a tool the gateway executes, the same recovery
applies when the call's arguments are not valid JSON or omit a property the tool's schema requires. The gateway
returns the validation message as a function_call_output error and the model can correct the call. Each corrected
attempt uses one model turn. If the model never sends valid arguments, the response ends with status incomplete.
The gateway does not validate arguments for a client-owned function. The response returns its function_call as the
model produced it, and the application must validate the arguments and handle bad input itself. Backend failures,
unsupported hosted-tool payloads and a custom_tool_call or tool_search_call that the request does not allow
still fail the response.
When a gateway-hosted tool fails, its function_call_output comes back with an id that starts with
fc_output_error_. Keep that id when you send the item back in a later request: the gateway reads it and tells the
model the result was a failure. If you drop the id or build the item yourself, the model receives the result without
that failure flag. The gateway does not decide failure from the result text.
Complete a client-owned function call
Declare a client-owned function in tools using the same shape as above. If the response ends with its
function_call, run the function and send its output back with the prior items. Keep the same tools value in the
next request:
import json
import os
prompt = "Look up the weather in Helsinki."
first = client.responses.create(
model=os.environ["MODEL_ID"],
input=prompt,
tools=tools,
store=False,
)
call = next(item for item in first.output if item.type == "function_call")
result = run_your_function(call.name, json.loads(call.arguments))
input_items = [
{"role": "user", "content": prompt},
*(item.model_dump() for item in first.output),
{
"type": "function_call_output",
"call_id": call.call_id,
"output": json.dumps(result),
},
]
second = client.responses.create(
model=os.environ["MODEL_ID"],
input=input_items,
tools=tools,
store=False,
)
print(second.output_text)
Keep store: false because the application resends the complete history in input.
On OpenAI models, client-owned functions and custom tools can also be grouped in a namespace tool. A namespaced
call is always returned to the client, and its namespace field must be kept when the call is sent back in input.
In an additional_tools item, OpenAI calls the tools of a namespace named functions without a namespace. Do not
use the namespace name functions in top-level tools or a search result. See
Use tools for the rules.
OpenAI models also accept client-owned custom tools, which take free-form text instead of JSON arguments. A
custom_tool_call is always returned to the client, which answers it with a custom_tool_call_output whose output
is a string or an array of content parts. See Use tools for the rules.
On OpenAI models, a client tool_search tool lets the model ask the application for more tools. The application runs
the search and answers the tool_search_call with a tool_search_output that lists functions or namespaces of
functions. The model can then call those functions, and each call is returned to the application. See
Use tools for the rules.
OpenAI models also accept an additional_tools input item, which declares client-owned functions, custom tools and
namespaces inside input instead of the top-level tools array. The gateway never runs a declared tool; each call to
one is returned to the application. Send the item again in each later request. See
Use tools for the rules.
Use the published tool definitions
The example above declares one tool by hand. In an application that should automatically use every available tool,
call tools/list at runtime and convert each compatible definition before sending the Responses request:
MCP tools/list field | Responses API field |
|---|---|
name | name |
description | description |
inputSchema | parameters |
Add "type": "function" to each converted definition. Include only names matching
^[A-Za-z0-9_-]{1,64}$, then place the resulting array in the Responses request's tools field. Perform this
transformation with the JSON facilities in your application's language rather than shell text processing.
If discovery succeeds with an empty list, the request sends "tools": [] and the model cannot call a gateway-hosted
tool. Published tools with incompatible names are skipped. They remain available to MCP clients but cannot be copied
verbatim into a Responses API function declaration.
Clear tool descriptions improve routing. Ask the model for document evidence when retrieval is required. Otherwise, the model may answer from its own knowledge without calling a tool.
Troubleshooting
| Symptom | What to check |
|---|---|
tools/list returns an empty array | Check access, readiness, reachability, and OAuth credentials. |
A tool from an OAuth-backed MCP server is missing from tools | Read result._meta under confidentialmind.com/mcp-auth-required. It names the server and the reason: no token was sent, the token type is not supported, or the server rejected the token. |
/v1/responses returns 403 | The API key needs invoke access to the selected model. |
/v1/responses fails with a normalized model-service message | See Model backend errors on the Responses API for the public meaning. Ask your administrator to inspect the Model Gateway diagnostics for the exact provider reason. |
/v1/responses rejects a tool name | Use a name matching ^[A-Za-z0-9_-]{1,64}$. |
/v1/responses returns 400 for a namespace, custom or tool_search tool, or an additional_tools item | Use a direct model provided by OpenAI, not Azure OpenAI, and set store: false. See Use tools for the declaration rules. |
| The response contains no tool call | Confirm function-calling configuration, then explicitly ask for evidence. |
The response ends with function_call | Confirm the function name belongs to a tool published by the gateway. |
A function_call_output reports an undeclared or disallowed tool | Check the function name in tools and the tool_choice policy. The error says the tool "was not declared in this request or not allowed by tool policy". The gateway returns the error without executing the function and lets the model continue. |
A function_call_output reports invalid arguments | The model sent arguments the tool's schema rejects, for example tool call arguments are not valid JSON or missing required property "query". The gateway does not run the tool. It returns the message to the model, which can send the call again with corrected arguments. Check that parameters in your request matches the tool's inputSchema from tools/list. |
A function_call_output says a tool is currently unavailable | The gateway could not reach that tool's server. The message names the tool but not the reason. The model continues with the results of the other tools. Ask your administrator to inspect the Model Gateway logs for the reason. |
| The model cannot see an image from an earlier tool result | Check whether the request still declares that tool. If it does not, an in-platform or custom OpenAI-compatible model receives the earlier call and its result as plain text, without the parts that are not text. Declare the tool again to send the result unchanged. See Tool access is decided for each request. |
mcp_auth_required result | Send the user's token; see MCP tool calls. |
/mcp/ returns the batch error | Send one JSON-RPC message per request when using mcp_authorizations. |