Use Model Gateway Tools with the Responses API

The Model Gateway's Responses API lets a model use gateway-hosted RAG and MCP tools during a response. The gateway executes those tools, returns their results to the model, and repeats until the model produces a final answer. Your application receives the complete tool-calling trace and answer from one request to /v1/responses.

This guide uses a non-streaming request so the complete trace is easy to inspect.

Prerequisites

Set your API base URL and API key:

export API_BASE_URL="https://api.<your-domain>"
export API_KEY="cm_api_…"

The API key needs invoke access to a chat model that supports function calling and access to the RAG or MCP tools you want to use. For a self-hosted vLLM model, automatic function calling requires --enable-auto-tool-choice and a compatible --tool-call-parser. The examples require curl and jq.

The examples verify TLS certificates. Add -k to curl only when testing an environment with a self-signed certificate.

This guide covers RAG and MCP tools that do not need an end-user sign-in. To call a tool from a service each user signs in to, send the user's access token with the request — see Call tools on the MCP endpoint.

1. Discover available tools

The Model Gateway exposes an MCP endpoint at /mcp/. Call tools/list to see the tools available to your API key:

curl -fsS -X POST "${API_BASE_URL}/mcp/" \
  -H "Authorization: Bearer ${API_KEY}" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "tools/list",
    "params": {}
  }' \
  | sed -n 's/^data: //p' \
  | jq -e '
      if (.result.tools | type) == "array" then
        .result.tools
      else
        error("tools/list did not return a tool array")
      end'

Each tool includes:

name — the name published by the MCP endpoint
description — when the model should use the tool
inputSchema — the arguments accepted by the tool

To use a published tool as a /v1/responses function, its name must match ^[A-Za-z0-9_-]{1,64}$.

The list is filtered by the caller's permissions. An empty array means no tool definitions were returned. This can be caused by missing access, a tool that is not ready or reachable, or missing credentials for an OAuth-backed MCP tool.

Some MCP servers need their own OAuth authorization from the end user. The tools of such a server are left out of tools until it is authorized. The response then lists the server in result._meta, under the key confidentialmind.com/mcp-auth-required:

{
  "_meta": {
    "confidentialmind.com/mcp-auth-required": [
      {
        "type": "mcp_auth_required",
        "code": "mcp_auth_required",
        "server": "github",
        "auth_type": "oauth_authorization_code",
        "reason": "missing_token"
      }
    ]
  }
}

Each entry describes one server:

server — the name of the MCP server that needs authorization
auth_type — the authorization the server expects, oauth_authorization_code
reason — missing_token when the request carried no token for that server, unsupported_token_type when the token is not a bearer token, and rejected_token when the server refused the token

The key is absent when no server needs authorization. Tools from all other servers are returned as usual, so use this list to show the user which servers are still waiting to be authorized.

2. Choose a model

List the chat models available to the key:

curl -sS "${API_BASE_URL}/v1/models" \
  -H "Authorization: Bearer ${API_KEY}" \
  | jq -r '.data[].id'

Listing a chat model does not guarantee that it supports function calling. Select a model configured for function calling and a compatible tool from the previous steps:

export MODEL_ID="YOUR_MODEL_ID"
export TOOL_NAME="YOUR_RAG__search_documents"

3. Call the Responses API

Declare the gateway-hosted tool as a function in the request. Copy its description from tools/list, and map its inputSchema to the Responses API's parameters field.

curl -sS --max-time 90 "${API_BASE_URL}/v1/responses" \
  -H "Authorization: Bearer ${API_KEY}" \
  -H "Content-Type: application/json" \
  --data @- <<JSON | jq -r '
    if .error then
      error(.error.message)
    else
      .output[] |
      if .type == "function_call" then
        "TOOL CALL   -> \(.name) \(.arguments)"
      elif .type == "function_call_output" then
        "TOOL RESULT <- \((.output | tostring)[0:140])"
      elif .type == "message" then
        "\nFINAL ANSWER:\n" + ([.content[]?.text] | join(" "))
      elif .type == "reasoning" then
        "(reasoning...)"
      else
        .type
      end
    end'
{
  "model": "${MODEL_ID}",
  "input": "Answer from the documents: What are the main conclusions?",
  "stream": false,
  "store": false,
  "max_output_tokens": 2048,
  "tools": [
    {
      "type": "function",
      "name": "${TOOL_NAME}",
      "description": "Search the relevant organizational documents.",
      "parameters": {
        "type": "object",
        "properties": {
          "query": {
            "type": "string"
          }
        },
        "required": ["query"]
      }
    }
  ]
}
JSON

For a gateway-hosted tool, the response's output array contains the ordered trace:

function_call — the tool selected by the model and its arguments
function_call_output — the result returned by the gateway-hosted tool
message — the model's final answer after it has used the result

The gateway handles this loop inside the original HTTP request. If the model emits several gateway-hosted tool calls in one turn, the gateway executes them concurrently and preserves their planned order in the trace.

Only gateway-hosted tools are executed this way. A function that is not registered with the gateway is returned as a function_call for the client to handle.

Use the published tool definitions

The example above declares one tool by hand. As an alternative, retrieve every tool available to the API key at call time so the request uses the definitions published by the gateway:

if TOOLS_JSON=$(
  curl -fsS -X POST "${API_BASE_URL}/mcp/" \
    -H "Authorization: Bearer ${API_KEY}" \
    -H "Content-Type: application/json" \
    -H "Accept: application/json, text/event-stream" \
    -d '{
      "jsonrpc": "2.0",
      "id": 1,
      "method": "tools/list",
      "params": {}
    }' \
  | sed -n 's/^data: //p' \
  | jq -ce '
      if (.result.tools | type) != "array" then
        error("tools/list did not return a tool array")
      else
        [.result.tools[]
          | select(.name | test("^[A-Za-z0-9_-]{1,64}$"))
          | {
              type: "function",
              name: .name,
              description: .description,
              parameters: .inputSchema
            }]
      end'
); then
  curl -sS --max-time 120 "${API_BASE_URL}/v1/responses" \
    -H "Authorization: Bearer ${API_KEY}" \
    -H "Content-Type: application/json" \
    --data @- <<JSON | jq -r '
    if .error then
      error(.error.message)
    else
      .output[] |
      if .type == "function_call" then
        "TOOL CALL   -> \(.name) \(.arguments)"
      elif .type == "function_call_output" then
        "TOOL RESULT <- \((.output | tostring)[0:140])"
      elif .type == "message" then
        "\nFINAL ANSWER:\n" + ([.content[]?.text] | join(" "))
      elif .type == "reasoning" then
        "(reasoning...)"
      else
        .type
      end
    end'
{
  "model": "${MODEL_ID}",
  "input": "Answer from the documents: What are the main conclusions?",
  "stream": false,
  "store": false,
  "max_output_tokens": 2048,
  "tools": ${TOOLS_JSON}
}
JSON
else
  echo "Unable to load published tool definitions." >&2
fi

The conditional calls tools/list and skips the Responses request if discovery does not return a tool array, without exiting the current shell. jq keeps names that satisfy the Responses API function-name constraint, converts each compatible MCP definition, and renames inputSchema to parameters. The resulting array is inserted into the outer request's tools field.

If discovery succeeds with an empty list, the request sends "tools": [] and the model cannot call a gateway-hosted tool. Published tools with incompatible names are skipped. They remain available to MCP clients but cannot be copied verbatim into a Responses API function declaration.

Clear tool descriptions improve routing. Ask the model for document evidence when retrieval is required. Otherwise, the model may answer from its own knowledge without calling a tool.

Troubleshooting

Symptom	What to check
`tools/list` returns an empty array	Check access, readiness, reachability, and OAuth credentials.
A tool from an OAuth-backed MCP server is missing from `tools`	Read `result._meta` under `confidentialmind.com/mcp-auth-required`. It names the server and the `reason`: no token was sent, the token type is not supported, or the server rejected the token.
`/v1/responses` returns `403`	The API key needs `invoke` access to the selected model.
`/v1/responses` fails with a normalized model-service message	See Model backend errors on the Responses API for the public meaning. Ask your administrator to inspect the Model Gateway diagnostics for the exact provider reason.
`/v1/responses` rejects a tool name	Use a name matching `^[A-Za-z0-9_-]{1,64}$`.
The response contains no tool call	Confirm function-calling configuration, then explicitly ask for evidence.
The response ends with `function_call`	Confirm the function name belongs to a tool published by the gateway.
`mcp_auth_required` result	Send the user's token; see MCP tool calls.
`/mcp/` returns the batch error	Send one JSON-RPC message per request when using `mcp_authorizations`.

Prerequisites​

1. Discover available tools​

2. Choose a model​

3. Call the Responses API​

Use the published tool definitions​

Troubleshooting​