Skip to main content
Version: 3.6.0

Responses API

Use POST /v1/responses for text generation, streaming, conversations, and model tool use. The API follows the OpenAI Responses request and response shapes where documented here, with platform-specific conversation handling and gateway-hosted tools.

For a single prompt and answer, send a string in input. For multiple turns, either send the history as an input array or let the platform store it and continue with conversation_id.

Responses also supports direct Anthropic models. That guide covers connector setup, translated content and settings, tool ownership, and the history requirements for preserved thinking.

Before you begin​

You need:

  • the API URL shown in the platform portal
  • an API key with invoke access to a chat model
  • the model ID from GET /v1/models

The raw HTTP examples use curl. Set these values once:

export API_BASE_URL="https://api.<your-domain>"
export API_KEY="cm_api_…"
export MODEL_ID="YOUR_MODEL_ID"

The Python examples use the OpenAI Python package and the same environment variables. Install the package with your project's usual Python dependency manager.

Generate a response​

Send a non-streaming request when you want one JSON result:

curl -sS "${API_BASE_URL}/v1/responses" \
-H "Authorization: Bearer ${API_KEY}" \
-H "Content-Type: application/json" \
--data-binary @- <<JSON
{
"model": "${MODEL_ID}",
"input": "Explain confidential computing in two sentences.",
"store": false
}
JSON

The answer is in the output array. A message can contain one or more output_text parts:

{
"status": "completed",
"output": [
{
"type": "message",
"role": "assistant",
"content": [
{
"type": "output_text",
"text": "Confidential computing protects data while it is being processed..."
}
]
}
]
}

If the model calls a function that is undeclared or disallowed for the request, the output array includes a function_call and a function_call_output with an error message. The gateway does not execute the function and lets the model continue, so the response can still end with status completed. This also applies when no tools are declared. Read the answer from the message items, as the examples on this page do.

See Tool access is decided for each request for details.

With the OpenAI Python client:

import os

from openai import OpenAI

client = OpenAI(
base_url=f"{os.environ['API_BASE_URL']}/v1",
api_key=os.environ["API_KEY"],
)

response = client.responses.create(
model=os.environ["MODEL_ID"],
input="Explain confidential computing in two sentences.",
store=False,
)

if response.status != "completed":
raise RuntimeError(f"Response ended with status {response.status}: {response.error}")

print(response.output_text)

For direct model requests, store: false makes this a one-shot request. Storage otherwise defaults to true, and the gateway creates or continues a conversation independently of the selected model backend.

Give the model instructions​

Use instructions for behavior that should apply to the current request:

{
"model": "YOUR_MODEL_ID",
"instructions": "Answer for a non-technical reader. Be concise.",
"input": "What is a trusted execution environment?",
"store": false
}

instructions accepts a string or null. Any other JSON value returns 400 with the message instructions must be a string or null. The response's instructions field contains the original string you sent, or null when you omitted the field or sent null.

Preset agents use the system prompt saved in their configuration. Do not send instructions when invoking a preset agent; the field is not permitted on preset-agent requests. A preset-agent response always contains "instructions": null, so its configured prompt is not exposed.

For direct model requests, parameters such as max_output_tokens, temperature, and top_p can be sent when the selected model and its configuration support them. For an Anthropic model, temperature and top_p return 400 at any value, including defaults such as temperature: 1. Leave them out, or send null. Preset agents use their configured parameters; callers cannot override them.

Chat mode​

Set chat_mode to true to use the instruction bundle that Chat requests:

{
"model": "YOUR_MODEL_ID",
"input": "Explain this formula.",
"chat_mode": true,
"store": false
}

This gateway extension works with direct models and preset agents on POST /v1/responses. Omitting it or sending false disables the bundle. null and other JSON types return HTTP 400 with the message field chat_mode must be a boolean. Chat sends true on every request, including follow-ups and retries. API callers can opt in too; the flag selects behavior and does not establish client identity or grant permissions.

  • Direct models: fixed Chat formatting guidance, tenant assistant instructions, any caller instructions, then the current UTC date.
  • Preset agents: configured agent instructions, fixed Chat formatting guidance, then the current UTC date. Tenant assistant instructions do not apply, even when the agent's configured instructions are empty.

The gateway adds the currently callable tool names after these instructions on each model dispatch, including dispatches after tool results, independently of Chat mode. This context reflects the request, agent configuration, and existing permissions; it does not grant access to tools. Agent restrictions and configured output guardrails also apply independently. There is no separate UTC option.

The date is captured once per request and stays the same across tool turns, even if they cross UTC midnight. For direct models, tenant instructions are also loaded once per request. Later requests see updated tenant settings. If that lookup fails, the request returns an error before model execution. Agent requests do not perform the lookup.

Requests that omit chat_mode or set it to false receive no automatic UTC date. Earlier platform versions added it to every Responses request. If your integration relies on that date, send chat_mode: true to enable the full Chat bundle, or include the date in your own direct-model instructions or preset-agent instructions.

The gateway consumes chat_mode before sending the request to the model provider. Injected instructions are not saved as conversation messages or included in public instruction snapshots. Direct-model snapshots retain only caller instructions; agent snapshots contain instructions: null. A model can still mention an injected fact in its answer. Send chat_mode on each request; stored conversations do not remember the option.

With the Python SDK, pass the extension through extra_body:

response = client.responses.create(
model=os.environ["MODEL_ID"],
input="Explain this formula.",
store=False,
extra_body={"chat_mode": True},
)

Continue a conversation​

You can keep history in your application or let the platform store it.

ApproachWhen to use it
Send an input arrayPortable client-managed pattern; your application owns and resends the history.
Use conversation_idPlatform-stored history; later requests send only the new input.

Preset-agent requests accept only user messages from the caller, so use platform-stored history instead of resending client-managed history. Storage is disabled by default: set store: true on the first turn, then continue with the returned conversation_id. Do not combine conversation_id with store: false; the request returns 400.

Send history yourself​

Start with a user message, then add the response output and the next user message to the next request:

history = [{"role": "user", "content": "My deployment window starts at 09:00 UTC."}]

first = client.responses.create(
model=os.environ["MODEL_ID"],
input=history,
store=False,
)

history.extend(item.model_dump() for item in first.output)
history.append({"role": "user", "content": "What time does my deployment window start?"})

second = client.responses.create(
model=os.environ["MODEL_ID"],
input=history,
store=False,
)

print(second.output_text)

Set store: false on every client-managed turn. Storage is enabled by default, so omitting it would create a new stored conversation for each request even though your application is already supplying the history.

Let the platform store history​

On the first request, omit conversation_id. The response contains a new ID in conversation.id:

curl -sS "${API_BASE_URL}/v1/responses" \
-H "Authorization: Bearer ${API_KEY}" \
-H "Content-Type: application/json" \
--data-binary @- <<JSON
{
"model": "${MODEL_ID}",
"input": "My deployment window starts at 09:00 UTC. Remember that.",
"store": true
}
JSON

Copy the ID from the response's conversation object:

{
"conversation": {
"id": "YOUR_CONVERSATION_ID"
}
}

Pass that ID as conversation_id on the next request. You send only the new input; the gateway loads the stored history and supplies it to the model:

curl -sS "${API_BASE_URL}/v1/responses" \
-H "Authorization: Bearer ${API_KEY}" \
-H "Content-Type: application/json" \
--data-binary @- <<JSON
{
"model": "${MODEL_ID}",
"conversation_id": "YOUR_CONVERSATION_ID",
"input": "What time does my deployment window start?"
}
JSON

The same flow with the OpenAI Python client uses extra_body because conversation_id is a platform extension:

first = client.responses.create(
model=os.environ["MODEL_ID"],
input="My deployment window starts at 09:00 UTC. Remember that.",
store=True,
)

if first.conversation is None:
raise RuntimeError("The gateway did not create a conversation")

conversation_id = first.conversation.id

second = client.responses.create(
model=os.environ["MODEL_ID"],
input="What time does my deployment window start?",
extra_body={"conversation_id": conversation_id},
)

print(second.output_text)

Keep the conversation_id in your application's session or database if the conversation must survive a process restart. Start a new conversation by omitting it from a later request. Use the same API key for later turns.

Conversation behavior and limits​

BehaviorWhat to expect
CreateOmit conversation_id and set store: true. The response contains conversation.id.
ContinueSend the stored ID as conversation_id; the gateway supplies recent history before the new input.
History windowUp to 128 recent stored items are supplied. The model's context limit still applies.
AccessA conversation is scoped to its creator and tenant. Keep using the same API key.
Disable storageSet store: false. No conversation is created and conversation is absent from the response.
Change modelsPrefer the same model. Model-specific reasoning and tool items may be omitted after a change.
Manage conversationsThe public /v1 API currently does not expose list, retrieve, or delete operations.

When assistant text is saved before a response fails, the terminal event includes response.conversation.output_item_id, the stored assistant item's ID. That partial answer remains in conversation history and is supplied to the model on the next turn, subject to the history window above, unless it is deleted. Clients with conversation management access can delete it through the separate ConnectRPC endpoint POST /cmind.conversations.v1.ConversationItemService/DeleteItem, with conversationId set to conversation.id, itemId set to conversation.output_item_id, and tenantId set to the conversation's tenant ID in the JSON body. This is not a /v1 operation. Clients using only /v1 can omit conversation_id to start a new conversation without that saved partial answer.

conversation_id is the supported mechanism for continuing platform-stored history. The previous_response_id and conversation request fields are not currently supported and do not continue a platform conversation.

Stream a response​

Set stream: true to receive Server-Sent Events. Text arrives in response.output_text.delta events, followed by a terminal response.completed, response.incomplete, or response.failed event and [DONE].

stream = client.responses.create(
model=os.environ["MODEL_ID"],
input="Write a four-line poem about private AI.",
stream=True,
store=False,
)

for event in stream:
if event.type == "response.output_text.delta":
print(event.delta, end="", flush=True)
elif event.type == "response.failed":
raise RuntimeError(event.response.error)

print()

When storage is enabled, the terminal event's response.conversation.id contains the conversation ID to use for the next request.

A preset agent with output guardrails holds its final answer until the checks finish. The stream sends response.created and content-free response.in_progress keepalives while waiting. Intermediate assistant text, reasoning, tool calls and tool results stay private. These keepalives do not report generation or tool progress. Clients displaying text deltas receive no answer text until generation and judging finish, so users see a wait followed by the complete result. The selected final answer or block message then arrives as a group of SSE events without pacing, followed by the terminal event and [DONE].

An answer can take longer than 10 minutes. The gateway ends a model turn only when the model sends nothing for 10 minutes or the turn lasts 2 hours, with or without stream. See Response time limits for what the client receives.

Use tools​

The Responses API can run function tools published by the Model Gateway, including RAG and connected MCP tools. The gateway can execute those tools and feed their results back to the model within the same request.

→ Use Model Gateway tools with the Responses API

Tool availability depends on the selected model's function-calling support and configuration. OpenAI-hosted tool types such as web search, file search, code interpreter, computer use, and image generation are not provided by this endpoint, regardless of the selected model.

A model can still return an unsupported hosted-tool call, such as web_search_call. The gateway does not run it and does not send it to you. The response fails instead. A streaming request ends with a response.failed event and [DONE]. Text that already arrived stays in the stream. See Model backend errors on the Responses API.

tool_choice support depends on the request target:

  • Direct in-platform and custom OpenAI-compatible models support auto (the default), none, and allowed_tools with mode auto.
  • Direct models provided by OpenAI or Azure OpenAI also support required, a named function, and allowed_tools with mode required.
  • Direct Anthropic models support auto, none, and allowed_tools with mode auto. Haiku 5.5 also supports required, a named function, and allowed_tools with mode required. Other supported Anthropic models reject forced choices. See Anthropic tool support and qualification limits.
  • Preset-agent requests do not accept tool_choice; the agent uses its configured tools.

A named function uses {"type": "function", "name": "your_function"}. To restrict the callable set, use:

{
"type": "allowed_tools",
"mode": "auto",
"tools": [{"type": "function", "name": "your_function"}]
}

Unsupported choices return 400. A named function must appear in the request's tools array; allowed_tools can only select from functions declared there.

A namespace tool groups client-owned functions under one name. Direct models provided by OpenAI accept it when the request sets store: false:

{
"type": "namespace",
"name": "crm",
"description": "Customer records.",
"tools": [{"type": "function", "name": "lookup", "parameters": {"type": "object"}}]
}

The model's function_call carries "namespace": "crm" and "name": "lookup". The gateway never runs a namespaced tool, even when a gateway tool has the same name; it returns the call for your application to run, as described in Complete a client-owned function call. Send the full conversation in input on each request. A namespace needs a unique name matching ^[A-Za-z0-9_-]{1,64}$, a string description and at least one tool, with no repeated tool names inside the namespace. The description can be empty; a missing or null description returns 400. A namespace holds function tools and custom tools, but not other namespaces, and does not support defer_loading: true. A call to a namespaced custom tool is a custom_tool_call that carries the namespace. The exception is a namespace named functions in an additional_tools item (see below): OpenAI calls its tools by their plain names, with no namespace. Do not name a namespace functions in top-level tools or a search result. That case is untested: the gateway keeps those tools under their namespace, so a call by plain name from the model would not match them. A named tool_choice or allowed_tools cannot select a namespaced tool; under allowed_tools the model gets only the selected ordinary functions. Other models, including Azure OpenAI models, and requests with storage enabled, return 400 for a namespace declaration or a namespaced call in input.

A custom tool takes free-form text instead of JSON arguments. Direct models provided by OpenAI accept it when the request sets store: false. Omit format for free text, or constrain the text with a Lark grammar:

{
"type": "custom",
"name": "record_receipt",
"description": "Records a receipt code.",
"format": {"type": "grammar", "syntax": "lark", "definition": "start: \"receipt-\" /[a-z0-9]{8}/"}
}

The model returns a custom_tool_call with call_id, name and the text in input. When streaming, the text also arrives in response.custom_tool_call_input.delta and response.custom_tool_call_input.done events. The gateway never runs a custom tool, even when a gateway tool has the same name; your application runs it and sends back the call and a custom_tool_call_output with the full conversation in input. Its output is a string or an array of content parts, as in a function_call_output, such as [{"type": "input_text", "text": "..."}]. Other formats, including a regex grammar, and defer_loading: true return 400, as does a custom tool whose name matches a function or another custom tool. A named tool_choice cannot select a custom tool, and under allowed_tools the model gets no custom tools. Other models, including Azure OpenAI models, and requests with storage enabled, return 400 for a custom tool declaration or a custom item in input.

A client tool_search tool lets the model ask your application for more tools during the conversation. Direct models provided by OpenAI accept it when the request sets store: false. Set execution to "client"; description and an object parameters schema are optional and are sent to the model as given:

{
"type": "tool_search",
"execution": "client",
"description": "Searches the application's tool catalog.",
"parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}
}

When the model searches, the response ends with a tool_search_call that carries a call_id and an arguments object. The gateway never runs the search. Your application runs it and sends back the call and a tool_search_output, with the full conversation in input. Both items need execution: "client" and the same call_id: a result must answer an earlier tool_search_call in the same input. A result's tools array can be empty or list function tools and one level of namespaces of function tools. Returned functions and namespaces must meet the same name, description and uniqueness rules as declared ones, so a returned namespace's description can be empty but must be present, and a returned plain function cannot use the name of a tool the gateway runs; otherwise the request returns 400. The model can then call a returned function, plain or namespaced, and the gateway returns that function_call for your application to run. A returned function also replaces a top-level custom tool with the same namespace and name, so a custom_tool_call of that name fails the response. Returned functions belong to your application: the gateway never runs them, never adds them to the top-level tools it sends to the model, and keeps nothing between requests, so send the search call and its output again with each later request.

Send the tool_search_call back as the gateway returned it, then add the result. Each tool in the result is a function or a namespace of functions:

{
"type": "tool_search_output",
"call_id": "call_ts1",
"execution": "client",
"tools": [
{
"type": "function",
"name": "fetch_doc",
"description": "Fetches a document.",
"parameters": {"type": "object", "properties": {"id": {"type": "string"}}, "required": ["id"]}
},
{
"type": "namespace",
"name": "crm",
"description": "Customer records.",
"tools": [{"type": "function", "name": "lookup", "parameters": {"type": "object"}}]
}
]
}

A call to a returned namespaced function carries "namespace": "crm" and "name": "lookup", as a call to a declared namespace does.

When streaming, the search call arrives in response.output_item.added and response.output_item.done; there are no events specific to search. The added item can still have status in_progress, with no call_id and no arguments. Read the complete call from the done event or from the final response.output.

defer_loading is honored only on functions inside a tool_search_output; on a top-level function it is ignored. A returned namespace that sets defer_loading: true returns 400, so set the field on the functions inside the namespace instead. With tool_choice set to none, a named function or allowed_tools, the model cannot use client search or call returned functions; under allowed_tools the search declarations are also left out of the request to the model. Other models, including Azure OpenAI models, and requests with storage enabled, return 400 for a search declaration or a search item in input.

Client tool search, including calls to the returned functions, has been tested with streaming and non-streaming requests on OpenAI gpt-6-astra, gpt-6.1-sol and gpt-6-luna. Whether another OpenAI model supports it depends on that model. For a request that declares only client tool search, use tool_choice: "auto"; this is the configuration tested on these three models.

An additional_tools input item declares client-owned tools inside input instead of the top-level tools array. Direct models provided by OpenAI accept it when the request sets store: false:

{
"type": "additional_tools",
"id": "at_1",
"role": "developer",
"tools": [
{
"type": "namespace",
"name": "functions",
"description": "",
"tools": [
{"type": "custom", "name": "exec", "description": "Runs a script."},
{
"type": "function",
"name": "wait",
"parameters": {"type": "object", "properties": {"cell_id": {"type": "string"}}, "required": ["cell_id"]}
}
]
}
]
}

role can be omitted; any value other than "developer", including null, returns 400. The gateway always sends the item to the model with "role": "developer", and keeps the id when you send one. tools must be an array and can be empty. It can hold function tools, custom tools and namespaces of function and custom tools, under the same rules as the top-level declarations above, including a namespace description that can be empty but must be present. Nested namespaces, hosted tools, MCP servers, tool_search and defer_loading: true return 400.

The gateway sends the item to the model in its place in input and never moves its tools into the top-level tools. Every declared tool belongs to your application: the gateway never runs it and returns the model's function_call or custom_tool_call, with its namespace when it has one, for your application to run. Calls to the tools of a namespace named functions carry no namespace, so the gateway treats those tools as plain tools, under the plain-tool name and allowed_tools rules below. A plain tool in the same item cannot share a name with one of them; that returns 400. The item is still sent to the model unchanged. The gateway keeps nothing between requests, so send each additional_tools item again, in the same place, with each later request.

A later item can add tools or redeclare a tool with the same namespace and name. Between declarations in input, the latest one decides whether the tool is a function or a custom tool, including a plain function that a later item redeclares as a custom tool. A function declared in input, in an item or a search result, also replaces a top-level custom tool with the same namespace and name. A custom tool in an item replaces a top-level function only inside a namespace other than functions; with a plain name, including a child of an item's functions namespace, the request returns 400 (see below). After a function is replaced by a custom tool, a function_call of that name gets an error result and the model continues. After a custom tool is replaced by a function, a custom_tool_call of that name fails the response. Earlier items are still sent to the model unchanged.

A plain function in the item cannot use the name of a tool the gateway runs, and a plain custom tool in the item cannot use the name of a top-level function or a gateway tool; either returns 400. With tool_choice set to auto or required, the model can call the declared tools. With none, a named function or allowed_tools, it cannot. A named tool_choice cannot select a declared tool, and an allowed_tools entry that names a function only an additional_tools item declares returns 400. Other models, including Azure OpenAI models, preset agents, and requests that do not set store: false return 400 when input contains an additional_tools item, including one from earlier turns.

The additional_tools item has been tested on OpenAI gpt-6-astra with streaming and non-streaming requests for namespaced calls and with streaming requests for calls to the tools of a functions namespace, and through Codex CLI 0.167.0 on gpt-6-astra, gpt-6.1-sol and gpt-6-luna. Whether another OpenAI model supports it depends on that model.

Handle unsuccessful responses​

Before generation begins, authentication, authorization, validation, and backend connection failures use HTTP error statuses. After a response has begun, the HTTP status can remain 200; inspect the response status and error, or the terminal streaming event, to determine the outcome.

If the backend stream ends unexpectedly, the gateway sends response.failed followed by [DONE] while the client connection remains writable. Usage from completed model turns is still reported.

See Model backend errors on the Responses API for the stable client-facing errors and retry guidance.

Compatibility notes​

This endpoint is a supported subset of the OpenAI Responses API. In particular:

  • only POST /v1/responses is exposed
  • background (even when false) and previous_response_id are not currently supported; omit these fields
  • platform conversations use conversation_id
  • tool_choice support varies by model provider and request target; see tool-choice support
  • function and MCP tools are supported, and namespace, custom and client tool_search tools and additional_tools input items on direct OpenAI models (not Azure OpenAI) with store: false; OpenAI-hosted tool types, including hosted tool search, are not
  • reasoning.context (auto, current_turn or all_turns) is supported only on direct OpenAI models (not Azure OpenAI) with store: false; other values and other targets return 400, and omitting it or sending null leaves it unset. The response's reasoning.context is the value the model provider reports, which can differ from the value you sent
  • preset-agent requests accept only user messages, use configured model parameters, and default to store: false
  • preset agents with output guardrails delay the final answer until checks finish; see Stream a response
  • input, output, tool use, sampling, and multimodal support can vary by model

Use this page as the Model Gateway contract. Fields and operations not documented here are not part of the supported public surface.

The gateway translates supported fields instead of forwarding arbitrary JSON properties. Unknown properties are ignored; preset field restrictions and unsupported-feature checks still apply. Undocumented backend request options, such as top_k and seed, are not forwarded. The supported chat_template_kwargs dictionary retains its template-defined keys. Malformed known settings or malformed/unsupported input items return HTTP 400 before conversation creation, model dispatch or streaming begins.

Malformed known backend fields and unsupported content parts fail the response. Unrecognized events within otherwise permitted event families are ignored. Provider-hosted tool events remain rejected.