Model Gateway

The Model Gateway is the single, OpenAI-compatible access point for every model on the platform. Whatever the backend — an operator-managed in-cluster deployment (vLLM/SGLang), an external provider (OpenAI, Azure, …), or an alias to either — clients talk to one base URL and the gateway routes the request to the right place, applying authorization and usage accounting along the way.

Point any OpenAI-compatible client at the base URL https://api.<your-domain>/v1 and pass an API key:

curl -sS "https://api.<your-domain>/v1/chat/completions" \
  -H "Authorization: Bearer cm_api_…" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3-6-27b-fp8", "messages": [{"role": "user", "content": "Hello"}]}'

Endpoints

Endpoint	Purpose
POST /v1/chat/completions	Chat completions
POST /v1/embeddings	Embeddings
POST /v1/rerank	Reranking
POST /v1/responses	Responses API — agentic tool loop (guide)
GET /v1/models	List available models

Every endpoint follows the OpenAI API, so existing SDKs and tools — the OpenAI client libraries, Open WebUI, and the like — work unchanged once you set the base URL and key.

Authentication

Every request needs an API key as a bearer token, and the key must have invoke granted on the model — or on an alias that points at it. A request for a model it can't invoke returns 403.

For external models you still authenticate with your platform API key. The gateway holds the provider's credentials and attaches them when forwarding, so OpenAI or Azure keys never live in your client.

Calling a model

Pass the model's id — exactly as returned by GET /v1/models — in the model field. The gateway resolves it to the right backend, in-cluster or external, transparently; you never address a backend directly. Administrators can also publish an alias: a stable public name pointing at a model that can be repointed later without changing client code.

An alias can only point to a platform-owned model. It cannot point to an external model owned by a single tenant; the platform refuses that change and does not save the alias. The alias itself can still be shared with selected tenants or with everyone.

Request bodies are forwarded to the backend unchanged, so standard OpenAI features work as-is:

• Streaming — set "stream": true for Server-Sent Events; add "stream_options": {"include_usage": true} for a final token-usage event.
• Structured output — use OpenAI's response_format (json_schema or json_object) to constrain output to a schema, where the backend supports it.
• Image input — send image_url content parts to any model that lists image among its input modalities (see below).

Successful chat and embeddings responses carry the usual usage token counts.

Errors

Errors use the OpenAI JSON shape — {"error": {"message": …, "type": …}}:

Status	Meaning
400	Malformed request, or the model doesn't exist / isn't ready
401	Missing or invalid API key
403	The key isn't granted invoke on the model, or on an alias for it
502	The backend failed or was unreachable

Model backend errors on the Responses API

The following behavior applies when the selected model backend rejects a request to /v1/responses. It does not apply to gateway errors that happen before the model is invoked, such as an invalid platform API key or missing invoke permission.

The gateway does not return provider-owned error messages, types, or codes to the client. Clients receive image-specific guidance when the model service identifies unsupported image input. Unsupported audio, video, and unidentified input modalities use the generic invalid-request message.

The client receives a stable OpenAI-compatible error using one of these public messages:

Failure classification	Public message	Meaning
Unsupported input modality explicitly identified as image	The selected model does not support image input. Remove the image or choose a model that supports images, then try again.	The model service specifically identified image input as unsupported.
Unsupported audio, video, or unidentified input modality	The selected model could not process this request. Check your input and try again.	The model service rejected an input modality, but the gateway does not currently provide modality-specific guidance for it.
Context-window limit, usually 400 or 422	This request is too long for the selected model. Shorten the conversation or attached content, then try again.	The request exceeds the model's context window.
401 from the model service	The model service could not authenticate the request. Contact your administrator.	The credentials held by the platform for the model service were rejected. This does not refer to the caller's platform API key.
403 from the model service	The model service denied this request. Contact your administrator if you need access.	The model service refused the request.
408 or 504	The model took too long to respond. Try again.	The model did not respond before the timeout.
429	The model is receiving too many requests. Wait a moment and try again.	The model service is limiting the request rate.
Any other 4xx error	The selected model could not process this request. Check your input and try again.	The model rejected the request for another reason.
Any other 5xx error	The model service is temporarily unavailable. Try again.	The model service failed or the gateway could not reach it.

When the backend rejection is returned as an ordinary HTTP response, the gateway preserves its 400–599 status. If a direct 429 response includes a Retry-After header, the gateway forwards that header.

Once a Responses API response has begun, including during a gateway tool loop, the HTTP status can remain 200. For a non-streaming request, inspect the response's status and error fields. For a streaming request, a model-backend failure is returned as a canonical response.failed event. Its response.error object contains the public message, type, and code. A normally terminated stream then ends with [DONE].

Provider details are not returned to the client. Ask your administrator to inspect the Model Gateway diagnostics when the exact provider reason is needed.

This normalization is specific to /v1/responses. It does not change the behavior of /v1/chat/completions, /v1/embeddings, or /v1/rerank.

Model capabilities

Every model advertises what it can do, so callers and the UI can pick the right model for a task. Capabilities have three independent axes:

• type — the API surface / task family the model serves. Exactly one applies, and it decides which endpoint the model is invoked through:

  type	Endpoint
  chat	/v1/chat/completions, /v1/responses
  embedding	/v1/embeddings
  rerank	/v1/rerank

  More surfaces (transcription, image generation, …) are planned for a future release.

• input_modalities — what the model accepts: text, image, audio, video.

• output_modalities — what the model produces: text, embedding, image, audio.

type describes the API surface, not every capability. A vision-capable chat model is still type: chat; its image support is expressed as input_modalities: [text, image]. Keeping the task type and the modalities as separate axes (rather than one flat list) means a model always has exactly one task, while still describing the data shapes it handles.

A model whose type was never set is treated as chat (backward-compatible default), so models that predate this feature keep working until they are re-typed.

Listing and filtering models

GET /v1/models returns the models the caller is allowed to invoke. The response is OpenAI-compatible, with capability fields added:

{
  "object": "list",
  "data": [
    {
      "id": "qwen3-6-27b-fp8",
      "object": "model",
      "owned_by": "confidentialmind",
      "display_name": "Qwen3.6 27B FP8",
      "type": "chat",
      "input_modalities": ["text"],
      "output_modalities": ["text"]
    }
  ]
}

Default: chat only

By default GET /v1/models returns only chat models:

curl -sS "https://api.<your-domain>/v1/models" \
  -H "Authorization: Bearer <token>"

This keeps chat-first clients (e.g. Open WebUI, the platform chat UI) clean — embedding and reranker models don't clutter a model picker. Non-chat models appear only when asked for.

Filtering

Three query parameters filter the list, combined with AND:

• type — one of the values above, or all to disable the type filter.
• input_modalities — repeatable and/or comma-separated.
• output_modalities — repeatable and/or comma-separated.

Within a modality parameter the model must support every value listed.

# Everything, regardless of type
curl -sS "https://api.<your-domain>/v1/models?type=all" -H "Authorization: Bearer <token>"

# Just embedding models (e.g. to configure a RAG embedder)
curl -sS "https://api.<your-domain>/v1/models?type=embedding" -H "Authorization: Bearer <token>"

# Vision-capable chat models (accept image input, produce text)
curl -sS "https://api.<your-domain>/v1/models?type=chat&input_modalities=image&output_modalities=text" \
  -H "Authorization: Bearer <token>"

Quote the URL — an unquoted ?/& is interpreted by your shell.

An unknown type or modality value returns 400 Bad Request, so typos surface instead of silently returning an empty list.

How a model's capabilities are set

• Catalog (preset) deployments are typed automatically from the preset (e.g. the embedding preset deploys an embedding model). No extra input needed.
• Custom and external models default to chat; choose the correct type in the model create form before deploying.
• Existing deployments can be re-typed at any time from the model's Edit page — useful for classifying models created before capabilities existed (which otherwise read as chat). Editing the type is metadata-only and does not redeploy the model.

Input/output modalities are derived from the type by default (e.g. embedding → output [embedding]); set them explicitly when a model is multimodal (e.g. add image input for a vision chat model).