Skip to main content

Deploying a model

Prerequisites​

  • Model deployment privileges
  • Enough GPU VRAM
  • A model from Hugging Face
    • Some models require the user to sign an agreement to prevent misuse. In these cases, you will need a Huggingface account and token. Documentation for creating access tokens here: https://huggingface.co/docs/hub/en/security-tokens
    • Example: meta-llama/Llama-3.1-8B requires signing a community license agreement.

Simple Walkthrough​

In this walkthrough, we will deploy a Qwen3-0.6B model due to its small size for a quick deployment demonstration.

Setting the model:

  • Find the model card from Hugging Face. For our selected model, the model card is: https://huggingface.co/Qwen/Qwen3-0.6B
    • This model does not require signing any agreements.
  • From the ConfidentialMind portal on the left sidebar, click Models under Model endpoints and find Deploy model on the right.
  • Select Custom (first row)
  • Give the model deployment a name under Deployment name
  • Select the GPU(s) you would like to deploy the model on.
  • Enter the Hugging Face model ID under Model.
    • The ID format is generally: organization/model. There is also a clipboard icon next to the model ID at the top of the Hugging Face model card page that you can use.
    • In our example: Qwen/Qwen3-0.6B

Advanced Settings (Simple):

For this particular model, we will need the inference engine to override a few settings under the hood to custom ones. Click Advanced settings to open more configuration options. Near the bottom you will find Arguments. Simply add: --trust-remote-code. More explanations below.

Final steps:

  • Click Deploy model endpoint at the bottom.
  • Refresh the page to see the model status: Starting |Deployed | Stopped | Stopping | Error. The Qwen3-0.6B model should take ~5 - 10 minutes. Server initialization logs can be found under Logs.

Large custom models can remain in Starting while their weights load and the inference engine initializes. By default, the platform allows approximately 90 minutes for the model server to pass its startup health check. If it does not become healthy in that time, the server restarts and loading begins again. Check Logs to distinguish ongoing loading from repeated startup attempts.

Advanced Settings​

Image Config

Under the hood, the inference engine being launched is a vLLM container by default. vLLM is a highly optimized model serving and inference engine library. More information on vLLM can be found here: https://docs.vllm.ai/en/latest/.

Under Image config you can for example set whatever docker ID you would like to deploy. For example: vllm/vllm-openai:v0.26.0 maps to:

  • Image: vllm-openai
  • Custom registry URL: vllm
  • Version: v0.26.0

Arguments

In the expanded advanced settings, you can set additional parameters for the inference engine server. More details in the Advanced Settings section below. For now, we simply add 2 additional flags.

Near the bottom, you will see a set of fields for Arguments. These are inference engine server arguments passed to the container initialization step.

  • --trust-remote-code: A Hugging Face setting to override AutoModelForCausalLM class. It is typical for many models to have their own model classes in Hugging Face along with custom tokenizer logic. This flag allows the inference engine to run Python code from the model repository (pulled from Hugging Face using the model ID) to do so.
    • It is strongly recommended that models to be deployed are reviewed for trustworthiness to prevent malicious attacks. Models from well-known AI companies such as Meta, Qwen, and so on are generally safe.
    • To determine whether a model requires the trust-remote-code flag, reference the model card on Hugging Face and on vLLM documentation.

Further vLLM engine parameters can be found here: https://docs.vllm.ai/en/v0.26.0/configuration/engine_args/

  • Note: the default vLLM image version in our stack (at the time of this writing) is v0.26.0.

vLLM Limitations​

vLLM is an optimized inference engine for production environments. However, there are known limitations of vLLM to be aware of when choosing models. There is no universal engine that provides optimized inference for production environments for all possible types of machine learning models.

Supported models by vLLM are listed here: https://docs.vllm.ai/en/latest/models/supported_models.html

NB: Sometimes the very latest models from the latest release of vLLM can only be found on the release notes. https://github.com/vllm-project/vllm/releases

  • vLLM generally supports autoregressive transformer models with text outputs.
    • Diffusion-based models that generate images or videos are not supported.
    • Multimodal inputs however are supported for many models.
  • While vLLM supports most major model families, some niche models may not be supported.
  • Not all major models are supported from day 0. However, model support usually comes within a few weeks.
  • The most recent models often contain bugs that may affect performance. Usually these are worked on iteratively by the vLLM community, especially if the model demand is high. Generally it is recommended to reserve the most recent models from production environments until tested.