GPU Resource Management and Model Deployment

Portal

Open Manage platform, then Models to manage deployments. The tenant Models page lists available models. Select By node to see GPU memory and model placement. Colors distinguish deployments, not deployment health.

Select Configure defaults on the platform Models page to choose default models for new RAG endpoints.

The dialog is titled Default models. Choose an Embedding model, Completion model, Reranker model, and OCR model, then select Save. Only models that are shared with all tenants can be chosen, so a model that is not shared does not appear in these lists.

The screenshots and navigation below refer to the legacy portal. Troubleshooting, at the end of this page, applies to both portals.

Legacy portal GPU resource dashboard

The GPU Resource Dashboard provides a comprehensive view of your GPU resources across all nodes in the ConfidentialMind stack. This interface helps administrators monitor VRAM usage, identify available resources, and understand how models are distributed across the infrastructure.

On OpenShift, this data is collected by OpenShift user workload monitoring. If no VRAM values are shown, ask a cluster administrator to check the OpenShift/OKD prerequisites.

Key Features

• Node-level Overview: View all available nodes with their GPU specifications
• VRAM Utilization: Monitor total and free VRAM for each node and individual GPU
• Model Allocation: See which models are deployed on which GPUs
• Resource Metrics: Track VRAM reservations with color-coded visualization

Using the Dashboard

Understanding the Node View

Each node section displays:

• Node Name: The identifier for the physical server (e.g., confidentialmind-horizon-1)
• Available Resources: Total and free VRAM across all GPUs in the node (e.g., 10GB / 160GB free)
• GPU Details: Information about each GPU including type and memory capacity

GPU Allocation View

For each GPU within a node:

• GPU Identifier: GPU number and specifications (e.g., GPU 0: NVIDIA H100 80GB HBM3)
• Deployed Models: Visualized as colored blocks with model names
• VRAM Allocation: The amount of VRAM reserved for each model
• Free VRAM: Available memory shown at the end of each GPU bar

Model Deployment Status

Each model block shows:

• Model Name: Full name of the deployed model
• VRAM Usage: Amount of memory reserved for this model instance
• Deployment Health: Indicated by color (green for healthy)

Legacy portal default model configuration

The Default Model Settings interface allows administrators to configure which models should be automatically selected for different AI tasks when creating new deployments.

Key Features

• Task-specific Model Selection: Configure default models for different AI tasks
• Quick Configuration: Simple dropdown interface for model selection
• Standardization: Ensure consistency across new deployments

Configuration Options

1. Embedding Model: The default model used for generating vector embeddings

   • Example: gte-large-embeddings-GPU

2. Reranker Model: The default model used for reranking search or retrieval results

   • Example: bge-reranker-large

3. Completion Model: The default model used for text generation and completion tasks

4. OCR Model: The default model used to read text from images and scanned PDFs while a RAG ingests documents

   • Only models typed OCR can be selected
   • Example: dots.ocr

If a model you chose earlier is not typed OCR, the OCR model field shows it as unavailable. Choose a model typed OCR, or set the field to None.

Managing Default Models

1. Navigate to the Models section in the left sidebar
2. Click "Configure defaults" in the top menu
3. Select appropriate models for each task type from the dropdown menus
4. Click "Save" to apply these defaults to future deployments

Legacy portal model deployment interface

The Model Deployment interface provides a step-by-step workflow for deploying AI models to your infrastructure with fine-grained control over resource allocation and performance parameters.

Key Features

• Guided Deployment: Step-by-step process with clear navigation
• Node & GPU Selection: Choose specific hardware for deployment
• Resource Optimization: Configure tensor parallelism and memory utilization
• Advanced Settings: Fine-tune deployment parameters for optimal performance

Deployment Steps

1. Basic Configuration

• Deployment Name: Provide a descriptive name for the model deployment
• Node Selection: Choose which server node to deploy the model on
• GPU Selection: Select specific GPUs within the chosen node

2. GPU Configuration

• Select GPUs: Choose one or more GPUs for deployment. Each GPU shows its free and total memory under its name, for example 11.7 / 24 GiB free. A GPU that is fully in use shows 0 / 24 GiB free. When the platform cannot read how much memory a GPU is using, only the total is shown, for example 24 GiB total.
• VRAM needed: When a preset states its fixed model size, the form shows how much VRAM one replica needs. Pick a GPU whose free memory is at least that amount. Presets without a fixed requirement do not show this line.
• CUDA Settings: Configure CUDA_VISIBLE_DEVICES to control which GPUs are visible to the model
• Tensor Parallelism: Set TENSOR-PARALLEL-SIZE to determine how the model is distributed across GPUs

3. Model & Resource Settings

• Model Selection: Choose the model to deploy
• GPU Memory Utilization: The percent of each GPU's memory the model may use. The field appears after you select at least one GPU, and starts at 90. It is hidden for presets that use platform-managed runtime configuration. For other deployments, enter a whole number between 1 and 100, or leave it blank to let the inference server use its own default. Any other value is rejected with a message under the field.
  • While a percent is set, a line under the field shows what it means on each selected GPU, for example GPU 0: 20.9 / 24 GiB. The percent applies to every selected GPU on its own, so GPUs of different sizes show different amounts.
  • When the amount is larger than the memory that is free on that GPU, the line becomes a warning, for example GPU 0: 20.9 GiB exceeds the 10 GiB free on this GPU. This is a warning only — you can still deploy. No warning is shown when the free memory of the GPU is unknown.
• Advanced Settings: Configure additional parameters like quantization, batch size, and KV cache

Best Practices

1. Memory Allocation:

   • For inference workloads, 85-90% GPU utilization is typically optimal
   • Leave some headroom to prevent out-of-memory errors

2. Parallelism Configuration:

   • For models > 30B parameters, use tensor parallelism across multiple GPUs
   • Match tensor parallelism size to the number of GPUs selected

3. Node Selection:

   • Deploy related models to the same node to minimize network latency
   • Distribute high-traffic models across different nodes for load balancing

4. Optimization Tips:

   • Use 4-bit or 8-bit quantization for larger models to reduce VRAM requirements
   • Adjust batch size based on expected request patterns
   • Consider inference requirements when selecting between different model sizes

Troubleshooting

Common Issues

1. Insufficient VRAM:

   • Stop an unused operator-managed model deployment to release its GPUs without deleting its configuration
   • Reduce model size or precision
   • Increase tensor parallelism
   • Move other models to different GPUs

2. Deployment Failures:

   • Check GPU compatibility with model requirements
   • Ensure selected node has sufficient resources

3. Long Startup or Repeated Restarts:

   • Large custom models can take up to approximately 90 minutes to load before the startup health check times out
   • Check the model server logs to confirm that weight or shard loading is progressing
   • If the logs start over after approximately 90 minutes, verify the model arguments, available resources, and model compatibility

4. Performance Issues:

   • Adjust batch size for your workload patterns
   • Check for CPU bottlenecks in preprocessing
   • Consider GPU bus bandwidth limitations when using multiple GPUs