Skip to main content

GPU Resource Management and Model Deployment

GPU Resource Dashboard​

The GPU Resource Dashboard provides a comprehensive view of your GPU resources across all nodes in the ConfidentialMind stack. This interface helps administrators monitor VRAM usage, identify available resources, and understand how models are distributed across the infrastructure.

GPU Resource Dashboard

Key Features​

  • Node-level Overview: View all available nodes with their GPU specifications
  • VRAM Utilization: Monitor total and free VRAM for each node and individual GPU
  • Model Allocation: See which models are deployed on which GPUs
  • Resource Metrics: Track VRAM reservations with color-coded visualization

Using the Dashboard​

Understanding the Node View​

Each node section displays:

  • Node Name: The identifier for the physical server (e.g., confidentialmind-horizon-1)
  • Available Resources: Total and free VRAM across all GPUs in the node (e.g., 10GB / 160GB free)
  • GPU Details: Information about each GPU including type and memory capacity

GPU Allocation View​

For each GPU within a node:

  • GPU Identifier: GPU number and specifications (e.g., GPU 0: NVIDIA H100 80GB HBM3)
  • Deployed Models: Visualized as colored blocks with model names
  • VRAM Allocation: The amount of VRAM reserved for each model
  • Free VRAM: Available memory shown at the end of each GPU bar

Model Deployment Status​

Each model block shows:

  • Model Name: Full name of the deployed model
  • VRAM Usage: Amount of memory reserved for this model instance
  • Deployment Health: Indicated by color (green for healthy)

Default Model Configuration​

The Default Model Settings interface allows administrators to configure which models should be automatically selected for different AI tasks when creating new deployments.

Default Model Settings

Key Features​

  • Task-specific Model Selection: Configure default models for different AI tasks
  • Quick Configuration: Simple dropdown interface for model selection
  • Standardization: Ensure consistency across new deployments

Configuration Options​

  1. Embedding Model: The default model used for generating vector embeddings

    • Example: gte-large-embeddings-GPU

  2. Reranker Model: The default model used for reranking search or retrieval results

    • Example: bge-reranker-large

  3. Completion Model: The default model used for text generation and completion tasks

    • Example: Llama 3.3 70B 4bit

Managing Default Models​

  1. Navigate to the Models section in the left sidebar
  2. Click "Configure defaults" in the top menu
  3. Select appropriate models for each task type from the dropdown menus
  4. Click "Save" to apply these defaults to future deployments

Model Deployment Interface​

The Model Deployment interface provides a step-by-step workflow for deploying AI models to your infrastructure with fine-grained control over resource allocation and performance parameters.

Model Deployment Interface

Key Features​

  • Guided Deployment: Step-by-step process with clear navigation
  • Node & GPU Selection: Choose specific hardware for deployment
  • Resource Optimization: Configure tensor parallelism and memory utilization
  • Advanced Settings: Fine-tune deployment parameters for optimal performance

Deployment Steps​

1. Basic Configuration​

  • Deployment Name: Provide a descriptive name for the model deployment
  • Node Selection: Choose which server node to deploy the model on
  • GPU Selection: Select specific GPUs within the chosen node

2. GPU Configuration​

  • Select GPUs: Choose one or more GPUs for deployment
  • CUDA Settings: Configure CUDA_VISIBLE_DEVICES to control which GPUs are visible to the model
  • Tensor Parallelism: Set TENSOR-PARALLEL-SIZE to determine how the model is distributed across GPUs

3. Model & Resource Settings​

  • Model Selection: Choose the model to deploy
  • GPU Memory Utilization: Adjust the percentage of GPU memory to allocate (typically 80-90%)
  • Advanced Settings: Configure additional parameters like quantization, batch size, and KV cache

Best Practices​

  1. Memory Allocation:

    • For inference workloads, 85-90% GPU utilization is typically optimal
    • Leave some headroom to prevent out-of-memory errors

  2. Parallelism Configuration:

    • For models > 30B parameters, use tensor parallelism across multiple GPUs
    • Match tensor parallelism size to the number of GPUs selected

  3. Node Selection:

    • Deploy related models to the same node to minimize network latency
    • Distribute high-traffic models across different nodes for load balancing

  4. Optimization Tips:

    • Use 4-bit or 8-bit quantization for larger models to reduce VRAM requirements
    • Adjust batch size based on expected request patterns
    • Consider inference requirements when selecting between different model sizes

Troubleshooting​

Common Issues​

  1. Insufficient VRAM:

    • Reduce model size or precision
    • Increase tensor parallelism
    • Move other models to different GPUs

  2. Deployment Failures:

    • Check GPU compatibility with model requirements
    • Ensure selected node has sufficient resources

  3. Long Startup or Repeated Restarts:

    • Large custom models can take up to approximately 90 minutes to load before the startup health check times out
    • Check the model server logs to confirm that weight or shard loading is progressing
    • If the logs start over after approximately 90 minutes, verify the model arguments, available resources, and model compatibility

  4. Performance Issues:

    • Adjust batch size for your workload patterns
    • Check for CPU bottlenecks in preprocessing
    • Consider GPU bus bandwidth limitations when using multiple GPUs