Tenstorrent Prerequisites

Complete these steps before enabling Tenstorrent support in CM Platform. Enabling it adds eligible Tenstorrent nodes to the existing model-serving infrastructure; NVIDIA GPU nodes remain eligible.

1. Install the Tenstorrent Kubernetes stack

Install and validate the Tenstorrent host software and Kubernetes components before installing CM Platform. The cluster must provide:

• Kubernetes 1.34 or newer with the stable resource.k8s.io/v1 Dynamic Resource Allocation (DRA) API
• a supported Tenstorrent kernel driver and firmware
• tt-operator, with Fabric Manager and the DRA driver enabled
• a ResourceSlice that publishes the node's devices with driver tenstorrent.com

Follow the Tenstorrent Cloud-Native Support documentation for installation and platform-specific requirements. CM Platform has been validated with tt-operator 0.3.0; use a Tenstorrent-supported configuration that provides the components above.

Verify the API and operator resources before continuing:

kubectl api-resources --api-group=resource.k8s.io
kubectl -n tt-operator-system get pods
kubectl get deviceclasses.resource.k8s.io
kubectl get resourceslices.resource.k8s.io

Do not enable Tenstorrent support until the operator pods are healthy and the Tenstorrent devices appear in a ResourceSlice.

Hardware required by the bundled runtime

The bundled Tenstorrent vLLM runtime supports Blackhole Galaxy hardware. Each model replica requires eight available devices from the same tray on one node. The platform allocates these devices together; users do not select individual devices in the portal.

This requirement applies to the bundled runtime, not to every Tenstorrent runtime. Follow the Tenstorrent documentation linked above to prepare the hardware.

2. Label the Tenstorrent nodes

CM Platform selects Tenstorrent nodes using the tenstorrent.com/has-tt=true label. This is a CM Platform marker and is not created by tt-operator, even when the operator's Node Feature Discovery component adds its own hardware labels.

Add the label to every Tenstorrent node that CM Platform may use:

kubectl label node <node-name> tenstorrent.com/has-tt=true

List the eligible nodes:

kubectl get nodes -l tenstorrent.com/has-tt=true

A node without this label is skipped: the model agent does not cache model weights there and model-serving workloads cannot use it through the Tenstorrent integration.

Optional node taint

To reserve a Tenstorrent node for workloads that explicitly tolerate it, use the tenstorrent-node taint key:

kubectl taint node <node-name> tenstorrent-node=true:NoSchedule

When Tenstorrent support is enabled, CM Platform's model agent tolerates this taint. The Tenstorrent Fabric Manager agent and DRA kubelet plugin must also be configured to tolerate it. A node without the taint works as well.

The platform does not add arbitrary tolerations for other custom NoSchedule taints. If you use another taint key, provide the corresponding tolerations to every component that must run on the node.

3. Prepare the model cache

The model agent stores weights in a node-local cache. The directory must exist or be creatable and writable on every eligible Tenstorrent node. The default path is /mnt/llm_cache; see Model Cache Path to override it.

4. Enable Tenstorrent support

Add the feature flag to the values file used to install CM Platform:

featureFlags:
  tenstorrent: true

Then install or upgrade the platform as described in the CM Platform Deployment Guide. The flag is off by default and enabling it adds Tenstorrent nodes without removing NVIDIA nodes from model-agent placement.

Tenstorrent support remains opt-in while CM Platform supports Kubernetes clusters without the stable resource.k8s.io/v1 DRA API. This compatibility flag is intended to be retired once Kubernetes 1.34 or newer and DRA v1 become platform-wide requirements.

5. Verify model-agent placement

After the Argo CD applications are synced and healthy, inspect the model-agent DaemonSet and its pods:

kubectl -n ome get daemonsets
kubectl -n ome get pods -o wide

A model-agent pod must run on every Tenstorrent node that may serve models. On a mixed cluster, model-agent pods should continue running on the eligible NVIDIA nodes as well.

If a Tenstorrent node has no model-agent pod, check its label, taints, operator components, DRA ResourceSlice, and model-cache directory before deploying a model to it.