Air-Gapped Installation

How to install and operate CM Platform in an environment with no internet access. This guide covers what differs from the standard Platform Deployment flow — read that guide first; every step not mentioned here is unchanged.

How it works

A standard installation pulls artifacts from public sources: container images from public registries, dependency Helm charts from public chart repositories, the GitOps repository from GitHub, and model weights from HuggingFace. An air-gapped installation replaces all four with sources inside your environment:

Artifact	Standard source	Air-gapped source
Container images	Public registries + CM registry	Your OCI registry, pre-seeded
Dependency Helm charts	Public chart repositories	Your OCI registry (as OCI charts), pre-seeded
GitOps repository	CM's GitHub	Your git server, mirrored
Model weights	HuggingFace (downloaded at deploy time)	Pre-staged on the GPU nodes' local disk

The platform is retargeted at install time through configuration only — no code or chart edits. Everything below funnels into a handful of values (dependencies.repoMirror.*) plus a GitOps repository URL.

Prerequisites (in addition to the standard ones)

Your environment must provide:

1. An OCI registry reachable from the cluster, holding the CM release artifact set (see Preparing artifacts below). Both container images and Helm charts are served from it. If the registry requires authentication for pulls, provide a read-only account in three places: dependencies.repoMirror.auth in the values file (covers all Helm chart pulls, including the bootstrap), node-level containerd credentials for image pulls (on k3s, a configs: entry for the registry host in /etc/rancher/k3s/registries.yaml on every node), and cm-images.json (application images, as below).
2. A git server hosting a mirror of the CM platform-gitops repository, reachable from the cluster over HTTPS (a read-only account or deploy token is sufficient).
3. Cluster DNS that resolves your in-environment hostnames (registry, git server) from inside pods, and fails external lookups fast. The default CoreDNS forwarder pointed at a now-unreachable upstream makes every external lookup hang ~5 s, which degrades several components. On k3s: add a coredns-custom ConfigMap for your in-env names and remove the dead forward from the Corefile so unknown names return immediately. Note that k3s regenerates the CoreDNS ConfigMap on every k3s restart — codify the Corefile change (e.g. via your provisioning) rather than relying on a one-off edit.
4. CA trust for the registry and git endpoints on the nodes and in the cluster — or publicly-trusted certificates. A self-signed git server certificate can alternatively be accepted per-repo with gitRepositories.gitops.https.insecure: true.
5. TLS certificates for the platform itself must be self-signed or bring-your-own — Let's Encrypt requires internet access. Note that self-signed leaf certificates longer than 398 days are rejected by macOS clients and Chrome; keep tls.selfSigned.durationMonth ≤ 13. With self-signed certificates, distribute the platform CA to admin workstations (extract ca.crt from the cm-selfsigned-ca secret in the cert-manager namespace and add it to the OS trust store) — browsers only exempt interstitial-accepted hostnames one at a time, so portal API calls to sibling subdomains fail until the CA is trusted.

:::warning k3s and the removed default route If the egress cut removes the nodes' default route entirely, k3s ≥ v1.30 fails to (re)start with no default routes found in "/proc/net/route" — its node-IP autodetection requires one. Pin the addresses explicitly in /etc/rancher/k3s/config.yaml on every node before cutting the network:

node-ip: <node-ip>
flannel-iface: <interface>

A running cluster does not show this until the next k3s restart or node reboot. :::

Preparing artifacts

Artifact preparation happens outside the air gap and is driven by CM's delivery process. Agree on your registry base (<registry>/<project>) with CM up front — it is baked into every image and chart reference. The release ships two artifact manifests (BOMs) pinning every third-party image and chart the platform can pull; use them where your transfer channel requires a scan manifest. What arrives through the channel:

1. Registry content. CM seeds your registry with the full release set — or provides the seed script and a read-only source token where your process requires you to run the transfer yourself. The layout the platform expects: third-party images at <registry>/<project>/mirror/<source-path>, application images at <registry>/<project>/<name>:<hash>, dependency charts at oci://<registry>/<project>/charts/<name>, and the installer chart at oci://<registry>/<project>/helm/cm-platform.

2. The GitOps repository, mirrored into your git server. The branch the platform tracks is rendered by CM for your environment with the registry pointers baked in.

:::note Renders and the registry move together Every update to the GitOps branch pins application images by new content hashes, so a branch update is always delivered together with a refreshed registry seed. If the branch in your git server ever gets ahead of the registry content, pods reference images the registry doesn't hold yet. :::

3. Model weights for the models you plan to serve, staged under /mnt/llm_cache/models/<name> on every GPU node (see Model weights).

4. The installer inputs. A handful of files are needed on the installing workstation itself, outside the registry/git delivery: the two CRD manifests from the standard guide's Install CRDs step, example values profiles, and the installer chart. CM provides these together as the air-gap bundle (cm-airgap-bundle-<version>.tar.gz, with a SHA256SUMS integrity manifest) — bring it in through your transfer channel alongside the images and weights. If your process allows fetching them yourself instead, they come from their original locations: the CRD manifests from the URLs in the standard guide, and the installer chart via helm pull from CM's public registry or from your seeded in-env registry (oci://<registry>/<project>/helm/cm-platform).

:::tip Single installation tarball CM can package the workstation-side artifacts as one installation tarball (cm-airgap-install-<version>.tar.gz) instead of separate pieces. It contains the air-gap bundle above, the GitOps branch as a git bundle, the transfer manifest (artifact-list.csv, type,source,target), a pre-filled repoMirror values overlay, a node image-mirror example, and an INSTALL.md — under one SHA256SUMS. The large artifacts are delivered separately: the registry content (mirrored from the manifest with your own tooling) and the model weights. :::

Installing

Follow the standard deployment guide with these deviations:

• Install the CRDs from local files (from the air-gap bundle's crds/ directory, or wherever you fetched them) instead of the public URLs:

  kubectl create -f crds/argocd-application-crd.yaml
  kubectl create -f crds/prometheus-servicemonitor-crd.yaml

• Add the air-gap block to your values file:

  dependencies:
    repoMirror:
      # All third-party images resolve to <imageRegistry>/<source-path>
      imageRegistry: <registry>/<project>/mirror
      # All dependency charts (and the ArgoCD bootstrap) resolve here
      chartRegistry: oci://<registry>/<project>/charts
      # Only if the registry requires authenticated pulls (read-only account).
      # Covers chart pulls; image pulls use node-level containerd credentials.
      auth:
        username: "<robot-account>"
        password: "<robot-password>"
  
  gitRepositories:
    gitops:
      url: "https://<git-server>/<group>/platform-gitops.git"
      revision: "<your-branch>"
      https:
        username: "<deploy-token-username>"
        token: "<deploy-token>"
        insecure: true   # only if the git server cert is self-signed
  
  tls:
    certificateSource: selfsigned   # or "own", or "customTlsSecret" for a corporate CA
    selfSigned:
      durationMonth: 12
    # For "customTlsSecret", name a kubernetes.io/tls Secret that you have
    # already created in the istio-ingress namespace:
    # customTlsSecret:
    #   name: cm-platform-tls-<my-custom>

  With customTlsSecret the certificate stays outside the platform — you create and renew the Secret yourself, before you install. See TLS Settings.

• cm-images.json should contain pull credentials for your registry (a read-only robot/service account) instead of CM's registry.

• Install from the local chart instead of the public OCI reference:

  helm upgrade --install cm-platform ./cm-platform-<version>.tgz \
    --namespace cm-platform --create-namespace \
    --values basic-values.yaml \
    --set-file imageRegistries.cmApps.auth=image-registries-auth/cm-images.json

  (gitRepositories.gitops.accessKey is not needed — HTTPS credentials replace the SSH key.)

Post-deployment verification is unchanged: watch the ArgoCD applications go Synced/Healthy. Since there is no public DNS inside the gap, remember that the wildcard platform domain must resolve to the ingress IP through your in-environment DNS.

OpenShift

On OpenShift the install differs from the k3s-oriented steps above in how images are redirected and how node credentials and CA trust are supplied. Cluster prerequisites (OpenShift Service Mesh 3, NFD, the MachineConfigs) are in OpenShift/OKD Prerequisites.

• Image redirection is node-level, via ImageDigestMirrorSet / ImageTagMirrorSet — the OpenShift analog of the k3s containerd mirror, not a rewrite of image references. Image references keep their release names (confidentialmind.azurecr.io/main/...) and the node redirects the pull to your registry. Map confidentialmind.azurecr.io/main → your base; because CM images are tag-pinned and some third-party (tekton) references are digest-pinned, you need both an ImageTagMirrorSet (tags) and an ImageDigestMirrorSet (digests). The pre-root-app bootstrap ArgoCD also pulls quay.io/argoproj/*, so map quay.io (and any other upstream hosts) to your mirror path too, or it fails to pull before the platform ArgoCD exists.

• The recommended registry layout preserves the source path under an org prefix — e.g. <registry>/<org>/confidentialmind.azurecr.io/main/... — so the mirror map is a single prefix and the shipped GitOps branch stays stock (no per-customer render). This is the layout oc-mirror produces.

• The air-gap values differ accordingly: leave image references at the release mirror (the node redirects them) and name your registry only for charts, which ArgoCD's Helm client pulls directly, bypassing the node mirror:

  dependencies:
    repoMirror:
      imageRegistry: confidentialmind.azurecr.io/main/mirror   # redirected at the node to your registry
      chartRegistry: oci://<registry>/<org>/confidentialmind.azurecr.io/main/charts
      auth: { username: "<robot>", password: "<robot-pass>" }

• Node pull credentials go in the cluster global pull secret (openshift-config), not a containerd configs: entry. CA trust for a self-signed registry goes in additionalTrustedCA on image.config.openshift.io, not the node OS trust store.

• Verify redirection coverage — every platform image covered by a mirror of the correct pull type — with tools/scripts/airgap-verify.sh --platform ocp.

:::note Evolving The OpenShift path is being validated end to end; expect refinements here as it is exercised against live clusters. :::

Model weights

Air-gapped model serving uses the platform's pre-staged local path model source: weights are read directly from the node's disk instead of being downloaded.

• Stage each model under the model cache root (default /mnt/llm_cache, configurable via dependencies.ome.modelCacheBasePath): e.g. /mnt/llm_cache/models/<name>/.
• The directory must contain the complete HuggingFace-format model — including config.json, which the platform reads for model metadata. A useful pattern: hf download <org>/<model> --local-dir <staging>/<name>, then transfer the directory.
• Stage the weights on every GPU node the deployment can be scheduled on — the path is node-local, not shared storage.
• In the portal's deploy form, choose Pre-staged local path as the model source and enter the staged directory path.

Current limitations

• External / proxied models are incompatible with air gap — model endpoints that proxy an external provider need internet access and will fail.
• In-platform application building (Tekton pipelines) still targets the CM registry for build artifacts and is not yet retargetable to an in-env registry.