Air-Gapped Installation

How to install and operate CM Platform in an environment with no internet access. This guide covers what differs from the standard Platform Deployment flow — read that guide first; every step not mentioned here is unchanged.

How it works

A standard installation pulls artifacts from public sources: container images from public registries, dependency Helm charts from public chart repositories, the GitOps repository from GitHub, and model weights from HuggingFace. An air-gapped installation replaces all four with sources inside your environment:

Artifact	Standard source	Air-gapped source
Container images	Public registries + CM registry	Your OCI registry, pre-seeded
Dependency Helm charts	Public chart repositories	Your OCI registry (as OCI charts), pre-seeded
GitOps repository	CM's GitHub	Your git server, or a bare repository on any SSH host
Model weights	HuggingFace (downloaded at deploy time)	Pre-staged on the GPU nodes' local disk

The platform is retargeted at install time through configuration only — no code or chart edits. Everything below funnels into a handful of values (dependencies.repoMirror.*) plus a GitOps repository URL.

Prerequisites (in addition to the standard ones)

Your environment must provide:

1. An OCI registry reachable from the cluster, holding the CM release artifact set (see Preparing artifacts below). Both container images and Helm charts are served from it. If the registry requires authentication for pulls, provide a read-only account in three places: dependencies.repoMirror.auth in the values file (covers all Helm chart pulls, including the bootstrap), node-level containerd credentials for image pulls (on k3s, a configs: entry for the registry host in /etc/rancher/k3s/registries.yaml on every node), and cm-images.json (application images, as below).

2. A git host serving the CM platform-gitops branch, reachable from the cluster: either a git server reached over HTTPS (a read-only account or deploy token is sufficient), or any host reached over SSH: your own git server, or a host with sshd and git that holds a bare copy of the repository. Over SSH the platform also needs the host's SSH keys (see GitOps repository without a git server).

3. Cluster DNS that resolves your in-environment hostnames (registry, git server) from inside pods, and fails external lookups fast. The default CoreDNS forwarder pointed at a now-unreachable upstream makes every external lookup hang ~5 s, which degrades several components. On k3s: add a coredns-custom ConfigMap for your in-env names and remove the dead forward from the Corefile so unknown names return immediately. Note that k3s regenerates the CoreDNS ConfigMap on every k3s restart — codify the Corefile change (e.g. via your provisioning) rather than relying on a one-off edit.

4. CA trust for the registry and git endpoints — or publicly-trusted certificates. A private CA has to be trusted in two places: by the nodes, which pull the images, and inside the installer's pre-install hook and ArgoCD, which fetch charts and the GitOps branch with their own clients and never see node trust. The same goes for the services that reach the MCP servers and external model endpoints your users register, when those are served under your CA. For all of these, give the installer the CA certificates in PEM form:

   trustedCABundle:
     certificates: |
       -----BEGIN CERTIFICATE-----
       ...
       -----END CERTIFICATE-----

   or pass a file with --set-file trustedCABundle.certificates=ca.pem. ArgoCD uses this bundle instead of its built-in roots, so it must contain every CA ArgoCD has to trust: your CA, and the public roots as well if any of those hosts has a publicly trusted certificate. On OpenShift, let the cluster supply the bundle instead; see below. Pods that use the bundle wait until it exists rather than fail, so an install started too early looks stuck: a pod stays in ContainerCreating, and kubectl describe on it names the missing ca-bundle.crt. As a last resort, gitRepositories.gitops.https.insecure: true makes the platform accept a git server certificate it cannot verify. That turns off TLS verification of the GitOps source entirely, so trust the CA instead where you can.

5. TLS certificates for the platform itself must be self-signed or bring-your-own — Let's Encrypt requires internet access. Note that self-signed leaf certificates longer than 398 days are rejected by macOS clients and Chrome; keep tls.selfSigned.durationMonth ≤ 13. With self-signed certificates, distribute the platform CA to admin workstations (extract ca.crt from the cm-selfsigned-ca secret in the cert-manager namespace and add it to the OS trust store) — browsers only exempt interstitial-accepted hostnames one at a time, so portal API calls to sibling subdomains fail until the CA is trusted. That CA is valid for 10 years, so distributing it is normally a one-time step; the 398-day limit above applies to the platform leaf certificate browsers check, not to the CA. cert-manager renews the CA a year before it expires — after that renewal, extract ca.crt again and replace it in the trust store on every workstation.

:::warning k3s and the removed default route If the egress cut removes the nodes' default route entirely, k3s ≥ v1.30 fails to (re)start with no default routes found in "/proc/net/route" — its node-IP autodetection requires one. Pin the addresses explicitly in /etc/rancher/k3s/config.yaml on every node before cutting the network:

node-ip: <node-ip>
flannel-iface: <interface>

A running cluster does not show this until the next k3s restart or node reboot. :::

Preparing artifacts

Artifact preparation happens outside the air gap and is driven by CM's delivery process. Agree on your registry base (<registry>/<project>) with CM up front — it is baked into every image and chart reference. The release ships two artifact manifests (BOMs) pinning every third-party image and chart the platform can pull; use them where your transfer channel requires a scan manifest. What arrives through the channel:

1. Registry content. CM seeds your registry with the full release set — or provides the seed script and a read-only source token where your process requires you to run the transfer yourself. The layout the platform expects: third-party images at <registry>/<project>/mirror/<source-path>, application images at <registry>/<project>/<name>:<hash>, and dependency charts at oci://<registry>/<project>/charts/<name>. The installer chart is not part of the registry content; it arrives with the installer inputs below.

   If your registry sits behind a reverse proxy or ingress, check how long the proxy allows a single request. Traefik v3, for example, closes a request after 60 seconds by default. A large image layer uploaded in one request can take longer than that and fails with HTTP 499 or 504, while smaller layers succeed. Use a transfer tool that uploads layers in chunks, such as regctl with --blob-chunk, or raise the proxy's limit for the transfer.

2. The GitOps repository, mirrored into your git server or onto your SSH host. The branch the platform tracks is rendered by CM for your environment with the registry pointers baked in.

:::note Renders and the registry move together Every update to the GitOps branch pins application images by new content hashes, so a branch update is always delivered together with a refreshed registry seed. If the branch in your git server ever gets ahead of the registry content, pods reference images the registry doesn't hold yet. :::

3. Model weights for the models you plan to serve, staged under /mnt/llm_cache/models/<name> on every GPU node (see Model weights).

4. The installer inputs. A handful of files are needed on the installing workstation itself, outside the registry/git delivery: the two CRD manifests from the standard guide's Install CRDs step, example values profiles, and the installer chart. CM provides these together as the air-gap bundle (cm-airgap-bundle-<version>.tar.gz, with a SHA256SUMS integrity manifest) — bring it in through your transfer channel alongside the images and weights. If your process allows fetching them yourself instead, they come from their original locations: the CRD manifests from the URLs in the standard guide, and the installer chart via helm pull from CM's public registry (oci://confidentialmindpub.azurecr.io/cm-platform).

   Use the chart version that matches your release: chart X.Y.Z for release vX.Y.Z (helm pull ... --version X.Y.Z). A release's air-gap bundle carries the chart of the release candidate it was promoted from, cm-platform-X.Y.Z-rc.N.tgz. It has the same templates, but with gitRepositories.gitops.revision unset it looks for the vX.Y.Z-rc.N tag, so keep revision set to the branch CM rendered for your environment, as the values below show.

:::tip Single installation tarball CM can package the workstation-side artifacts as one installation tarball (cm-airgap-install-<version>.tar.gz) instead of separate pieces. It contains the air-gap bundle above, the GitOps branch as a git bundle, the transfer manifest (artifact-list.csv, type,source,target), a pre-filled repoMirror values overlay, a node image-mirror example, and an INSTALL.md — under one SHA256SUMS. The large artifacts are delivered separately: the registry content (mirrored from the manifest with your own tooling) and the model weights. :::

Installing

Follow the standard deployment guide with these deviations:

• Install the CRDs from local files (from the air-gap bundle's crds/ directory, or wherever you fetched them) instead of the public URLs:

  kubectl create -f crds/argocd-application-crd.yaml
  kubectl create -f crds/prometheus-servicemonitor-crd.yaml

• Add the air-gap block to your values file:

  dependencies:
    repoMirror:
      # All third-party images resolve to <imageRegistry>/<source-path>
      imageRegistry: <registry>/<project>/mirror
      # All dependency charts (and the ArgoCD bootstrap) resolve here
      chartRegistry: oci://<registry>/<project>/charts
      # Only if the registry requires authenticated pulls (read-only account).
      # Covers chart pulls; image pulls use node-level containerd credentials.
      auth:
        username: "<robot-account>"
        password: "<robot-password>"
  
  gitRepositories:
    gitops:
      url: "https://<git-server>/<group>/platform-gitops.git"
      # The branch CM rendered for your environment. Unset, the chart looks for its release tag.
      revision: "<your-branch>"
      https:
        username: "<deploy-token-username>"
        token: "<deploy-token>"
        # insecure: true turns off TLS verification of the GitOps source.
        # Use it only if the cluster cannot verify the git server's certificate.
  
  # Only when the registry or git server uses a private CA (see prerequisite 4).
  # On OpenShift, use openshiftInjected: true instead of certificates.
  trustedCABundle:
    certificates: |
      -----BEGIN CERTIFICATE-----
      ...
      -----END CERTIFICATE-----
  
  tls:
    certificateSource: selfsigned   # or "own", or "customTlsSecret" for a corporate CA
    selfSigned:
      durationMonth: 12
    # For "customTlsSecret", name a kubernetes.io/tls Secret that you have
    # already created in the istio-ingress namespace:
    # customTlsSecret:
    #   name: cm-platform-tls-<my-custom>

  With customTlsSecret the certificate stays outside the platform — you create and renew the Secret yourself, before you install. See TLS Settings.

• cm-images.json should contain pull credentials for your registry (a read-only robot/service account) instead of CM's registry.

• Install from the local chart instead of the public OCI reference:

  helm upgrade --install cm-platform ./cm-platform-<version>.tgz \
    --namespace cm-platform --create-namespace \
    --values basic-values.yaml \
    --set-file imageRegistries.cmApps.auth=image-registries-auth/cm-images.json

  (gitRepositories.gitops.accessKey is not needed — HTTPS credentials replace the SSH key. To serve the repository over SSH instead, see the next section.)

Post-deployment verification is unchanged: watch the ArgoCD applications go Synced/Healthy. Since there is no public DNS inside the gap, remember that the wildcard platform domain must resolve to the ingress IP through your in-environment DNS.

GitOps repository without a git server

ArgoCD only reads the GitOps branch, so it does not need a git server. Any host the cluster's pods can reach over SSH can serve the branch, with nothing installed beyond sshd and git. The host keeps the branch as a bare repository, the form a git server stores, which has no checked-out files. ArgoCD fetches it over SSH with a key of its own.

A git server you reach over SSH needs steps 3 and 4 as well; use the deploy key it gives you instead of the key pair in step 2.

1. On the host, create a user that is limited to git operations, and turn the delivered git bundle (in the installation tarball's gitops/ directory) into a bare repository:

   sudo useradd -m -s "$(command -v git-shell)" git
   sudo -u git git clone --bare -b <your-branch> /path/to/platform-gitops-<your-branch>.bundle \
     /home/git/platform-gitops.git

   Where sshd runs without PAM, as on Alpine, it treats an account without a password as locked and refuses its key logins too. Unlock the account with sudo usermod -p '*' git.

2. Create a key pair for ArgoCD, then append the public key argocd-gitops.pub to /home/git/.ssh/authorized_keys on the host (directory mode 700, file mode 600, both owned by git):

   ssh-keygen -t ed25519 -N '' -C argocd-gitops -f argocd-gitops

3. Collect the host's SSH keys. ArgoCD verifies the host it connects to and trusts only the public git forges by default, so it needs these keys in known_hosts format:

   ssh-keyscan -t ed25519 <host> > known_hosts
   ssh-keygen -lf known_hosts   # compare with ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub on the host

   Scan only the key type you compare. ArgoCD accepts any key listed for the host, so a key you did not check is trusted as much as one you did.

   The host name must be written as it appears in the repository URL. For a port other than 22, use ssh-keyscan -p <port>, which writes the [<host>]:<port> form. The installer stops with an error when knownHosts has no line for the host in the URL.

4. Point the install at the repository, and pass the key and the host keys to the helm command:

   gitRepositories:
     gitops:
       url: "ssh://git@<host>/home/git/platform-gitops.git"
       revision: "<your-branch>"

     --set-file gitRepositories.gitops.accessKey=argocd-gitops \
     --set-file gitRepositories.gitops.ssh.knownHosts=known_hosts

ArgoCD must trust the host keys before url points at the host. The bootstrap ArgoCD receives them on the first install only; after that, ArgoCD reads its own configuration from the repository, which it cannot fetch from a host it does not trust. The installer checks that knownHosts names the host, but not the keys themselves: a first install with the wrong keys leaves the root application reporting knownhosts: key mismatch. To recover:

1. Run the helm upgrade again with the correct known_hosts file. This puts the keys in the configuration ArgoCD will read from the repository.

2. Replace the wrong lines in the ssh_known_hosts entry of ArgoCD's known-hosts ConfigMap with the correct ones, so that ArgoCD can reach the repository now:

   kubectl -n argocd edit configmap argocd-ssh-known-hosts-cm

   The change reaches ArgoCD within a few minutes. Once ArgoCD has fetched the repository, it manages the ConfigMap itself, with the same keys.

On a running platform, make changes in an order that keeps the repository reachable. To move to an SSH host from another URL, first add knownHosts in one helm upgrade while still on the old URL, wait for the argocd application to sync and a few more minutes for the keys to reach ArgoCD, then change url and the credentials in a second upgrade. To rotate the host's keys, add the new keys alongside the old ones the same way before the host starts serving them, and remove the old ones afterwards. Do not edit the ConfigMap on a running platform: ArgoCD reverts it.

To deliver an update, fetch the new bundle into the bare repository:

sudo -u git git -C /home/git/platform-gitops.git fetch \
  /path/to/platform-gitops-<new-branch>.bundle '+refs/heads/*:refs/heads/*'

Then run the install command again with revision set to the new branch. Pass the key and the host keys again, as on the install: helm does not keep values from earlier runs, and an upgrade without them removes them, which cuts ArgoCD off from the repository.

helm upgrade --install cm-platform ./cm-platform-<version>.tgz \
  --namespace cm-platform \
  --values basic-values.yaml \
  --set gitRepositories.gitops.revision=<new-branch> \
  --set-file imageRegistries.cmApps.auth=image-registries-auth/cm-images.json \
  --set-file gitRepositories.gitops.accessKey=argocd-gitops \
  --set-file gitRepositories.gitops.ssh.knownHosts=known_hosts

OpenShift

On OpenShift the install differs from the k3s-oriented steps above in how images are redirected and how node credentials and CA trust are supplied. Cluster prerequisites (OpenShift Service Mesh 3, NFD, the MachineConfigs) are in OpenShift/OKD Prerequisites.

• Image redirection is node-level, via ImageDigestMirrorSet / ImageTagMirrorSet — the OpenShift analog of the k3s containerd mirror, not a rewrite of image references. Image references keep their release names (confidentialmind.azurecr.io/main/...) and the node redirects the pull to your registry. Map confidentialmind.azurecr.io/main → your base. Every image the platform pulls today is tag-pinned, so the ImageTagMirrorSet is what carries the install; apply the ImageDigestMirrorSet alongside it anyway, because adding one later costs another node roll. Two operators still reference their upstream images directly — the Keycloak operator (quay.io) and the CNPG pgBackRest plugin (docker.io) — so map quay.io and docker.io to your mirror path (<base>/mirror) as well, or those two fail to pull.

• The recommended registry layout preserves the source path under an org prefix — e.g. <registry>/<org>/confidentialmind.azurecr.io/main/... — so the mirror map is a single prefix and the shipped GitOps branch stays stock (no per-customer render). This is the layout oc-mirror produces.

• The air-gap values differ accordingly: leave image references at the release mirror (the node redirects them) and name your registry only for charts, which ArgoCD's Helm client pulls directly, bypassing the node mirror:

  dependencies:
    repoMirror:
      imageRegistry: confidentialmind.azurecr.io/main/mirror   # redirected at the node to your registry
      chartRegistry: oci://<registry>/<org>/confidentialmind.azurecr.io/main/charts
      auth: { username: "<robot>", password: "<robot-pass>" }

  The OpenShift values profile already has dependencies: and dns: blocks. Add these keys inside them: a second top-level dependencies: or dns: key in the same file replaces the first, and the profile's settings are lost without an error.

• Node pull credentials go in the cluster global pull secret (openshift-config), not a containerd configs: entry.

• A private CA goes in the cluster-wide trust: a user-ca-bundle ConfigMap in openshift-config, named as the proxy's trustedCA. OpenShift adds it to node trust, which covers image pulls, and injects it — merged with the system roots — into any ConfigMap labelled config.openshift.io/inject-trusted-cabundle. Let the installer create those for the pre-install hook, ArgoCD and the services that connect to MCP servers:

  trustedCABundle:
    openshiftInjected: true

  Use this or certificates, not both; the install stops if both are set. If you install before the cluster has filled the bundle, the install waits instead of failing: the pre-install hook pod stays in ContainerCreating until ca-bundle.crt exists. If it never appears, check that user-ca-bundle is the ConfigMap the proxy's trustedCA names.

  manager-go and model-gateway, which reach MCP servers and external model endpoints, wait the same way for the bundle in api-services. To check whether the cluster has filled it in:

  oc get configmap trusted-ca-bundle -n api-services -o jsonpath='{.data.ca-bundle\.crt}'

  Use openshiftInjected on OpenShift only. On other clusters nothing fills the bundle, and these pods never start.

  additionalTrustedCA on image.config.openshift.io covers image pulls only; it is not injected, so ArgoCD would still reject the git server and chart registry.

  On a running cluster the CA reaches node trust only in a second node roll, which starts after the first one finishes — the Machine Config Operator picks the new trustedCA up late. Wait until oc get controllerconfig machine-config-controller -o jsonpath='{.spec.additionalTrustBundle}' is non-empty and both oc get mcp pools show UPDATED=True before installing.

• Verify redirection coverage — every platform image covered by a mirror of the correct pull type — with tools/scripts/airgap-verify.sh --platform ocp. On OpenShift the script skips the openshift-* and kube-* namespaces, because those images come from OpenShift itself and are not the platform's to mirror. If your cluster runs other software that brings its own images — a storage (CSI) driver, for example — name its namespaces so they are skipped as well:

  tools/scripts/airgap-verify.sh --platform ocp --exclude-ns <namespace>,<namespace>

  Separate several namespaces with commas; * is allowed in a name. The script lists the namespaces it skipped, so you can check they are the ones you meant.

:::note Evolving The OpenShift path is being validated end to end; expect refinements here as it is exercised against live clusters. :::

Adding a private CA to an installed platform

You can add trustedCABundle to a platform that is already running, or change it later: set it in your values file and run the same helm upgrade --install command again. Both options work this way. ArgoCD picks up a changed bundle within about a minute. manager-go and model-gateway read it when they start, so restart them after a change:

kubectl -n api-services rollout restart deploy/manager-go deploy/model-gateway

ArgoCD reads the new configuration from your git server, so it still has to reach that server during this upgrade. If it reaches it today only because gitRepositories.gitops.https.insecure is set, keep that setting for this upgrade and remove it in a second one.

On OpenShift with openshiftInjected: true, add your CA to the proxy's trustedCA first and wait for the node rolls described above.

Model weights

Air-gapped model serving uses the platform's pre-staged local path model source: weights are read directly from the node's disk instead of being downloaded.

• Stage each model under the model cache root (default /mnt/llm_cache, configurable via dependencies.ome.modelCacheBasePath): e.g. /mnt/llm_cache/models/<name>/.
• The directory must contain the complete HuggingFace-format model, including config.json, which the platform reads for model metadata. For embedding models, include the model's tokenizer files, such as tokenizer.json, tokenizer_config.json and any required vocabulary files. A RAG endpoint copies its embedding model's tokenizer from this directory. Missing tokenizer files prevent the model from becoming ready and a RAG endpoint that uses it from starting. A useful pattern: hf download <org>/<model> --local-dir <staging>/<name>, then transfer the directory.
• Stage the weights on every GPU node the deployment can be scheduled on — the path is node-local, not shared storage.
• In the portal's deploy form, choose Pre-staged local path as the model source and enter the staged directory path.

Current limitations

• External / proxied models are incompatible with air gap — model endpoints that proxy an external provider need internet access and will fail.