Skip to main content

CM Platform Deployment Guide

A deployment guide for CM Platform on Kubernetes using Helm and ArgoCD.

Overview​

CM Platform Installer automates the deployment of Confidential Mind's platform on Kubernetes clusters using ArgoCD and Helm. It simplifies the deployment process through a Helm chart that sets up all required components with minimal configuration.

Installing into an environment without internet access? Read this guide first, then follow the deviations in Air-Gapped Installation.

Prerequisites​

CM Platform deployment requires:

1. Kubernetes Cluster​

  • A running Kubernetes cluster
  • A valid kubeconfig file with admin permissions
  • A CSI driver for persistent storage, optionally Longhorn CSI (if using Longhorn see prerequisites below)
  • amd64 nodes running model-serving components must support x86-64-v3 (AVX2-generation CPUs, roughly 2015 or newer); arm64 nodes have no such floor

For OpenShift/OKD: See OpenShift/OKD Prerequisites for additional node-level configuration required before deployment.

2. CM Repositories Access​

  • CM Platform GitOps Repository Access (explained in "Prepare Required Keys and Credentials" \ "CM Git Repository Access" )
  • CM OCI Registry Access (explained in "Prepare Required Keys and Credentials" \ "CM OCI Registry Access" )

3. DNS Configuration​

A wildcard DNS record pointing to your cluster's ingress IP or hostname:

Record typeWhen to useExample
AIngress IP is known*.<base>.<domain>.<com> → <K8s_ingress_public_IP>
CNAMELB provides a hostname rather than a bare IP*.<base>.<domain>.<com> CNAME <lb-hostname>

A wildcard DNS record (*.base.domain.com) resolves all subdomains under the base domain to the same IP, enabling dynamic service deployment without individual DNS entries per service. This is essential for microservices platforms.

4. TLS Planning​

Load balancer TLS handling: Before choosing a certificate source, verify how your load balancer handles port 443 — passthrough (Istio manages TLS end-to-end) vs. TLS termination (LB decrypts and re-encrypts). This affects which option works correctly in your environment. See Troubleshooting for details.

CM Platform supports four certificate sources:

OptionDescriptionRequires
selfsigned (default)Auto-generated self-signed certificateNothing — works out of the box
letsencryptAutomatically issued by Let's Encrypt via ACMEValid public DNS + reachable HTTP endpoint on port 80
ownBring your own wildcard certificatePEM files placed in own-cert/ directory
customTlsSecretServe a TLS certificate that you create and renew yourselfAn existing kubernetes.io/tls Secret in the istio-ingress namespace

See TLS Settings for the values file snippets for each option.

For selfsigned, own, and customTlsSecret certificates, LB-level TLS re-encryption is compatible — the platform runs no ACME validation in these modes. In these cases a CNAME DNS record to the LB hostname also works — see DNS Configuration above. One caveat for customTlsSecret: if the issuer you configure uses an ACME http01 solver, it inherits the same port 80 requirement as letsencrypt.

5. Required Tools​

  • kubectl
  • helm

Installation Process​

1. Prepare Directory Structure​

export clustername=<your-clustername--or--projectname>
mkdir cm-platform-$clustername
cd cm-platform-$clustername

mkdir git-access-keys
mkdir image-registries-auth

2. Prepare Required Keys and Credentials​

  1. CM Git Repository Access:
  • Generate SSH Key 
    ssh-keygen -t ed25519 -f git-access-keys/cm-platform-gitops -N ""

    Leave the passphrase empty — passphrase-protected keys are not supported by the CM Platform installer.

  • Share the public key (cm-platform-gitops.pub) with CM team
  1. CM OCI Registry Access:
  • Obtain cm-images.json from CM team
  • Save it as image-registries-auth/cm-images.json

3. Custom TLS Certificates (Optional)​

If using your own TLS certificates:

  • Create own-cert directory 
    mkdir own-cert
  • Place your certificate files:
    • cm-platform-$clustername/own-cert/fullchain.pem (full certificate chain)
    • cm-platform-$clustername/own-cert/privkey.pem (private key)

The certificate must be a wildcard certificate (e.g. *.cm-platform-prod.example.com) valid for any subdomain under the base domain.

4. Prepare Configuration Files​

Basic setup profile​

Download and modify the examples values file. The example has a description for each parameter.

curl -o basic-values.yaml https://raw.githubusercontent.com/ConfidentialMind/examples/main/platform-deployment/custom-values-profiles/basic-values.yaml

(advanced) For more configuration parameters check the Optional Settings section in the bottom:

Other setup profiles​

OpenShift/OKD:

curl -o openshift-values.yaml https://raw.githubusercontent.com/ConfidentialMind/examples/main/platform-deployment/custom-values-profiles/openshift-values.yaml

5. Verify Directory Structure​

tree .

Expected minimal structure:

cm-platform-<your-clustername--or--projectname>/
├── basic-values.yaml              # Your basic custom settings
├── ***-values.yaml # (Optional)   # Your other custom settings
├── git-access-keys/
│   ├── cm-platform-gitops         # Git access private key
│   └── cm-platform-gitops.pub     # Git access public key
├── image-registries-auth/
│   └── cm-images.json             # OCI registry credentials
└── own-cert/ # (Optional)         # Custom certificates
    ├── fullchain.pem
    └── privkey.pem

Deployment​

1. Install CRDs​

  • Argo CD "Application"
kubectl create -f https://raw.githubusercontent.com/argoproj/argo-cd/refs/tags/v3.3.4/manifests/crds/application-crd.yaml
  • Prometheus "ServiceMonitor"

This CRD might exist in your cluster, therefore, kubectl create will fail, which is normal behaviour.

kubectl create -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.88.0/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml

2. Install CM Platform​

Standard Installation (with Let's Encrypt, self-signed, or your own TLS Secret):​

helm upgrade --install cm-platform oci://confidentialmindpub.azurecr.io/cm-platform \
  --namespace cm-platform --create-namespace \
  --values basic-values.yaml \
  --set-file gitRepositories.gitops.accessKey=git-access-keys/cm-platform-gitops \
  --set-file imageRegistries.cmApps.auth=image-registries-auth/cm-images.json

With certificateSource: customTlsSecret you run this same command — no certificate files are passed to Helm. Create the Secret in the istio-ingress namespace before you run it.

Installation with Your Own Certificate Files (certificateSource: own):​

helm upgrade --install cm-platform oci://confidentialmindpub.azurecr.io/cm-platform \
  --namespace cm-platform --create-namespace \
  --values basic-values.yaml \
  --set-file gitRepositories.gitops.accessKey=git-access-keys/cm-platform-gitops \
  --set-file imageRegistries.cmApps.auth=image-registries-auth/cm-images.json \
  --set-file tls.own.fullchainCertificate=own-cert/fullchain.pem \
  --set-file tls.own.privateKey=own-cert/privkey.pem

Post-Deployment Steps​

1. Verify Deployment​

  • Wait until all the deployed Argo CD applications become "Synced" and "Healthy":

    (it normally takes around 15–25 minutes)

watch kubectl get applications -n argocd
  • For more advanced monitoring or debugging use ArgoCD UI:
# Retrieve ArgoCD admin password
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

# Port-forward to ArgoCD UI
kubectl port-forward svc/argocd-server -n argocd 8080:80

# Access ArgoCD at: https://localhost:8080
# Username: admin
# Password: (the retrieved admin password)

2. Access CM Platform​

Once deployment is complete:

  1. Open https://portal.<base>.<domain>.<com>
  2. Contact CM team for initial credentials
  3. Proceed to deploy your first endpoint through the platform interface

Troubleshooting​

Common Issues​

  1. ArgoCD Apps Out of Sync or Unhealthy: Check Argo CD UI for the actual cause
  2. Let's Encrypt Certificate issues: Check DNS resolution
  3. DNS resolution: Verify wildcard DNS is correctly configured
    • A workaround for a dynamic LB IP assignment case: If the ingress IP is not known before deployment, deploy with certificateSource: selfsigned first, retrieve the IP after ArgoCD syncs (kubectl get svc istio-ingressgateway -n istio-ingress), configure DNS, then run helm upgrade switching to letsencrypt.
  4. TLS / certificate issues at the Load Balancer: Depending on the infrastructure, a load balancer may be configured to terminate TLS itself (re-encrypting traffic to the backend) rather than passing it through to Istio. This can result in the LB's own certificate being presented to clients instead of the one managed by cert-manager, or in unexpected certificate validation errors. This is not necessarily wrong — LB-level TLS termination can be intentional — but it must be taken into account when planning the TLS topology and when troubleshooting certificate issues. Verify how your LB handles port 443 (passthrough vs. termination) and align it with the intended certificate management approach.
    • Quick fix — TCP passthrough: Configure the LB to forward port 443 as plain TCP to Istio without terminating TLS. Simple and effective when Istio/cert-manager manages certificates end-to-end, but may not be the right choice if LB-level TLS termination is required by your infrastructure or security policy.
  5. Longhorn CSI issues: Ensure Longhorn prerequisites are installed
    • For K3s clusters with Longhorn CSI, install prerequisites on host OS:
      • Debian/Ubuntu: apt install open-iscsi nfs-common cryptsetup systemctl enable --now iscsid
      • RedHat/CentOS: dnf install iscsi-initiator-utils nfs-utils cryptsetup systemctl enable --now iscsid
  6. Sync stops early after a database volume was grown: CNPG refuses to shrink a volume, so if your values file still declares the original size, the cm-platform-base sync fails on the database and everything sequenced after it never applies. Declare the size the volume actually has — see Database storage size.
  7. An application reports a CORS origin error: every origin in corsCustomAllowedOrigins must include a scheme, so portal.example.com is rejected where https://portal.example.com is accepted. helm upgrade still succeeds — the affected application (for example manager-go) then fails to render in ArgoCD and stops applying, and the error names the service and the offending value. Add the scheme and re-run the upgrade. See Extra CORS Origins.

Optional Settings​

TLS Settings​

Let's Encrypt Certificate:

tls:
  certificateSource: letsencrypt
  letsEncrypt:
    # Email for Let's Encrypt certificate generation
    email: <admin>@<your-domain>.<com>

Custom Certificate:

tls:
  # Source of the TLS certificate
  certificateSource: own

Make sure to create cm-platform-$clustername/own-cert directory and place PEM files there The files should be loaded via helm ... --set-file ...

Externally Managed TLS Secret (advanced):

tls:
  certificateSource: customTlsSecret
  customTlsSecret:
    # Existing kubernetes.io/tls Secret in the istio-ingress namespace
    name: cm-platform-tls-<my-custom>

Use this when none of the options above fit the certificate flow — a cert-manager Issuer/ClusterIssuer with a DNS-01 or non-ACME solver, a corporate CA, external-secrets syncing from a vault, or a Secret created by hand. You keep full control of issuance and renewal; the platform only points the Istio ingress Gateway at the Secret you name and creates nothing itself.

cert-manager is already installed as a platform dependency, so this is also the supported way to use it directly: create your own Issuer or ClusterIssuer with whatever solver you need, plus a Certificate whose secretName is the Secret you name here. The platform will not touch those CRs — only the built-in letsencrypt and selfsigned modes create cert-manager resources of their own.

own vs customTlsSecret: the two overlap — both serve a certificate you obtained yourself — but they differ in how that certificate reaches the cluster. own takes the private key and full chain as plaintext PEM files passed into every helm upgrade/install of cm-platform (via --set-file), which means the key material lives in your installer invocation and values. customTlsSecret removes that requirement entirely: the Secret is created and rotated out-of-band, and the platform only ever learns its name.

Requirements:

  • The Secret must be of type kubernetes.io/tls and must exist in the istio-ingress namespace — Istio resolves credentialName in the gateway workload's namespace.
  • It must cover every enabled subdomain (api, auth, tools, portal, plus chat/portalv2/docs when those feature flags are on), or a wildcard for the base domain.
  • Create the Secret before switching certificateSource. Until it exists, Istio has no certificate to serve and every host is unreachable over HTTPS.
  • Name it cm-platform-tls-<any custom name>. Sticking to that convention keeps the platform's TLS Secret easy to spot among everything else in istio-ingress, and it keeps you off the exact name cm-platform-tls — when migrating from letsencrypt, own, or selfsigned, ArgoCD still tracks that one from the previous mode and will prune it on the next sync.
  • Renewal is yours to handle. Istio picks up in-place Secret updates without a Gateway change or restart.

If the certificate is signed by a private CA that browsers don't trust, note that HSTS is only stripped for certificateSource: selfsigned — clients will need the CA distributed to their trust stores.

CNPG Backup settings​

db:
  cnpg:
    backup:
      enabled: false # Set to 'true' to enable pgBackRest backup for CNPG.
      pgBackRest:
        s3:
          bucket: <bucket-name>
          endpointURL: <hostname>
          disableVerifyTLS: false # default
          region: dummy # default
          accessKeyId: <access-key-id>
          secretAccessKey: <access-key-secret>
        encryption:
          key: <encryption-key>
        # Backup schedule (6-digit cron format [with seconds]). Example: '0 0 */4 * * *' = every 4 hours
        schedules:
            # Full backup. E.g. '0 0 2 * * 0' = Weekly full backup, Sundays at 2 AM
          - type: full
            time: "0 0 2 * * 0" # default
            # Differential backup. E.g. '0 0 2 * * 1-6' = Daily differential backup, Mon-Sat at 2 AM
          - type: diff
            time: "0 0 2 * * 1-6" # default
            # Incremental backup. E.g. '0 0 */1 * * *' = Incremental backup every 1 hour
          - type: incr
            time: "0 0 */1 * * *" # default
        # Retention policy pgBackRest for backups
        retention:
          # Number of full backups to retain
          full: 2 # default
          # Number of differential backups to retain. Note that full backups are included in the count of differential backups
          diff: 3 # default

Database storage size​

The platform databases — postgresql and keycloak-postgresql — each get a 10 GiB volume by default. Set a larger size per database here:

db:
  cnpg:
    clusterSizes:
      # Quoted string with a unit — a plain number is rejected.
      keycloak-postgresql: "20Gi"

CNPG never shrinks a volume. If one was already expanded in the cluster, declare that size here — otherwise your values file keeps asserting the smaller original and the sync fails when CNPG refuses the decrease. To read the current sizes:

kubectl -n databases get clusters.postgresql.cnpg.io \
  -o custom-columns='DATABASE:.metadata.name,SIZE:.spec.storage.size'

Only the two names above are accepted. A wrong name or an unquoted size is not caught by helm upgrade, which still succeeds — the cm-platform-base application in ArgoCD then fails to render, reports the valid names, and stops applying anything sequenced after it. So after changing this setting, verify the deployment as described under Post-Deployment Steps rather than trusting the Helm exit code.

Requires installer 0.2.15 or newer — older installers ignore the setting without warning.

Secrets Provider configuration for External Secrets Operator​

secrets:
  
  # Use External Secrets Operator (ESO) ClusterSecretStore to store the secrets or push them directly from ESO Generators to Kubernetes Secrets.
  # IMPORTANT: When set to `false`, the Platform falls back to using `Kubernetes` ClusterSecretStore which stores passwords as Secrets in a dedicated namespace. Using `Kubernetes` ClusterSecretStore is not recommended for production use.
  # Read more about possible providers: https://external-secrets.io/latest/provider/kubernetes/
  useSecretsProvider: false
  
  # ClusterSecretStore provider configuration. This section is passed directly to the ClusterSecretStore resource. Tested with OpenBao.
  provider: {}
    
    # It's an example configuration for OpenBao or Hashicorp Vault provider
    # vault:
    #   address: http://openbao.openbao.svc.cluster.local:8200
    #   token: your-openbao-token
    
    # It's a dummy example to illustrate how to add other providers
    # exampleProvider:
    #   ...provider specific properties...

  # Extra credentials for the secrets provider, if needed. This section includes sensitive data like tokens or passwords that can't be passed directly in the 'provider' section. E.g., for OpenBao/Vault token secret, which is then referenced in 'provider.vault.auth.tokenSecretRef' later on, it will create the required secrets in all the relevant namespaces. For non-tested providers, check External Secrets Operator documentation and CM Documentation, and create the secrets manually, if required.
  providerExtraCredentials:
    vault:
      token: ""

Extra CORS Origins​

The browser sends the hostname it loaded the page from as Origin, and each service accepts only the origins it knows about — the portal subdomain, plus auth for sign-in and chat when the chat frontend is enabled. If users reach the platform through a proxy on some other hostname, add that hostname: a proxy that rewrites Host normally passes Origin through untouched, so the platform sees a name only you know about.

corsCustomAllowedOrigins:
  # Full origins, including the scheme. Each service has its own route, so
  # list every one the browser calls directly.
  manager:
    - https://portal.example.com
  keycloak:
    - https://portal.example.com
  managerGo:
    - https://portal.example.com

manager and managerGo are separate services on separate routes — managerGo serves the /cmind.*.v1 gRPC APIs, including API key management. Set whichever ones the browser actually calls.

Model Cache Path​

The node-local directory where model weights are cached on the serving nodes. It defaults to /mnt/llm_cache. Override it if that path is unsuitable on your nodes (for example, to point at a larger or faster local disk):

dependencies:
  ome:
    # Node-local base directory for cached model weights. Must exist (or be
    # creatable) and be writable on every node that runs model-serving pods —
    # it is mounted as a hostPath, not a PersistentVolume.
    modelCacheBasePath: /mnt/llm_cache # default

JWT Signing Key​

The platform signs its internal JWTs with a key it generates on install and keeps in the secret store.

The default is ed25519. Set the algorithm before the first install if you have a policy that requires a particular one — the key is generated once, so changing these values later has no effect on an installation that already has one:

passport:
  signingKey:
    # ed25519 (default), ecdsa, or rsa.
    type: ed25519
    # Key size. rsa: 2048, 3072 or 4096. ecdsa: 256, 384 or 521.
    # ed25519 has a single size and ignores this, so leave it empty.
    size: ""

To use a key you manage yourself instead of a generated one, supply it as a PEM-encoded PKCS#8 private key. A key given here takes precedence and nothing is generated:

istio:
  keys:
    privateKeyPem: ""

Rotating the key​

The platform never replaces the key on its own, and re-running the installer keeps the one you already have. Rotating it is a manual operation, and it interrupts API traffic — plan for a maintenance window.

  1. Delete the stored key from your secrets provider. It is held under cm-platform-passport. A replacement is generated on the next sync and reaches the cluster within a few minutes.

  2. Restart the services that signed with the old key — they read it at startup and will not pick up a new one otherwise:

    kubectl -n api-services rollout restart deployment/authn deployment/manager-go

  3. Requests are rejected until Istio picks up the new key set, which it caches for up to 20 minutes. Restart the control plane to cut that short:

    kubectl -n istio-system rollout restart deployment/istiod

In-platform documentation site​

The platform serves its own documentation site, and the portal's View documentation link points at it. To publish it, give it a subdomain:

dns:
  subDomains:
    docs: docs   # serves the docs site at https://docs.<base>.<domain>

With the standard wildcard DNS record and a selfsigned or own (wildcard) certificate, that is the only change needed — the docs host resolves through the wildcard and is already covered by the certificate. For letsencrypt, the docs host is added to the platform certificate's SAN list and validated over HTTP-01 like every other subdomain; the wildcard DNS record already makes it resolvable, so no extra record is required.

If dns.subDomains.docs is left unset, the platform does not route the docs site and the portal's documentation link falls back to the public site at https://docs.confidentialmind.com. Nothing else is affected — in particular the platform certificate does not gain an unresolvable SAN.