Skip to main content

K3s Cluster Creation Guide

Introduction​

K3s is a lightweight, production-ready Kubernetes distribution that's ideal for running the ConfidentialMind Platform. This guide uses k3s-ansible to create a production-ready cluster in a repeatable way.

Prerequisites​

  • LoadBalancer routing all Kube API traffic to control plane nodes
    • port 6443 (and usually 80 and 443)
    • pass through for 6443; no HTTPS re-encryption
    • :6443 should be accessible from all the nodes
  • All nodes must have network connectivity to each other
  • SSH access to all cluster nodes from Ansible Control Node -- the machine where Ansible will be executed
  • Required Open Ports (K3s official requirements) (normally, K3s script manages to open all the required ports on OS level; pls make sure that your network configurations won't block the traffic)
    • TCP
      • 22: SSH access
      • 80: HTTP Ingress (for ACME verification mostly)
      • 443: HTTPS Ingress
      • 2379-2380: Required only for HA with embedded etcd
      • 6443: K3s supervisor and Kubernetes API Server
      • 10250: Kubelet metrics
    • UDP
      • 8472: Required only for Flannel VXLAN
      • 51820: Required only for Flannel Wireguard with IPv4
      • 51821: Required only for Flannel Wireguard with IPv6

Installation Steps​

1. Prepare Ansible Control Node​

Install required packages:

  • python3
  • python3-pip
  • git

Install Ansible

pip3 install ansible

Clone k3s-ansible repository

git clone https://github.com/k3s-io/k3s-ansible.git
cd k3s-ansible

2. Configure Inventory​

Create a new inventory file inventory-<new-cluster>.yml:

---
k3s_cluster:
  children:
    server:
      hosts:
        # it is possible to run on a single node, which works as a Control Plane, a Worker, and a GPU workload node at the same time
        # 192.168.0.1:
        # 192.168.0.2:
        # 192.168.0.3:
    agent:
      hosts:
        # GPU machines
        # 192.168.0.4:
        # 192.168.0.n:
  vars:
    ansible_port: 22
    ansible_user: <admin-username> # a user with SSH-key access and passwordless sudo rights on all the nodes
    k3s_version: v1.32.3+k3s1
    
    # Generate a random token with:
    # openssl rand -base64 64
    # or
    # pwgen -s 64 1
    token: "<your-generated-token>"
    
    # API endpoint will be the IP of the first server node
    # See "LoadBalancer" in "Prerequisites" section
    api_endpoint: "<LoadBalancer-IP>"
    
    # Server configuration
    extra_server_args: >-
      --disable traefik
      --etcd-snapshot-schedule-cron "0 */4 * * *"
      --etcd-snapshot-retention 20
      --disable local-storage
    
    # In case of a single node cluster:
    # - uncomment "cluster-init" to enable etcd DB mode 
    # - uncomment "default-runtime nvidia" to run Nvidia GPU workload
    # - uncomment "tls-san ..."
    #   if there is no LB with reverse proxy but you need to access Kube API through some router public IP;
    #   it will instruct k3s to include the external IP in Kube API certificate
    # 
    #   --cluster-init
    #   --default-runtime nvidia
    #   --tls-san "<the-router-external-IP>"

    # The below might be helpful in case of a complicated network setup on the nodes
    # when k3s can't or should not use the default route interface
    #   --flannel-iface=enX0
    #   --node-external-ip 1.2.3.4
    
    # Agent configuration (GPU nodes usually)
    extra_agent_args: >-
      --default-runtime nvidia

3. Generate Cluster Token​

Generate a secure token for your cluster:

# Option 1: Using openssl
openssl rand -base64 64

# Option 2: Using pwgen
pwgen -s 64 1

Add the generated token to your inventory file. For production environments, consider using ansible-vault to encrypt the token:

# Encrypt the token
ansible-vault encrypt_string 'your-generated-token' --name 'token'

4. Set Up SSH Access​

Ensure your SSH key is distributed to all nodes:

# Generate SSH key if needed
ssh-keygen -t ed25519 -C "k3s-ansible"

# Copy SSH key to each node
ssh-copy-id -i ~/.ssh/id_ed25519.pub <admin-username>@192.168.0.1

5. Make sure the LoadBalancer is configured and working​

See "LoadBalancer" in "Prerequisites" section

6. Deploy K3s​

Run the installation:

# Test connectivity
ansible -i inventory-<new-cluster>.yml all -m ping

# Deploy K3s cluster
ansible-playbook -i inventory-<new-cluster>.yml site.yml

7. Verify Installation​

# Copy kubeconfig from server node
ansible -i inventory-<new-cluster>.yml server[0] -m fetch -a "src=/etc/rancher/k3s/k3s.yaml dest=~/.kube/config flat=yes"

# Update server address if needed
sed -i "s/127.0.0.1/$(grep -A1 'server:' inventory-<new-cluster>.yml | tail -1 | awk '{print $2}')/g" ~/.kube/config

# Verify nodes
kubectl get nodes

# Check system pods
kubectl get pods -A

Next Steps​

After successfully setting up your K3s cluster continue with the Platform Deployment Guide