Skip to main content

K3s Cluster Creation Guide

Introduction​

K3s is a lightweight, production-ready Kubernetes distribution that's ideal for running the ConfidentialMind Platform. This guide uses k3s-ansible to create a production-ready cluster in a repeatable way.

Prerequisites​

  • LoadBalancer routing all Kube API traffic to control plane nodes
    • port 6443 (and usually 80 and 443)
    • pass through for 6443; no HTTPS re-encryption
    • :6443 should be accessible from all the nodes
  • All nodes must have network connectivity to each other
  • SSH access to all cluster nodes from Ansible Control Node -- the machine where Ansible will be executed
  • Required Open Ports (K3s official requirements) (normally, K3s script manages to open all the required ports on OS level; pls make sure that your network configurations won't block the traffic)
    • TCP
      • 22: SSH access
      • 80: HTTP Ingress (for ACME verification mostly)
      • 443: HTTPS Ingress
      • 2379-2380: Required only for HA with embedded etcd
      • 6443: K3s supervisor and Kubernetes API Server
      • 10250: Kubelet metrics
    • UDP
      • 8472: Required only for Flannel VXLAN
      • 51820: Required only for Flannel Wireguard with IPv4
      • 51821: Required only for Flannel Wireguard with IPv6

Installation Steps​

1. Prepare Ansible Control Node​

Install required packages:

  • python3
  • python3-pip
  • git

Install Ansible

pip3 install ansible

Clone k3s-ansible repository

git clone https://github.com/k3s-io/k3s-ansible.git
cd k3s-ansible

2. Configure Inventory​

Create a new inventory file inventory-<new-cluster>.yml:

---
k3s_cluster:
  children:
    server:
      hosts:
        # it is possible to run on a single node, which works as a Control Plane, a Worker, and a GPU workload node at the same time
        # 192.168.0.1:
        # 192.168.0.2:
        # 192.168.0.3:
    agent:
      hosts:
        # GPU machines
        # 192.168.0.4:
        # 192.168.0.n:
  vars:
    ansible_port: 22
    ansible_user: <admin-username> # a user with SSH-key access and passwordless sudo rights on all the nodes
    k3s_version: v1.32.3+k3s1
    
    # Generate a random token with:
    # openssl rand -base64 64
    # or
    # pwgen -s 64 1
    token: "<your-generated-token>"
    
    # API endpoint will be the IP of the first server node
    # See "LoadBalancer" in "Prerequisites" section
    api_endpoint: "<LoadBalancer-IP>"
    
    # Server configuration
    extra_server_args: >-
      --disable traefik
      --etcd-snapshot-schedule-cron "0 */4 * * *"
      --etcd-snapshot-retention 20
      --disable local-storage
    
    # In case of a single node cluster:
    # - uncomment "cluster-init" to enable etcd DB mode 
    # - uncomment "default-runtime nvidia" to run Nvidia GPU workload
    # - uncomment "tls-san ..."
    #   if there is no LB with reverse proxy but you need to access Kube API through some router public IP;
    #   it will instruct k3s to include the external IP in Kube API certificate
    # 
    #   --cluster-init
    #   --default-runtime nvidia
    #   --tls-san "<the-router-external-IP>"

    # The below might be helpful in case of a complicated network setup on the nodes
    # when k3s can't or should not use the default route interface
    #   --flannel-iface=enX0
    #   --node-external-ip 1.2.3.4
    
    # Agent configuration (GPU nodes usually)
    extra_agent_args: >-
      --default-runtime nvidia

3. Generate Cluster Token​

Generate a secure token for your cluster:

# Option 1: Using openssl
openssl rand -base64 64

# Option 2: Using pwgen
pwgen -s 64 1

Add the generated token to your inventory file. For production environments, consider using ansible-vault to encrypt the token:

# Encrypt the token
ansible-vault encrypt_string 'your-generated-token' --name 'token'

4. Set Up SSH Access​

Ensure your SSH key is distributed to all nodes:

# Generate SSH key if needed
ssh-keygen -t ed25519 -C "k3s-ansible"

# Copy SSH key to each node
ssh-copy-id -i ~/.ssh/id_ed25519.pub <admin-username>@192.168.0.1

5. Make sure the LoadBalancer is configured and working​

See "LoadBalancer" in "Prerequisites" section

6. Deploy K3s​

Run the installation:

# Test connectivity
ansible -i inventory-<new-cluster>.yml all -m ping

# Deploy K3s cluster
ansible-playbook -i inventory-<new-cluster>.yml site.yml

7. Verify Installation​

# Copy kubeconfig from server node
ansible -i inventory-<new-cluster>.yml server[0] -m fetch -a "src=/etc/rancher/k3s/k3s.yaml dest=~/.kube/config flat=yes"

# Update server address if needed
sed -i "s/127.0.0.1/$(grep -A1 'server:' inventory-<new-cluster>.yml | tail -1 | awk '{print $2}')/g" ~/.kube/config

# Verify nodes
kubectl get nodes

# Check system pods
kubectl get pods -A

Next Steps​

After successfully setting up your K3s cluster continue with the Platform Deployment Guide

Nodes are drained before their host machines are rebooted or powered off. See Graceful Node Shutdown for the procedure.