Skip to main content
Version: 3.5.0-rc.2

Graceful node shutdown

Drain a Kubernetes node before powering off, rebooting, or resetting the host machine that runs it.

Best practices​

Containers are not virtual machines, but a container host is treated like a hypervisor: the hosted instances are evacuated and stopped properly before the host machine is powered off. In Kubernetes the hosted instances are pods, and evacuating them is called draining.

Draining a node ahead of shutdown protects three things on a CM Platform cluster:

  • Data integrity. Databases and replicated volumes finish their in-flight writes and stop cleanly, so the node rejoins without recovery or replica rebuilds.
  • Recovery time. Pods reschedule immediately rather than after the control plane's node-monitor and volume-detach timeouts, and model pods reload their weights once, at a time you choose.
  • Volume handover. Attached volumes are released cleanly and reattach on another node without manual intervention.

Clusters also run stateful components of their own — database clusters, and storage providers such as Longhorn CSI — each with its own replicas, health state, and eviction rules. Confirm these report healthy before a drain, and configure them to evict or stop gracefully during node maintenance.

Draining and shutting down a node​

Run from a workstation with cluster admin access, one node at a time:

# 1. Evict the workloads. Also cordons the node, so nothing new is scheduled onto it.
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

# 2. Confirm the node is empty. Only DaemonSet pods remain.
kubectl get pods -A --field-selector spec.nodeName=<node>

# 3. Power off or reboot the host.
ssh <node> sudo shutdown -h now

Platform service pods finish the requests they are handling before they stop, so step 1 can wait on them: up to 45 seconds for most platform pods, and about two and a half minutes for the Model Gateway, whose streamed answers can run for minutes. Let the drain wait. Forcing the eviction ends those requests with an error.

Return the node to service once it reports Ready:

kubectl get node <node>          # wait for Ready
kubectl uncordon <node>

:::warning Control-plane nodes and quorum etcd members keep quorum throughout. Drain and reboot one control-plane node at a time, and confirm it is Ready with a healthy etcd member before moving to the next. Quorum loss leaves the cluster read-only until it is restored. :::

A stalled drain is usually a PodDisruptionBudget holding a workload at its configured minimum — often a storage provider's, keeping the node in place until another healthy replica of each volume exists — or a pod with no controller to recreate it elsewhere. Resolve the cause; the budget is what keeps the service available through the drain.

Platform services that run more than one pod have a budget too, which lets one pod of each service be down at a time. The drain waits until the replacement pod is running on another node; kubectl get pdb -n api-services shows which service it is waiting for.

On a single-node cluster no replacement can start elsewhere, so these budgets hold the drain indefinitely. Add --disable-eviction to step 1 there. It deletes the pods instead of evicting them, which skips every budget on the node, and each pod still gets its full time to finish its requests.

References

Automation: further reading​

Manual drains cover planned maintenance. Graceful Node Shutdown covers the rest — shutdown run on the host, a UPS low-battery signal, a hypervisor ACPI power-off. Concepts to explore:

  • Graceful Node Shutdown — the kubelet holds a systemd delay-type inhibitor lock, and systemd waits while pods terminate in priority order. Activated by setting shutdownGracePeriod and shutdownGracePeriodCriticalPods, both of which default to zero; the feature gate itself is enabled by default. See Node Shutdowns and Graceful Node Shutdown Goes Beta.
  • systemd InhibitDelayMaxSec (/etc/systemd/logind.conf) — the maximum time systemd holds a shutdown open. Node-level grace periods fit within it. Kubelet fields are in Kubelet Configuration (v1beta1).
  • Pod-level grace periods — terminationGracePeriodSeconds and preStop hooks determine what each pod does with the time the node-level budget gives it. See Termination of Pods.
  • Cluster-wide reboot coordination — for rolling kernel and OS patching. Kubernetes ships no built-in controller and publishes no official how-to, so look for the pattern rather than a product: a reboot-coordinator DaemonSet that watches a reboot-required flag, takes a cluster-wide lock, then cordons, drains, reboots, and uncordons one node at a time, respecting PodDisruptionBudgets. The CNCF landscape lists implementations under Automation & Configuration.

After a non-graceful shutdown​

Following a power cut, hard reset, or shutdown without a drain:

  1. Check the filesystems. Review the boot log for I/O errors and replayed journals (journalctl -b -p err), and run fsck on any filesystem that did not mount cleanly.
  2. Confirm the machine is powered off, not mid-restart. For a node that stays down with pods in Terminating, apply the node.kubernetes.io/out-of-service=nodeshutdown:NoExecute taint — the control plane then force-deletes those pods and detaches their volumes for rescheduling. Remove the taint once the node is back.
  3. Let storage settle. Databases replay WAL and replicated volumes rebuild their replicas; allow both to finish before adding load.
  4. Reboot cleanly. Once the node reports Ready, run a cordon-drain-reboot cycle, then kubectl uncordon <node>.
  5. Add the drain to the procedure. Where the shutdown came from a person or a script, fold the drain into it, and see the automation reading above for the unattended cases.

References