Skip to main content

Graceful node shutdown

Drain a Kubernetes node before powering off, rebooting, or resetting the host machine that runs it.

Best practices​

Containers are not virtual machines, but a container host is treated like a hypervisor: the hosted instances are evacuated and stopped properly before the host machine is powered off. In Kubernetes the hosted instances are pods, and evacuating them is called draining.

Draining a node ahead of shutdown protects three things on a CM Platform cluster:

  • Data integrity. Databases and replicated volumes finish their in-flight writes and stop cleanly, so the node rejoins without recovery or replica rebuilds.
  • Recovery time. Pods reschedule immediately rather than after the control plane's node-monitor and volume-detach timeouts, and model pods reload their weights once, at a time you choose.
  • Volume handover. Attached volumes are released cleanly and reattach on another node without manual intervention.

Clusters also run stateful components of their own — database clusters, and storage providers such as Longhorn CSI — each with its own replicas, health state, and eviction rules. Confirm these report healthy before a drain, and configure them to evict or stop gracefully during node maintenance.

Draining and shutting down a node​

Run from a workstation with cluster admin access, one node at a time:

# 1. Evict the workloads. Also cordons the node, so nothing new is scheduled onto it.
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

# 2. Confirm the node is empty. Only DaemonSet pods remain.
kubectl get pods -A --field-selector spec.nodeName=<node>

# 3. Power off or reboot the host.
ssh <node> sudo shutdown -h now

Return the node to service once it reports Ready:

kubectl get node <node>          # wait for Ready
kubectl uncordon <node>

:::warning Control-plane nodes and quorum etcd members keep quorum throughout. Drain and reboot one control-plane node at a time, and confirm it is Ready with a healthy etcd member before moving to the next. Quorum loss leaves the cluster read-only until it is restored. :::

A stalled drain is usually a PodDisruptionBudget holding a workload at its configured minimum — often a storage provider's, keeping the node in place until another healthy replica of each volume exists — or a pod with no controller to recreate it elsewhere. Resolve the cause; the budget is what keeps the service available through the drain.

References

Automation: further reading​

Manual drains cover planned maintenance. Graceful Node Shutdown covers the rest — shutdown run on the host, a UPS low-battery signal, a hypervisor ACPI power-off. Concepts to explore:

  • Graceful Node Shutdown — the kubelet holds a systemd delay-type inhibitor lock, and systemd waits while pods terminate in priority order. Activated by setting shutdownGracePeriod and shutdownGracePeriodCriticalPods, both of which default to zero; the feature gate itself is enabled by default. See Node Shutdowns and Graceful Node Shutdown Goes Beta.
  • systemd InhibitDelayMaxSec (/etc/systemd/logind.conf) — the maximum time systemd holds a shutdown open. Node-level grace periods fit within it. Kubelet fields are in Kubelet Configuration (v1beta1).
  • Pod-level grace periods — terminationGracePeriodSeconds and preStop hooks determine what each pod does with the time the node-level budget gives it. See Termination of Pods.
  • Cluster-wide reboot coordination — for rolling kernel and OS patching. Kubernetes ships no built-in controller and publishes no official how-to, so look for the pattern rather than a product: a reboot-coordinator DaemonSet that watches a reboot-required flag, takes a cluster-wide lock, then cordons, drains, reboots, and uncordons one node at a time, respecting PodDisruptionBudgets. The CNCF landscape lists implementations under Automation & Configuration.

After a non-graceful shutdown​

Following a power cut, hard reset, or shutdown without a drain:

  1. Check the filesystems. Review the boot log for I/O errors and replayed journals (journalctl -b -p err), and run fsck on any filesystem that did not mount cleanly.
  2. Confirm the machine is powered off, not mid-restart. For a node that stays down with pods in Terminating, apply the node.kubernetes.io/out-of-service=nodeshutdown:NoExecute taint — the control plane then force-deletes those pods and detaches their volumes for rescheduling. Remove the taint once the node is back.
  3. Let storage settle. Databases replay WAL and replicated volumes rebuild their replicas; allow both to finish before adding load.
  4. Reboot cleanly. Once the node reports Ready, run a cordon-drain-reboot cycle, then kubectl uncordon <node>.
  5. Add the drain to the procedure. Where the shutdown came from a person or a script, fold the drain into it, and see the automation reading above for the unattended cases.

References