Properly Resetting Your kubeadm-bootstrapped Cluster Nodes — #HeptioProTip
In this particular #HeptioProTip, we’ll be focusing on kubeadm, which is a bootstrapping tool for Kubernetes clusters. Though there’s an ecosystem of similar tools to choose from, the Kubernetes community (including Heptio) has put a lot of effort into improving kubeadm in the last few months, and it has gained a lot of traction and is our hope it will become the canonical way to stand up Kubernetes clusters.
Currently, Kubernetes relies heavily on hosts’ iptables to implement its networking policies. kubeadm sets up hundreds of these iptable rules on your behalf, which act as a de-facto “networking control plane”. To configure these rules, you can specify various bootstrap parameters during kubeadm init. (You can further customize inter-pod networking rules with CNI network providers like Calico and Weave, but we won’t be discussing that here).
This is great, but what happens if you are using kubeadm and you forget some important bootstrap parameters, or simply want to re-do the bootstrap process? Fortunately, there is a convenient command called kubeadm reset. This command deletes all Kubernetes components that kubeadm knows about and leaves the host in a just-like-new state.
That almost solves the problem, but there’s one more thing: for the sake of caution, kubeadm reset does not delete any of the iptables rules it originally created. In other words, if you try to bootstrap your cluster with a different pod networking CIDR range or different networking options, you might run into trouble.
Please note if you are using a firewall configuration tool like
ufw, which uses iptables as system-of-record, the commands below might render your system inaccessible.
Because of this, we recommend that you flush all iptables rules:
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
after you run kubeadm reset and before you re-bootstrap the node (kubeadm init) with new parameters.
This ensures that you really have a blank slate, and potentially saves you a lot of nasty network debugging.
We hope you enjoyed this #HeptioProTip! Follow @heptio for more tips and tricks.
