Clarification on Draining Nodes During Kubernetes Upgrade Using kubeadm

Certified Kubernetes Administrator
#1

Hi everyone,

I’m preparing for the CKA exam and have a few doubts regarding the process of upgrading a Kubernetes cluster using kubeadm. Specifically, I need clarification on the following points:

  1. Draining Nodes Prior to Upgrade:
Do we need to drain the nodes before performing the kubeadm upgrade itself, or should it be done prior to installing/upgrading kubectl and kubelet on each node, as suggested in the Kubernetes documentation? What’s the correct order?
  1. Taints and Draining Worker Nodes:
When draining a worker node, should we check and remove any taints on the control plane or worker nodes beforehand? This would ensure that the pods are safely rescheduled to other nodes. Is this a necessary step?
Draining Nodes in Minor vs Patch Upgrades:
  1. Draining Nodes in Minor vs Patch Upgrades:
Is draining nodes a step only required for minor version upgrades, or does it also apply when performing a patch upgrade (e.g., from v1.24.0 to v1.24.1)?

Your insights will be really helpful as I finalize my exam preparation for tommorrow. Thanks in advance!

#2

An upgrade should work whether or not the nodes are drained. The idea behind draining nodes and why it is good practice is to prevent application downtime in production clusters. What drain does is

  1. Make the node unschedulable so no new pods get placed on it (cordon)
  2. Evict existing pods (effectively kubectl delete), so that they return to scheduling to be recreated on another node.

If there are no nodes that can accept the pod (perhaps due to taints) then it remains in Pending state, which means it is still with the scheduler until a suitable node becomes available. When the node is uncordoned after upgrade, pending pods will be created on it.

In a production cluster, all your deployments should have at least 2 replicas, and there should be enough nodes that are valid for the pods being rescheduled. Whilst a pod is being rescheduled, other replicas of the deployment are still alive to handle traffic and prevent service interruption.

At the end of the day, and with all exam questions, the grader marks only the end state of the cluster, not the steps you took to achieve that state. So for a cluster upgrade it will check

  • That the nodes are at the requested version
  • That all the pods that should be running are still running.
1 Like
#3

Hi,

@Alistair_KodeKloud

Does it matter if we remove the taints from the control plane before draining a worker node and then revert them after the node upgrade, especially if the cluster has only two nodes, including the worker and control plane nodes?

#4

No. If anything you should not unless the question expects pods to be on the tainted node at the end of the upgrade. It will tell you if that is the case.

Note what I said about how the exam is graded. This means that if you drain a node and the pods can’t schedule, they will just remain in pending state. When you uncordon the node at the end, the pods will return to that node.
Pods are running - question is graded ok.

#5

Hi,

@Alistair_KodeKloud

But in that case will the pods (single pods) that are not part of any deployment,replicasets or statefulsets return back to the node from which the pod was evicted ? What could be done for those types of pods will they be lost or be in pending state when drained if the other nodes has taints . Will the question mention that finally after the upgrade the pods should be in this node or the other .

#6

I suppose the drain command will error out if it encounters pods that aren’t managed by a ReplicationController, ReplicaSet, Job, DaemonSet, or StatefulSet.

However, you can override it by using --force flag, which would delete the pods(they will be lost) that aren’t managed by any controllers.

#7

Hi,

@Srikanth_Reddy

But if in the question it is mentioning that the pods should not be lost even if we drain the node then can we remove the taints of the controlplane node (in case of 2-node cluster) and revert it back after the upgrade process if no other requirements are mentioned in the question?

#8

Hi,

@Srikanth_Reddy @Alistair_KodeKloud

I have one more question, and I apologize for it not being related to the current topic. It’s regarding the etcdctl snapshot and restore process. After taking an etcdctl snapshot (backup) using etcdctl snapshot save, and after restoring the snapshot using etcdctl snapshot restore, do we need to run any specific commands to verify that the tasks were completed successfully? If so, could you please share the procedure to verify that both the backup and restore processes were done correctly?"

#9

Assuming those pods are managed by some controllers(which will mostly be the case).
They won’t be lost when you drain the node and other the nodes have taints on them, rather they will be in pending state until the node you are draining(to update it) comes back up.

  • so can you remove the taints of the controlplane node? Yes, you can remove unless specified other ways in the question.
  • should you remove taints? Not necessarily, as the pods will be scheduled on the worker node once it comes back up.

again it all depends on what is asked in the question.

As @Alistair_KodeKloud mentioned:

At the end of the day, and with all exam questions, the grader marks only the end state of the cluster, not the steps you took to achieve that state. So for a cluster upgrade it will check

  • That the nodes are at the requested version
  • That all the pods that should be running are still running.
#10

There’s no way to verify a snapshot, other than to restore it. The act of taking a snapshot does not change cluster state. It simply creates the backup file specified in the snapshot command.

When restoring a snapshot the end state is

  • Cluster is not broken
  • Everything running in the cluster after restore is everything that was running at the point of the snapshot. Anything created after the snapshot will no longer be present. Anything deleted after the snapshot will be running again.

If you had a question that reads something like “Restore the snapshot found at /opt/snapshot.db”, then it should suffice to just ensure the cluster is still running. It may have different stuff running than before the restore - it may not.

You could do

kubectl get all -A > before.txt
etcdctl snapshot restore ...

update API server manifest as appropriate

kubectl get all -A > after.txt

then compare the content of the before and after files.