Certified Kubernetes Administrator Exam Series (Part-5): Cluster Maintenance

Introduction to Cluster Maintenance 

Kubernetes clusters require management practices to ensure that nodes, PODs, and other resources operate optimally. Cluster maintenance practices fall under three broad categories: 

• Operating System Upgrades, 
• Cluster Upgrades, and 
• Backup & Restore Methodologies.  

This section takes the candidate through various cluster maintenance processes, including upgrading the Operating System, the implications of evicting a node from the cluster, Kubernetes Releases & Versions, the cluster upgrade process, and upgrade best practices among others. Through live practicals, the candidate will perform an end-to-end cluster upgrade, which helps to get familiar with cluster administration tasks required for both the CKA Exam as well as professional practice. The course also takes on a real-world disaster recovery scenario where the candidate performs backup and restore practices to bring a failed cluster back to a working state. 

OS Upgrades

When a node in a cluster goes down, the PODs running inside it automatically become inaccessible. Users may not be able to access cluster services hosted in the PODs. PODs with instances running in other nodes, however, will have their workloads unaffected. If the failed node comes back online immediately, the Kubelet service starts, and the PODs become available. 

However, if the node stays unavailable for 5 minutes the PODs are permanently terminated. PODs that were part of a ReplicaSet are recreated inside other nodes. The time it takes to wait before PODs are terminated is known as the POD Eviction Timeout, and is set on the Kube-Controller-Manager with a default value of 5 minutes:

$ kube-controller-manager --pod-eviction-timeout= 5m0s

If a node comes back online after the POD Eviction Timeout, it is blank with no PODs scheduled on it. This means that only quick updates can be performed on the nodes and they should be rebooted before the timeout period or their PODs to be available. 

For node updates that are expected to take a longer time, like an Operating System Upgrade, the drain command is used. This redistributes the PODs to other nodes in the cluster:

$ kubectl drain node-1

When a node is drained, the PODs are gracefully terminated in the node then recreated in other cluster nodes. The node being drained is marked as unschedulable so no other PODs can be assigned to it. Even when it comes back online, the node is still unschedulable until the developers lift the tag off using the uncordon command:

$ kubectl uncordon node-1

The node does not recover any PODs previously scheduled on it. Rather, newer PODs in the cluster can be scheduled in the cluster once it has been uncordoned.

The cordon command marks a node as unschedulable. Unlike the drain command, however, this one does not terminate existing PODs on a node. It just ensures that no newer PODs are scheduled on the node:

$ kubectl cordon node-1

Kubernetes Software Versions

Clusters run on specific versions of Kubernetes. The Kubernetes cluster can be seen in the following output by using the kubectl get nodes command:

NAME           STATUS   ROLES                  AGE     VERSION
controlplane   Ready    control-plane,master   9m52s   v1.17.3
node01         Ready    <none>                 9m11s   v1.17.3

Kubernetes versions follow an x.y.z pattern. The Kubernetes VERSION here is v1.17.3. The anatomy of the version shows how the Cloud Native Foundation (CNCF) manages Kubernetes releases through a standard procedure. v1 is the major release version. The first major version was (v1.0) released in July 2015. .17 represents the minor version. Minor versions typically come with new features and improvements. The latest stable Kubernetes version is (v1.21). The .3 represents a patch version that comes with the latest bug fixes.

CNCF also releases Alpha and Beta versions of the software, to initially test the effectiveness of new features and improvements. Alpha versions typically have new features disabled by default and come with a lot of bugs. Beta releases feature well-tested code plus new features and improvements. These features finally make their way to the stable release. All stable releases can be found on the official releases page in the Kubernetes Github repository. All Kubernetes components can be downloaded in the tar.gz file which when extracted contains most of the control plane elements in the same version. The ETCD Cluster and CoreDNS Server are of different versions since they are managed by separate projects. 

Introduction to Cluster Upgrades

Core Control Plane Components do not have to run on the same Kubernetes release version. The kube-apiserver is the primary control plane component and is always in communication with other components. No component should, therefore, be running on a version newer than kube-apiserver. The kube-scheduler and kube-controller can run on a version that is at most one version older than the kube-apiserver. This means that, if kube-apiserver is running on version v1.17, then kube-scheduler and kube-controller can run on versions v1.17 and v1.16. kubelet and kube-proxy can run on versions that are up to 2 levels older than the kube-apiserver. kubectl can run on a version older or newer than the kube-apiserver. This makes it easy to upgrade a cluster component-by-component as required. 

Kubernetes only supports the 3 latest versions of software. Components are best upgraded during a minor release. It is also important to upgrade the components one minor version at a time, rather than skipping straight to the latest release. The upgrade procedure generally depends on where the cluster is set up. If the application is running on a managed cloud service like Google Cloud, the upgrade can be performed by a few simple clicks of a button. If the cluster was created from scratch, then every component has to be upgraded manually. If the cluster was set up using kubeadm, then cluster upgrades are performed by simply running the upgrade plan and upgrade apply commands. In a kubeadm cluster, the upgrade process  follows two major steps:

• Upgrading the master node
• Upgrading the worker nodes

When upgrading the master node, all control plane components become briefly unavailable. Cluster management functions are inactive but the worker nodes continue running workloads. Cluster components and resources cannot be accessed by Kubernetes API tools such as kubectl. 

There are different strategies with which the worker nodes can be upgraded:

• In the first strategy, all worker nodes are taken down simultaneously, upgraded then brought back up. During this type of upgrade, the PODs are down and users cannot access the application. After the upgrade, the nodes are brought back up so new PODs can be scheduled and the application starts running.
• The second strategy involves upgrading the worker nodes one at a time. In this case, when one node is being upgraded, its workload is shifted to the remaining nodes in the cluster. When it comes back up, workloads from the next node are moved into the upgraded node, allowing the next node to get an upgrade. This process is repeated until all nodes in the cluster have been upgraded.
• The third strategy involves introducing newer worker nodes running a new version of software into the cluster. As a new node joins the cluster, it is assigned workloads from an existing node. The old node then exits the cluster. This process is repeated until all cluster nodes have been replaced by new ones. This approach is particularly effective when the cluster is hosted on a managed cloud platform which makes it easy to provision new instances and decommission existing ones.

In a kubeadm cluster, the Master node upgrade can be planned by first upgrading kubeadm using the command:

$ apt-get upgrade -y kubeadm=1.17.0-00

The latest available upgrades are then checked by running the command:

$ kubeadm upgrade plan

This command outputs the current version information, and the latest release available for all cluster components. The upgrade can then be performed using the command:

$ kubeadm upgrade apply

Running the kubectl get nodes command will display the master node still running the older version of Kubernetes. This is because the command checks for the version of kubelet running on the host. The service is upgraded using the command: 

$ apt-get upgrade -y kubelet=1.17.0-00

To upgrade a worker node, its workloads are first safely redistributed using the drain command:

$ kubectl drain node-1 --ignore-daemonsets

This command reassigns PODs in the current node to other worker nodes within the cluster. It then cordons the node, meaning no additional PODs can be scheduled on it during the upgrade.

kubeadm is then upgraded using the command:

$ apt-get upgrade -y kubeadm=1.17.0-00

The kubelet service is then updated using the command: 

$ apt-get upgrade -y kubelet=1.17.0-00

The node can then be upgraded to the latest version using the command:

$ kubeadm upgrade node-1

The node now runs the newest version of Kubernetes software. For PODs to be scheduled on it, the node should be uncordoned using the command:

$ kubectl uncordon node-1

The node is now back on and can accept workloads from other nodes to be upgraded, and new PODs can be scheduled on it. This process is repeated until all worker nodes in the cluster have their software versions upgraded.

Backup and Restore Methods

Backup and Restore Using Resource Configuration

Kubernetes cluster resources are created by defining their configurations using the imperative or declarative approach. The declarative approach is preferred for developers who want to save and share their application’s configuration information. This is because, with the declarative approach, resource configurations are defined in manifest files that can be stored and shared between team members. These files can even be reused in other projects.

It is important to store copies of resource manifest files. A good practice is to have them in a repository where they can be accessed and managed by the development team. The repository should be configured with proper backup options. A great choice is a public, managed repository like GitHub, where developers don’t have to worry about providing backup and restoration options. In case a cluster is lost, it can be easily recovered by applying the configuration files stored in the repository. 

Not everyone in a team has to follow the declarative approach in creating resources. If resources were created imperatively, the cluster can be backed up by first querying the kube-apiserver on information regarding cluster resources. This configuration information can then be saved so that it can be referred to later when the resources need restoration. To query the Kubernetes API on resource configuration, the following command is used:

$ kubectl get all --all-namespaces -o yaml > darwin-service-config.yaml

This command gets configuration information on all resources in all namespaces within the cluster in YAML format then saves them in a folder named darwin-service-config.yaml. In case the cluster needs to be restored, the resources can easily be rebuilt using the configuration information in the file.

Some tools, such as VELERO, can also help create backups by querying the API and storing configuration information periodically.  

Backup and Restore Using ETCD

The ETCD Server stores data about cluster resources. Backing up the ETCD cluster can be an effective alternative to storing resource configuration information. The data directory where this information is stored is specified when configuring the ETCD service. Backup tools can be configured to access this location periodically for backup. 

The etcdctl tool includes a built-in snapshot utility that lets developers create a backup snapshot of the ETCD cluster instantaneously.  A snapshot is created using the command:

$ ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
     --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \
     snapshot save /tmp/snapshot.db

This saves the current ETCD cluster information in a file named snapshot.db. To save this file in another location, the file path is appended to the above command.

The status of the backup can be viewed using the command:

$ ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt      --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key      snapshot status /tmp/snapshot.db

The snapshot is restored by pointing etcdctl to the snapshot file:

ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
     --name=master \
     --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \
     --data-dir /var/lib/etcd-from-backup \
     --initial-cluster=master=https://127.0.0.1:2380 \
     --initial-cluster-token etcd-cluster-1 \
     --initial-advertise-peer-urls=https://127.0.0.1:2380 \
     snapshot restore /tmp/snapshot.db

Modify /etc/kubernetes/manifests/etcd.yaml: 
Update –data-dir to use new target location

--data-dir=/var/lib/etcd-from-backup

Update new initial-cluster-token to specify new cluster

--initial-cluster-token=etcd-cluster-1

Update volumes and volume mounts to point to new path

   volumeMounts:
    - mountPath: /var/lib/etcd-from-backup
      name: etcd-data
    - mountPath: /etc/kubernetes/pki/etcd
      name: etcd-certs
  hostNetwork: true
  priorityClassName: system-cluster-critical
  volumes:
  - hostPath:
      path: /var/lib/etcd-from-backup
      type: DirectoryOrCreate
    name: etcd-data
  - hostPath:
      path: /etc/kubernetes/pki/etcd
      type: DirectoryOrCreate
    name: etcd-certs

When ETCD restores from this backup, it initializes a new cluster configuration and configures its components to run as new members of the new cluster. This prevents newly created members from joining existing clusters. After the manifest files are modified ETCD cluster should automatically restart.

The cluster now runs using the configurations stored on the snapshot of the ETCD server.

Research Questions & Conclusion

This concludes the Cluster Maintenance section of the CKA certification exam. To test your knowledge, it is strongly recommended that you access research questions of all core concepts covered in the coursework and a test to prepare you for the exam. You can also send your feedback to the course developers, whether you have feedback or would like something changed within the course. 

Here is a quick quiz with a few questions and tasks to help you assess your knowledge. Leave your answers in the comments below and tag us back. 

Quick Tip – Questions below may include a mix of DOMC and MCQ types.

1. We need to take node01 out for maintenance. Empty the node of all applications and mark it unschedulable. The command to do so will be? Well, here is the answer –

$ kubectl drain node01 --ignore-daemonsets

2. You are tasked to upgrade the cluster. User’s accessing the applications must not be impacted. And you cannot provision new VMs. What strategy would you use to upgrade the cluster?

[A] Upgrade all nodes at once

[B] Users will be impacted since there is only one worker node

[C] Upgrade one node at a time while moving the workloads to the other

3. Upgrade the controlplane components to version v1.20.0

On the controlplane node, run the command run the following commands:
$ apt update
This will update the package lists from the software repository.


$apt install kubeadm=1.20.0-00
This will install the kubeadm version 1.20


$ kubeadm upgrade apply v1.20.0
This will upgrade kubernetes controlplane. Note that this can take a few minutes.


$ apt install kubelet=1.20.0-00 
This will update the kubelet with the version 1.20.


You may need to restart the kubelet after it has been upgraded.
Run: $ systemctl restart kubelet

4. Where is the ETCD server certificate file located?

[A] /etc/kubernetes/pki/server.crt

[B] /etc/kubernetes/pki/etcd/peer.crt

[C] /etc/kubernetes/pki/etcd/ca.crt

[D] /etc/kubernetes/pki/etcd/server.crt

5. Webapp is a critical app and we do not want it to be removed and we do not want to schedule any more pods on node01. Mark node01 as unschedulable so that no new pods are scheduled on this node.

$ kubectl cordon node01

6. Where is the ETCD CA Certificate file located?

[A] /etc/kubernetes/pki/etcd/ca.crt

[B] /etc/kubernetes/pki/ca.crt

[C] /etc/kubernetes/pki/etcd/peer.crt

[D] /etc/kubernetes/pki/etcd/ca.key

7. The master nodes in our cluster are planned for a regular maintenance reboot tonight. While we do not anticipate anything to go wrong, we are required to take the necessary backups. Take a snapshot of the ETCD database using the built-in snapshot functionality. Store the backup file at location /opt/snapshot-pre-boot.db.

Try on your own of doing this first, and then validate with the right approach as mentioned below.

ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key \ snapshot save /opt/snapshot-pre-boot.db

Conclusion

As Kubernetes clusters are highly distributed, cluster maintenance processes and procedures form a crucial task for any Kubernetes administrator. This section has explored various processes encompassing the maintenance of nodes and cluster resources. The class also introduces OS upgrades and Kubernetes software versions to help candidates make sure applications run in production as expected.

Candidates are also taken through cluster upgrade processes, and how to keep workloads running when updating nodes. Finally, the chapter explores various backup and restore methods available for production-grade clusters.

More details about KodeKloud’s CKA course with access to the lessons, labs, mock exams and demo can be found here – https://kodekloud.com/courses/certified-kubernetes-administrator-cka/.