Question on Upgrade kubernetes cluster .. What happened to the core PODs that we running on control plane?

Kubernetes
#1

I am doing a lab on upgrade Kubernetes cluster …

Now when i execute the command
kubectl drain controlplane --ignore-daemonsetsNow this control plane node will run other PODS

controlplane ~ ➜ kubectl drain controlplane --ignore-daemonsets
node/controlplane cordoned
Warning: ignoring DaemonSet-managed Pods: kube-flannel/kube-flannel-ds-4zm27, kube-system/kube-proxy-vtfqt
evicting pod kube-system/coredns-768b85b76f-wq6jp
evicting pod default/blue-fffb6db8d-sc6tq
evicting pod default/blue-fffb6db8d-ch8dr
evicting pod kube-system/coredns-768b85b76f-tmtnc
pod/blue-fffb6db8d-ch8dr evicted
pod/blue-fffb6db8d-sc6tq evicted
pod/coredns-768b85b76f-tmtnc evicted
pod/coredns-768b85b76f-wq6jp evicted
node/controlplane drained

What happened to the above PODs? I see that since the application was configured as replicaset it moved to node 01 in the lab . But about about the Core POD’s that were running on Control plane nodes ?Did they terminated or they moved to node 01 ?

Why i am asking the above question is where i run the command :I still see the POD’s running

controlplane ~ ➜ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default blue-fffb6db8d-227sn 1/1 Running 0 16m
default blue-fffb6db8d-4jnbx 1/1 Running 0 16m
default blue-fffb6db8d-4nqbk 1/1 Running 0 5m39s
default blue-fffb6db8d-tknm8 1/1 Running 0 16m
default blue-fffb6db8d-wllfh 1/1 Running 0 5m39s
kube-flannel kube-flannel-ds-4zm27 1/1 Running 0 90m
kube-flannel kube-flannel-ds-tkb7t 1/1 Running 0 90m
kube-system coredns-768b85b76f-g4tfd 1/1 Running 0 5m39s
kube-system coredns-768b85b76f-t5msc 1/1 Running 0 5m39s
kube-system etcd-controlplane 1/1 Running 0 90m
kube-system kube-apiserver-controlplane 1/1 Running 0 90m
kube-system kube-controller-manager-controlplane 1/1 Running 0 90m
kube-system kube-proxy-8xnv8 1/1 Running 0 90m
kube-system kube-proxy-vtfqt 1/1 Running 0 90m
kube-system kube-scheduler-controlplane 1/1 Running 0 91m

Name: controlplane
Roles: control-plane
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=controlplane
kubernetes.io/os=linux
node-role.kubernetes.io/control-plane=
node.kubernetes.io/exclude-from-external-load-balancers=
Annotations: flannel.alpha.coreos.com/backend-data: {“VNI”:1,“VtepMAC”:“2a:ed:1e:e3:57:3a”}
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: true
flannel.alpha.coreos.com/public-ip: 192.28.114.9
kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Sat, 19 Oct 2024 20:21:45 +0000
Taints: node.kubernetes.io/unschedulable:NoSchedule
Unschedulable: true
Lease:
HolderIdentity: controlplane
AcquireTime:
RenewTime: Sat, 19 Oct 2024 21:54:50 +0000
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message

NetworkUnavailable False Sat, 19 Oct 2024 20:22:08 +0000 Sat, 19 Oct 2024 20:22:08 +0000 FlannelIsUp Flannel is running on this node
MemoryPressure False Sat, 19 Oct 2024 21:54:02 +0000 Sat, 19 Oct 2024 20:21:40 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Sat, 19 Oct 2024 21:54:02 +0000 Sat, 19 Oct 2024 20:21:40 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Sat, 19 Oct 2024 21:54:02 +0000 Sat, 19 Oct 2024 20:21:40 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Sat, 19 Oct 2024 21:54:02 +0000 Sat, 19 Oct 2024 20:22:06 +0000 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 192.28.114.9
Hostname: controlplane
Capacity:
cpu: 36
ephemeral-storage: 1016057248Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 214587056Ki
pods: 110
Allocatable:
cpu: 36
ephemeral-storage: 936398358207
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 214484656Ki
pods: 110
System Info:
Machine ID: 0da64fac86854105ace2e88bf3ef613e
System UUID: 9cf43679-f805-af36-6141-fda15feea999
Boot ID: c43a8b80-e9ad-4915-a36d-eafbaea1628f
Kernel Version: 5.4.0-1106-gcp
OS Image: Ubuntu 22.04.5 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.6.26
Kubelet Version: v1.30.0
Kube-Proxy Version: v1.30.0
PodCIDR: 10.244.0.0/24
PodCIDRs: 10.244.0.0/24
Non-terminated Pods: (6 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age

kube-flannel kube-flannel-ds-4zm27 100m (0%) 0 (0%) 50Mi (0%) 0 (0%) 92m
kube-system etcd-controlplane 100m (0%) 0 (0%) 100Mi (0%) 0 (0%) 93m
kube-system kube-apiserver-controlplane 250m (0%) 0 (0%) 0 (0%) 0 (0%) 93m
kube-system kube-controller-manager-controlplane 200m (0%) 0 (0%) 0 (0%) 0 (0%) 93m
kube-system kube-proxy-vtfqt 0 (0%) 0 (0%) 0 (0%) 0 (0%) 92m
kube-system kube-scheduler-controlplane 100m (0%) 0 (0%) 0 (0%) 0 (0%) 93m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits

cpu 750m (2%) 0 (0%)
memory 150Mi (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events:
Type Reason Age From Message

Normal NodeNotSchedulable 7m46s kubelet Node controlplane status is now: NodeNotSchedulable

Name: node01
Roles:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=node01
kubernetes.io/os=linux
Annotations: flannel.alpha.coreos.com/backend-data: {“VNI”:1,“VtepMAC”:“ee:72:b2:7f:a8:8e”}
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: true
flannel.alpha.coreos.com/public-ip: 192.28.114.12
kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Sat, 19 Oct 2024 20:22:30 +0000
Taints:
Unschedulable: false
Lease:
HolderIdentity: node01
AcquireTime:
RenewTime: Sat, 19 Oct 2024 21:54:52 +0000
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message

NetworkUnavailable False Sat, 19 Oct 2024 20:22:36 +0000 Sat, 19 Oct 2024 20:22:36 +0000 FlannelIsUp Flannel is running on this node
MemoryPressure False Sat, 19 Oct 2024 21:54:55 +0000 Sat, 19 Oct 2024 20:22:30 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Sat, 19 Oct 2024 21:54:55 +0000 Sat, 19 Oct 2024 20:22:30 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Sat, 19 Oct 2024 21:54:55 +0000 Sat, 19 Oct 2024 20:22:30 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Sat, 19 Oct 2024 21:54:55 +0000 Sat, 19 Oct 2024 20:22:34 +0000 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 192.28.114.12
Hostname: node01
Capacity:
cpu: 36
ephemeral-storage: 1016057248Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 214587048Ki
pods: 110
Allocatable:
cpu: 36
ephemeral-storage: 936398358207
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 214484648Ki
pods: 110
System Info:
Machine ID: 4640faee703a4be3a2aecf05da0e8976
System UUID: 1c118f50-8f55-130d-4ecb-44ecf3647ea2
Boot ID: a6eff6dd-909b-4d0b-8e6b-12444b134df1
Kernel Version: 5.4.0-1106-gcp
OS Image: Ubuntu 22.04.5 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.6.26
Kubelet Version: v1.30.0
Kube-Proxy Version: v1.30.0
PodCIDR: 10.244.1.0/24
PodCIDRs: 10.244.1.0/24
Non-terminated Pods: (9 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age

default blue-fffb6db8d-227sn 0 (0%) 0 (0%) 0 (0%) 0 (0%) 18m
default blue-fffb6db8d-4jnbx 0 (0%) 0 (0%) 0 (0%) 0 (0%) 18m
default blue-fffb6db8d-4nqbk 0 (0%) 0 (0%) 0 (0%) 0 (0%) 7m52s
default blue-fffb6db8d-tknm8 0 (0%) 0 (0%) 0 (0%) 0 (0%) 18m
default blue-fffb6db8d-wllfh 0 (0%) 0 (0%) 0 (0%) 0 (0%) 7m52s
kube-flannel kube-flannel-ds-tkb7t 100m (0%) 0 (0%) 50Mi (0%) 0 (0%) 92m
kube-system coredns-768b85b76f-g4tfd 100m (0%) 0 (0%) 70Mi (0%) 170Mi (0%) 7m52s
kube-system coredns-768b85b76f-t5msc 100m (0%) 0 (0%) 70Mi (0%) 170Mi (0%) 7m52s
kube-system kube-proxy-8xnv8 0 (0%) 0 (0%) 0 (0%) 0 (0%) 92m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits

cpu 300m (0%) 0 (0%)
memory 190Mi (0%) 340Mi (0%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events:

#2

The pods belong to a deployment, and when you evict those pods from controlplane, the deployment will recreate them on an available node that is not cordoned. You can see this by doing k get po -o wide, which will show you that the pods have indeed been moved.

#3

@rob_kodekloud …Thanks for your response.

When we drain the control plane what will happen to below POD’s? Will they move to node01?

Or they will get terminated and once the control plane comes up it will spawn new PODs with the below name ?

The below POD’s are associated with control plane correct.

kube-flannel kube-flannel-ds-4zm27 1/1 Running 0 90m
kube-flannel kube-flannel-ds-tkb7t 1/1 Running 0 90m
kube-system coredns-768b85b76f-g4tfd 1/1 Running 0 5m39s
kube-system coredns-768b85b76f-t5msc 1/1 Running 0 5m39s
kube-system etcd-controlplane 1/1 Running 0 90m
kube-system kube-apiserver-controlplane 1/1 Running 0 90m
kube-system kube-controller-manager-controlplane 1/1 Running 0 90m
kube-system kube-proxy-8xnv8 1/1 Running 0 90m
kube-system kube-proxy-vtfqt 1/1 Running 0 90m
kube-system kube-scheduler-controlplane 1/1 Running 0 91m

#4

The kube-flannel and kube-proxy pods are owned by a daemonset, and are “ignored” by the version of the command you invoked. coredns is a deployment; I’m not sure how they behave under a drain, but I expect it’s like any other deployment. The rest are static pods, and I don’t think “drain” will affect them, especially since they cannot be moved by drain – they belong to the kubelet process which is node specific.

#5

The purpose of drain is to evict as many pods as can be evicted, then cordon the node. Certain pods cannot be evicted:

  • Daemonset pods, because the idea of a daemonset is to run exactly one pod on every node at all times, hence --ignore-daemonsets
  • Static pods like the control plane pods, because they are fixed to a node and not managed by the scheduler. When doing an upgrade, the control plane pods are what is being upgraded, so they are managed directly by kubeadm. kubeadm also upgrades the kube-proxy daemonset.

Everything else, including CoreDNS will be evicted as normal. This is why it’s good practice to have the CoreDNS deployment at a minimum of 2 replicas, since the eviction will move one pod at a time and DNS service won’t be interrupted.

In a production cluster you run ALL workloads with a minimum of 2 pods and Anti Affinity (to ensure the pods of the deployment run on different nodes at all times), even if one pod can easily manage the load on its own - this is so you get no service interruption if an eviction occurs.

1 Like