Upgraded k8 worker node from ubuntu 20.04 to 22.04. DNS resolution inside pods doesn't work

Kubernetes
,
#1

I have a k8 cluster based on Ubuntu 20.04
1 master and 3 worker nodes.

I drained one of the worker node. Put kubectl, kubeadm, kubelet & containerd packages on hold.

OS upgrade went smooth, but after upgrade pods (kube-system daemon-sets) kept crashing. One of the issue I found is that DNS resolution is not working inside pods residing on upgraded node. When I revert back to ubuntu 20.04 everything works fine. Anyone help/suggestion please

#2

Not sure what this might be. You may want to try the #kubeadm channel of Kubernetes Slack to see if anyone there has an idea what you might need to change.

#3

how to join slack, it asks for kodekloud account

#4

Not our slack (which is no longer available to the public anyway) but Kubernetes’ slack.

#5

I was able to solve it by following:

  1. Set containerd to use systemd to manage cgroups instead of the default cgroupfs on worker nodes only:

#sed -i ‘s/SystemdCgroup = false/SystemdCgroup = true/g’ /etc/containerd/config.toml

then restart Containerd

  1. Disable IP checksum offloading as it can cause issues with flannel’s UDP packets.

#ethtool -K flannel.1 tx-checksum-ip-generic off
#ethtool -K flannel.1 rx-checksum off

To make it persistent follow : k3s on rhel 8 network/dns probleme and metrics not work · Issue #5013 · k3s-io/k3s · GitHub

  1. Install the linux-modules-extra kernel module, This is required for the flannel containers to work properly, otherwise your pods might fail due to network issues.

#apt install linux-modules-extra-$(uname -r)