Networking Issue | Pods can not communicate

Kubernetes
#1

Cluster information:
Kubernetes version: v1.32.2
Cloud being used: VMs
Installation method: kubeadm
Host OS: Ubuntu: 22.04
CNI and version: calicoctl version
Client Version: v3.20.1
Release: Calico Enterprise
CRI and version: Containerd:v2.0.3

ISSUE

  • I have a master and two worker nodes.
  • pods on same node can communicate with each other (ping)
  • pods on different nodes can’t communicate with each other

TROUBLESHOOTING DONE

  1. reinstalled CNI (calico)
  2. Pods have been assigned IPs from designated CIDR
  3. Firewall is off/disabled
  4. node-node connectivity is good
  5. deleted all network policies and created a new one which allows all traffic

Info:
pods on worker-node1 are pingable only from worker-node1
pods on worker-node2 are pingable only from worker-node2

BUT

pods on worker-node1 are not pingable only from worker-node2 and vice versa
pods also can’t reach external network means apt update don’t work

#2

I haven’t used Calico lately.
Have you verified IPIP or VXLAN configs?

#3

@Santosh_KodeKloud thanks for reply!

no, i didn’t do the IPIP or VXLAN configs.

let me check that

#4

You also mentioned there’s an issue with DNS resolution as well.

You need to look at this for installation instruction for your specific cloud provider

#5

Mine is on VMs.
it’s not specifically on any cloud-provider.

but VMs are created on openstack

#6

Hi

my issue got resolved after node reboot

Thanks for help

#7

Issue got resolved after changing encapsulation from VXLAN to IPIP

could you tell why VXLAN didn’t work?

#8

I figured out that, DNS is working but very slow response

root@test-app-758bbbb66b-m5bxw:/usr/local/apache2# ping google.com PING google.com (192.178.24.238) 56(84) bytes of data. 64 bytes from mct04s04-in-f14.1e100.net (192.178.24.238): icmp_seq=1 ttl=55 time=33.9 ms 64 bytes from mct04s04-in-f14.1e100.net (192.178.24.238): icmp_seq=2 ttl=55 time=32.5 ms 64 bytes from mct04s04-in-f14.1e100.net (192.178.24.238): icmp_seq=3 ttl=55 time=32.3 ms 64 bytes from mct04s04-in-f14.1e100.net (192.178.24.238): icmp_seq=4 ttl=55 time=32.5 ms 64 bytes from mct04s04-in-f14.1e100.net (192.178.24.238): icmp_seq=5 ttl=55 time=32.3 ms ^C64 bytes from 192.178.24.238: icmp_seq=6 ttl=55 time=32.3 ms --- google.com ping statistics --- 6 packets transmitted, 6 received, 0% packet loss, time 25870ms rtt min/avg/max/mdev = 32.281/32.645/33.940/0.585 ms root@test-app-758bbbb66b-m5bxw:/usr/local/apache2# apt update 0% [Connecting to deb.debian.org]^C root@test-app-758bbbb66b-m5bxw:/usr/local/apache2# root@test-app-758bbbb66b-m5bxw:/usr/local/apache2# root@test-app-758bbbb66b-m5bxw:/usr/local/apache2# root@test-app-758bbbb66b-m5bxw:/usr/local/apache2# ping deb.debian.org PING debian.map.fastlydns.net (151.101.194.132) 56(84) bytes of data. 64 bytes from 151.101.194.132 (151.101.194.132): icmp_seq=1 ttl=55 time=96.5 ms 64 bytes from 151.101.194.132 (151.101.194.132): icmp_seq=2 ttl=55 time=95.6 ms 64 bytes from 151.101.194.132 (151.101.194.132): icmp_seq=3 ttl=55 time=95.5 ms ^C64 bytes from 151.101.194.132: icmp_seq=4 ttl=55 time=95.4 ms

moreover, i did this and it worked

root@apache-server-86bdb6869f-cf6jj:/usr/local/apache2# echo "151.101.194.132 deb.debian.org" | tee -a /etc/hosts > /dev/null root@apache-server-86bdb6869f-cf6jj:/usr/local/apache2# root@apache-server-86bdb6869f-cf6jj:/usr/local/apache2# root@apache-server-86bdb6869f-cf6jj:/usr/local/apache2# apt update Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB] Get:2 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB] Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB] Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8792 kB] Get:5 http://deb.debian.org/debian bookworm-updates/main amd64 Packages [13.5 kB] Get:6 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [250 kB] Fetched 9310 kB in 3s (3692 kB/s) Reading package lists... Done Building dependency tree... Done Reading state information... Done All packages are up to date. root@apache-server-86bdb6869f-cf6jj:/usr/local/apache2# root@apache-server-86bdb6869f-cf6jj:/usr/local/apache2# root@apache-server-86bdb6869f-cf6jj:/usr/local/apache2# root@apache-server-86bdb6869f-cf6jj:/usr/local/apache2# root@apache-server-86bdb6869f-cf6jj:/usr/local/apache2#

adding namserver 8.8.8.8 in /etc/resolv.conf also worked