Pods not running using Calico

Kubernetes
#1

root@kubemaster:~# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default nginx-6db489d4b7-hfh66 0/1 ContainerCreating 0 13h
kube-system calico-kube-controllers-6b64bcd855-qzgx7 0/1 ContainerCreating 0 13h
kube-system calico-node-fbbpj 0/1 Running 27 14h
kube-system calico-node-vzvdd 0/1 Running 29 14h
kube-system calico-node-xgldk 1/1 Running 2 13h
kube-system coredns-5644d7b6d9-p5vmm 1/1 Running 3 23h
kube-system coredns-5644d7b6d9-p8wws 1/1 Running 3 23h
kube-system etcd-kubemaster 1/1 Running 6 23h
kube-system kube-apiserver-kubemaster 1/1 Running 8 23h
kube-system kube-controller-manager-kubemaster 1/1 Running 20 23h
kube-system kube-proxy-g6wtf 1/1 Running 2 19h
kube-system kube-proxy-tfllf 1/1 Running 5 23h
kube-system kube-proxy-xpqb7 1/1 Running 2 19h
kube-system kube-scheduler-kubemaster 1/1 Running 19 23h

Warning FailedCreatePodSandBox 3m13s (x327 over 21m) kubelet, kubenode1 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “f71c0c46b9d13994f1adcc3a8cd60d688ff30b27803e6e1175ae3cf2944dcd73” network for pod “nginx-6db489d4b7-hfh66”: networkPlugin cni failed to set up pod “nginx-6db489d4b7-hfh66_default” network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

#2

It appears you have the calico pod running on one node (calico-node-xgldk) and not on the other two?

Which node is the working one, what is the the topology of your cluster?

#3

I am having one master node and 2 worked nodes joined to it. Can you please let me know how can i verify which node calico pod running and which it is not ?

#4

root@kubemaster:~# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
kubemaster Ready master 31h v1.16.3 192.168.0.102 Ubuntu 18.04.3 LTS 5.0.0-23-generic docker://18.9.7
kubenode1 Ready 27h v1.16.3 192.168.0.105 Ubuntu 18.04.3 LTS 5.0.0-23-generic docker://18.9.7
kubenode2 Ready 27h v1.16.3 192.168.0.104 Ubuntu 18.04.3 LTS 5.0.0-23-generic docker://18.9.7

#5

Seeing same error with coredns pod as well

Warning FailedCreatePodSandBox 2m33s (x200 over 16m) kubelet, kubenode1 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “0c7d1248a724c442c794db3272be188f2f9b473fe9c6eeedf43dd44f85119ac4” network for pod “coredns-5644d7b6d9-6h7jk”: networkPlugin cni failed to set up pod “coredns-5644d7b6d9-6h7jk_kube-system” network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

#6

Sorry - I though I had replied to this thread.

kube-system calico-node-fbbpj 0/1 Running 27 14h
kube-system calico-node-vzvdd 0/1 Running 29 14h
kube-system calico-node-xgldk 1/1 Running 2 13h

Which node is the pod calico-node-xgldk running on? check using kubectl get pods -o wide -n kube-system.

Definitely seems to be an issue with the connectivity between the nodes.

please refer (check nw requirements)
https://docs.projectcalico.org/v3.9/getting-started/kubernetes/requirements

#7

Did you get this issue fixed @amitsingh1509 ? If yes please provide the steps as I am also facing this issue.

#8

Have you checked the nw requirements for calico? can you list your configuration using calicoctl?

#9

Thanks for your inputs.
I wanted to bring my cluster and was facing this issue instead of further digging deep using calicoctl or understand network requirements [I felt the documentation is at higher level] mentioned in the calico site, I teared up my cluster, cleaned up environment, installed Kubeadm, kubectl & kubelet v1.16, docker version 18.09.9, calico 3.11 version & was able to bring up my cluster.

#10