Normal Scheduled 40s default-scheduler Successful . . .

Kubernetes-Slack Discussions
#1

Ahmed Aly:
Normal Scheduled 40s default-scheduler Successfully assigned kube-system/kube-proxy-7w9dg to node1
Warning FailedCreatePodSandBox 4s (x2 over 30s) kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed pulling image “http://k8s.gcr.io/pause:3.5|k8s.gcr.io/pause:3.5”: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 108.177.15.82:443: connect: connection refused

#2

unnivkn:
Looks like its an image pull issue. Could you please check the image http://k8s.gcr.io/pause:3.5|k8s.gcr.io/pause:3.5 is correct. Or else just test with different image eg: nginx

#3

Karoon Khatiwada:
I believe that pause is just a placeholder. Are you able to telnet to that IP and port from your local?

#4

Ahmed Aly:
image already there

#5

Ahmed Aly:
i try nslookup

#6

unnivkn:
http://k8s.gcr.io/pause:3.5|pause:3.5 may be a side car container which may be serving the main container

#7

Raamkanna Saranathan:
the error is pretty clear, the machine that your k8s is on is unable to reach 108.177.15.82:443

#8

Ahmed Aly:
ping 108.177.15.82
PING 108.177.15.82 tel:(108.177.15.82|(108.177.15.82) 56(84) bytes of data.
64 bytes from 108.177.15.82: icmp_seq=1 ttl=107 time=104 ms
64 bytes from 108.177.15.82: icmp_seq=2 ttl=107 time=104 ms
64 bytes from 108.177.15.82: icmp_seq=3 ttl=107 time=103 ms

#9

Ahmed Aly:
may be issue with CNI

#10

Ahmed Aly:
kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed pulling image “http://k8s.gcr.io/pause:3.5|k8s.gcr.io/pause:3.5”: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 108.177.15.82:443: connect: connection refused

#11

Ahmed Aly:
kube-system calico-kube-controllers-58497c65d5-95vwj 1/1 Running 1 (119m ago) 17h
kube-system calico-node-kwcsg 1/1 Running 1 (119m ago) 17h
kube-system coredns-78fcd69978-6lg4x 1/1 Running 1 (119m ago) 18h
kube-system coredns-78fcd69978-cnqpd 1/1 Running 1 (119m ago) 18h
kube-system etcd-k8s-master 1/1 Running 2 (119m ago) 18h
kube-system kube-apiserver-k8s-master 1/1 Running 2 (119m ago) 18h
kube-system kube-controller-manager-k8s-master 1/1 Running 1 (119m ago) 18h
kube-system kube-proxy-7w9dg 0/1 ContainerCreating 0 77m
kube-system kube-proxy-9hbjk 0/1 ContainerCreating 0 94m
kube-system kube-proxy-vzmj6 1/1 Running 1 (119m ago) 18h
kube-system kube-scheduler-k8s-master 1/1 Running 2 (119m ago) 18h

#12

Ahmed Aly:
docker pull http://k8s.gcr.io/pause:3.5|k8s.gcr.io/pause:3.5
3.5: Pulling from pause
Digest: sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07
Status: Image is up to date for http://k8s.gcr.io/pause:3.5|k8s.gcr.io/pause:3.5
http://k8s.gcr.io/pause:3.5|k8s.gcr.io/pause:3.5

#13

unnivkn:
what it is showing in kubelet logs: journalctl -fu kubelet

#14

Ahmed Aly:
k8s-master kubelet[891]: E0809 20:25:16.871305 891 projected.go:199] Error preparing data for projected volume kube-api-access-xxrvw for pod kube-system/calico-node-kwcsg: failed to fetch token: serviceaccounts “calico-node” not found
أغسطس 09 20:25:16 k8s-master kubelet[891]: E0809 20:25:16.871551 891 nestedpendingoperations.go:301] Operation for “{volumeName:http://kubernetes.io/projected/e3999fed-d4c9-4da9-8371-2cd3c4398170-kube-api-access-xxrvw|kubernetes.io/projected/e3999fed-d4c9-4da9-8371-2cd3c4398170-kube-api-access-xxrvw podName:e3999fed-d4c9-4da9-8371-2cd3c4398170 nodeName:}” failed. No retries permitted until 2021-08-09 20:27:18.871489993 +0300 +03 m=+6954.922304766 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume “kube-api-access-xxrvw” (UniqueName: “http://kubernetes.io/projected/e3999fed-d4c9-4da9-8371-2cd3c4398170-kube-api-access-xxrvw|kubernetes.io/projected/e3999fed-d4c9-4da9-8371-2cd3c4398170-kube-api-access-xxrvw”) pod “calico-node-kwcsg” (UID: “e3999fed-d4c9-4da9-8371-2cd3c4398170”) : failed to fetch token: serviceaccounts “calico-node” not found
أغسطس 09 20:27:18 k8s-master kubelet[891]: E0809 20:27:18.961976 891 projected.go:199] Error preparing data for projected volume kube-api-access-xxrvw for pod kube-system/calico-node-kwcsg: failed to fetch token: serviceaccounts “calico-node” not found
أغسطس 09 20:27:18 k8s-master kubelet[891]: E0809 20:27:18.962239 891 nestedpendingoperations.go:301] Operation for “{volumeName:http://kubernetes.io/projected/e3999fed-d4c9-4da9-8371-2cd3c4398170-kube-api-access-xxrvw|kubernetes.io/projected/e3999fed-d4c9-4da9-8371-2cd3c4398170-kube-api-access-xxrvw podName:e3999fed-d4c9-4da9-8371-2cd3c4398170 nodeName:}” failed. No retries permitted until 2021-08-09 20:29:20.962175889 +0300 +03 m=+7077.012990662 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume “kube-api-access-xxrvw” (UniqueName: “http://kubernetes.io/projected/e3999fed-d4c9-4da9-8371-2cd3c4398170-kube-api-access-xxrvw|kubernetes.io/projected/e3999fed-d4c9-4da9-8371-2cd3c4398170-kube-api-access-xxrvw”) pod “calico-node-kwcsg” (UID: “e3999fed-d4c9-4da9-8371-2cd3c4398170”) : failed to fetch token: serviceaccounts “calico-node” not found
أغسطس 09 20:29:21 k8s-master kubelet[891]: E0809 20:29:21.082113 891 projected.go:199] Error preparing data for projected volume kube-api-access-xxrvw for pod kube-system/calico-node-kwcsg: failed to fetch token: serviceaccounts “calico-node” not found

#15

Ahmed Aly:
issue seems to be in network calico

#16

unnivkn:
kube-api-access-xxrvw for pod kube-system/calico-node-kwcsg: failed to fetch token: serviceaccounts “calico-node” not found