Hi All, I have a question on the following exercise under *Section 13 troublesho . . .

Kubernetes-Slack Discussions
#1

Krishnaprasad:
Hi All, I have a question on the following exercise under Section 13 troubleshooting > Practice Test TroubleShooting nextworks > question 2. I understand that the kubeproxy pod is crashing with a crashloopbackoff. Kube-system kube-proxy-b2c7j 0/1 CrashLoopBackOff 4 119s

  1. getting the logs, I understand that the kubeprox pod is referring to the wrong configuration file. My question is 1. How do I know which configuration file is the correct one ?
    1. How do I correlate the configmap with the configuration file ? The solution says that “Check the logs of the kube-proxy pods kubectl -n kube-system logs <name_of_the_kube_proxy_pod> The configuration file /var/lib/kube-proxy/configuration.conf is not valid. The configuration path does not match the data in the ConfigMap. kubectl -n kube-system describe configmap kube-proxy shows that the file name used is config.conf”. How do I make a correlation between the configuration path and the data in the ConfigMap ?
#2

unnivkn:
Hi @Krishnaprasad please follow this:

#3

unnivkn:
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
2)

Troubleshooting Test 2: The same 2 tier application is having issues again. It must display a green web page on success. Click on the app tab at the top of your terminal to view your application. It is currently failed. Troubleshoot and fix the issue.

Stick to the given architecture. Use the same names and port numbers as given in the below architecture diagram. Feel free to edit, delete or recreate objects as necessary.

Error:

Connecting to Port 30081
We’re currently trying to connect to a HTTP service running on 30081. Services can sometimes take a few moments to start, even up to five minutes.

–Hint:
There seems to be an issue with the Service Proxy. Inspect and Fix the kube-proxy daemonset.
Check logs of the kube-proxy pod. It appears that the daemonset is using a wrong configuration file.
Compare the configuration file used in the daemonset with the configmap for kube-proxy.

Edit the kube-proxy daemonset to correct the configuration file using kubectl -n kube-system edit ds kube-proxy.

Refer answer at /var/answers/answer2.md

cat /var/answers/answer2.md

The kube-proxy pods are not running. As a result the rules needed to allow connectivity to the services have not been created.

  1. Check the logs of the kube-proxy pods
    kubectl -n kube-system logs <name_of_the_kube_proxy_pod>

  2. The configuration file /var/lib/kube-proxy/configuration.conf is not valid. The configuration path does not match the data in the ConfigMap.
    kubectl -n kube-system describe configmap kube-proxy shows that the file name used is config.conf which is mounted in the kube-proxy damonset pods at the path /var/lib/kube-proxy/config.conf

  3. However in the DaemonSet for kube-proxy, the command used to start the kube-proxy pod makes use of the path /var/lib/kube-proxy/configuration.conf.

Correct this path in ds to /var/lib/kube-proxy/config.conf as per the ConfigMap and recreate the kube-proxy pods.

This should get the kube-proxy pods back in a running state.
~

–From kube-proxy DS (daemonsets)

spec:
containers:
- command:
- /usr/local/bin/kube-proxy
- --config=/var/lib/kube-proxy/configuration.conf
- --hostname-override=$(NODE_NAME)
env:

–from kube-proxy configmaps

Data

config.conf:

apiVersion: http://kubeproxy.config.k8s.io/v1alpha1|kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
bindAddressHardFail: false
clientConnection:
acceptContentTypes: “”
burst: 0
contentType: “”

#4

unnivkn:
Solution:

controlplane $
controlplane $ kubectl get pods
No resources found in default namespace.
controlplane $

controlplane $ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-f9fd979d6-bmkhv 1/1 Running 0 6m51s
coredns-f9fd979d6-ftcpk 1/1 Running 0 6m37s
etcd-controlplane 1/1 Running 0 7m24s
kube-apiserver-controlplane 1/1 Running 0 7m24s
kube-controller-manager-controlplane 1/1 Running 0 7m23s
kube-proxy-n2jf5 0/1 CrashLoopBackOff 4 2m5s
kube-proxy-p6wzj 0/1 CrashLoopBackOff 4 2m5s
kube-scheduler-controlplane 1/1 Running 0 7m24s
controlplane $

controlplane $ kubectl get configmaps -n kube-system
NAME DATA AGE
coredns 1 7m56s
extension-apiserver-authentication 6 7m59s
kube-proxy 2 7m55s
kubeadm-config 2 7m57s
kubelet-config-1.19 1 7m57s
weave-net 0 6m57s
controlplane $

controlplane $ kubectl get daemonsets.apps -n kube-system
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-flannel-ds-arm 0 0 0 0 0 <none> 8m14s
kube-flannel-ds-arm64 0 0 0 0 0 <none> 8m14s
kube-flannel-ds-ppc64le 0 0 0 0 0 <none> 8m14s
kube-flannel-ds-s390x 0 0 0 0 0 <none> 8m14s
kube-proxy 2 2 0 2 0 http://kubernetes.io/os=linux|kubernetes.io/os=linux 2m49s
controlplane $

controlplane $ kubectl -n kube-system describe pod kube-proxy-n2jf5
Name: kube-proxy-n2jf5
Namespace: kube-system
Priority: 2000001000
Priority Class Name: system-node-critical
Node: controlplane/172.17.0.20
Start Time: Fri, 19 Feb 2021 05:17:52 +0000
Labels: controller-revision-hash=59dc5879c9
k8s-app=kube-proxy
pod-template-generation=1
Annotations: <none>
Status: Running
IP: 172.17.0.20
IPs:
IP: 172.17.0.20
Controlled By: DaemonSet/kube-proxy
Containers:
kube-proxy:
Container ID: docker://e84d3e6c197b975519b156caca204271c896a54c87c291c7ce84c465c4049feb
Image: http://k8s.gcr.io/kube-proxy:v1.18.0|k8s.gcr.io/kube-proxy:v1.18.0
Image ID: docker-pullable://k8s.gcr.io/kube-proxy@sha256:9e858386d52d0abaf936c1d10a763648ab7d85c8eb0af08a50a64238146e5571
Port: <none>
Host Port: <none>
Command:
/usr/local/bin/kube-proxy
–config=/var/lib/kube-proxy/configuration.conf
–hostname-override=$(NODE_NAME)
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Fri, 19 Feb 2021 05:20:57 +0000
Finished: Fri, 19 Feb 2021 05:20:57 +0000
Ready: False
Restart Count: 5
Environment:
NODE_NAME: (v1:spec.nodeName)
Mounts:
/lib/modules from lib-modules (ro)
/run/xtables.lock from xtables-lock (rw)
/var/lib/kube-proxy from kube-proxy (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-proxy-token-qrh6z (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-proxy:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kube-proxy
Optional: false
xtables-lock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
HostPathType: FileOrCreate
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
kube-proxy-token-qrh6z:
Type: Secret (a volume populated by a Secret)
SecretName: kube-proxy-token-qrh6z
Optional: false
QoS Class: BestEffort
Node-Selectors: http://kubernetes.io/os=linux|kubernetes.io/os=linux
Tolerations: op=Exists
CriticalAddonsOnly op=Exists
http://node.kubernetes.io/disk-pressure:NoSchedule|node.kubernetes.io/disk-pressure:NoSchedule op=Exists
http://node.kubernetes.io/memory-pressure:NoSchedule|node.kubernetes.io/memory-pressure:NoSchedule op=Exists
http://node.kubernetes.io/network-unavailable:NoSchedule|node.kubernetes.io/network-unavailable:NoSchedule op=Exists
http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute op=Exists
http://node.kubernetes.io/pid-pressure:NoSchedule|node.kubernetes.io/pid-pressure:NoSchedule op=Exists
http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute op=Exists
http://node.kubernetes.io/unschedulable:NoSchedule|node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message

Normal Scheduled 3m43s default-scheduler Successfully assigned kube-system/kube-proxy-n2jf5 to controlplane
Normal Pulling 3m42s kubelet, controlplane Pulling image “http://k8s.gcr.io/kube-proxy:v1.18.0|k8s.gcr.io/kube-proxy:v1.18.0”
Normal Pulled 3m35s kubelet, controlplane Successfully pulled image “http://k8s.gcr.io/kube-proxy:v1.18.0|k8s.gcr.io/kube-proxy:v1.18.0” in 6.321273009s
Normal Created 2m3s (x5 over 3m35s) kubelet, controlplane Created container kube-proxy
Normal Started 2m3s (x5 over 3m35s) kubelet, controlplane Started container kube-proxy
Normal Pulled 2m3s (x4 over 3m34s) kubelet, controlplane Container image “http://k8s.gcr.io/kube-proxy:v1.18.0|k8s.gcr.io/kube-proxy:v1.18.0” already present on machine
Warning BackOff 2m2s (x9 over 3m33s) kubelet, controlplane Back-off restarting failed container
controlplane $
controlplane $

controlplane $ kubectl -n kube-system logs kube-proxy-n2jf5
F0219 05:20:57.504053 1 server.go:490] failed complete: open /var/lib/kube-proxy/configuration.conf: no such file or directory
controlplane $

controlplane $ kubectl -n kube-system edit configmaps kube-proxy
Edit cancelled, no changes made.
controlplane $

#5

unnivkn:
controlplane $ kubectl -n kube-system describe configmap kube-proxy
Name: kube-proxy
Namespace: kube-system
Labels: app=kube-proxy
Annotations: http://kubeadm.kubernetes.io/component-config.hash|kubeadm.kubernetes.io/component-config.hash: sha256:6b794190f1cbb93cac91a87afaf10a0aca5e87fbabf1918704beea1a120eb66d

Data

config.conf:

apiVersion: http://kubeproxy.config.k8s.io/v1alpha1|kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
bindAddressHardFail: false
clientConnection:
acceptContentTypes: “”
burst: 0
contentType: “”
kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
qps: 0
clusterCIDR: 10.244.0.0/16
configSyncPeriod: 0s
conntrack:
maxPerCore: null
min: null
tcpCloseWaitTimeout: null
tcpEstablishedTimeout: null
detectLocalMode: “”
enableProfiling: false
healthzBindAddress: “”
hostnameOverride: “”
iptables:
masqueradeAll: false
masqueradeBit: null
minSyncPeriod: 0s
syncPeriod: 0s
ipvs:
excludeCIDRs: null
minSyncPeriod: 0s
scheduler: “”
strictARP: false
syncPeriod: 0s
tcpFinTimeout: 0s
tcpTimeout: 0s
udpTimeout: 0s
kind: KubeProxyConfiguration
metricsBindAddress: “”
mode: “”
nodePortAddresses: null
oomScoreAdj: null
portRange: “”
showHiddenMetricsForVersion: “”
udpIdleTimeout: 0s
winkernel:
enableDSR: false
networkName: “”
sourceVip: “”
kubeconfig.conf:

apiVersion: v1
kind: Config
clusters:

  • cluster:
    certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    server: https://172.17.0.20:6443
    name: default
    contexts:
  • context:
    cluster: default
    namespace: default
    user: default
    name: default
    current-context: default
    users:
  • name: default
    user:
    tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    Events: <none>
    controlplane $

controlplane $
controlplane $ kubectl -n kube-system edit daemonsets.apps kube-proxy
daemonset.apps/kube-proxy edited
controlplane $

updated in kube-proxy DS (daemonsets)

spec:
containers:
- command:
- /usr/local/bin/kube-proxy
- --config=/var/lib/kube-proxy/config.conf ##updated
- --hostname-override=$(NODE_NAME)
env:

controlplane $ kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
coredns-f9fd979d6-bmkhv 1/1 Running 0 17m
coredns-f9fd979d6-ftcpk 1/1 Running 0 17m
etcd-controlplane 1/1 Running 0 17m
kube-apiserver-controlplane 1/1 Running 0 17m
kube-controller-manager-controlplane 1/1 Running 0 17m
kube-proxy-lhwg9 1/1 Running 0 19s
kube-proxy-pxphp 1/1 Running 0 22s
kube-scheduler-controlplane 1/1 Running 0 17m
controlplane $

controlplane $
controlplane $ kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
coredns-f9fd979d6-bmkhv 1/1 Running 0 18m
coredns-f9fd979d6-ftcpk 1/1 Running 0 18m
etcd-controlplane 1/1 Running 0 19m
kube-apiserver-controlplane 1/1 Running 0 19m
kube-controller-manager-controlplane 1/1 Running 0 19m
kube-proxy-lhwg9 1/1 Running 0 99s
kube-proxy-pxphp 1/1 Running 0 102s
kube-scheduler-controlplane 1/1 Running 0 19m
controlplane $

Kube-proxy is running now: but still same error:

Error:
Connecting to Port 30081
We’re currently trying to connect to a HTTP service running on 30081. Services can sometimes take a few moments to start, even up to five minutes.

controlplane $ kubectl apply -f “https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d ‘\n’)”
serviceaccount/weave-net created
http://clusterrole.rbac.authorization.k8s.io/weave-net|clusterrole.rbac.authorization.k8s.io/weave-net created
http://clusterrolebinding.rbac.authorization.k8s.io/weave-net|clusterrolebinding.rbac.authorization.k8s.io/weave-net created
http://role.rbac.authorization.k8s.io/weave-net|role.rbac.authorization.k8s.io/weave-net created
http://rolebinding.rbac.authorization.k8s.io/weave-net|rolebinding.rbac.authorization.k8s.io/weave-net created
daemonset.apps/weave-net created
controlplane $

controlplane $
controlplane $ kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
coredns-f9fd979d6-bmkhv 1/1 Running 0 19m
coredns-f9fd979d6-ftcpk 1/1 Running 0 18m
etcd-controlplane 1/1 Running 0 19m
kube-apiserver-controlplane 1/1 Running 0 19m
kube-controller-manager-controlplane 1/1 Running 0 19m
kube-proxy-lhwg9 1/1 Running 0 115s
kube-proxy-pxphp 1/1 Running 0 118s
kube-scheduler-controlplane 1/1 Running 0 19m
weave-net-46d5g 2/2 Running 0 4s
weave-net-655l2 1/2 Running 0 4s
controlplane $

controlplane $
controlplane $ kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
coredns-f9fd979d6-bmkhv 1/1 Running 0 19m
coredns-f9fd979d6-ftcpk 1/1 Running 0 18m
etcd-controlplane 1/1 Running 0 19m
kube-apiserver-controlplane 1/1 Running 0 19m
kube-controller-manager-controlplane 1/1 Running 0 19m
kube-proxy-lhwg9 1/1 Running 0 2m
kube-proxy-pxphp 1/1 Running 0 2m3s
kube-scheduler-controlplane 1/1 Running 0 19m
weave-net-46d5g 2/2 Running 0 9s
weave-net-655l2 2/2 Running 0 9s
controlplane $
controlplane $

Note: App is wkg now as expected.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%